Level the ecosystem docs: cross-module topic pages linking module details, campaign and data-analysis notebooks, tutorial drift repair, aggregate reference sync, and submodules at current mains

Author: rartino

docs/analysis.md

@@ -0,0 +1,30 @@
+# Analysis
+
+Analysis starts with collected records, not with a new storage layer. Query a
+local store using its search DSL, pass compositions and energies to
+*httk-analyse*, and keep the returned hull or phase diagram as the immutable
+analysis snapshot. For a binary phase diagram, the documented API is:
+
+```python
+from httk.analyse.matsci import PhaseDiagram
+
+diagram = PhaseDiagram.from_compositions(
+    [{"A": 1}, {"B": 1}, {"A": 1, "B": 1}],
+    [0.0, 0.0, -2.0],
+    ids=["A", "B", "AB"],
+)
+assert tuple(diagram.hull_indices) == (0, 1, 2)
+assert diagram.energy_above_hull[2] == 0.0
+```
+
+`energy_above_hull` identifies entries above the stable lower envelope, while
+`phase_lines` exposes the supported boundaries. Query stores before building a
+diagram so the analysis input is explicit and reproducible. For data outside
+your local store, `OptimadeStore` is the synchronous read-only OPTIMADE client;
+discover an endpoint, query it, and save any resources you want to analyze
+locally.
+
+## Read next
+
+- [Generic lower hulls](https://docs.httk.org/httk-analyse/dev/main/generic-hulls/) and [phase diagrams](https://docs.httk.org/httk-analyse/dev/main/phase-diagrams/).
+- [OPTIMADE client](https://docs.httk.org/httk-serve/dev/main/optimade/client/), [data queries](https://docs.httk.org/httk-data/dev/main/db/), and [database details](https://docs.httk.org/httk-data/dev/main/details/db/).

docs/architecture.md

@@ -73,3 +73,5 @@ storage backend instead of leaking into the models or their callers.
 Most views convert from their backend only when data is actually requested.
 Constructing a view does not eagerly convert the underlying data, which avoids
 unnecessary conversions when a view is only passed along or partially used.
+
+Read next: {doc}`structures`, {doc}`data`, {doc}`campaigns`, and {doc}`analysis`.

docs/campaigns.md

@@ -0,0 +1,56 @@
+# Compute campaigns, small and large
+
+Start locally with the four-command cycle: initialize a workspace, create a
+job, run its manager, and collect the result. The packaged `vasp-relax`
+workflow accepts a POSCAR and needs no runner authoring:
+
+```console
+$ httk workflow workspace init . --name default
+$ httk workflow job new --workflow vasp-relax --input structure=POSCAR
+$ httk workflow run
+$ httk workflow collect --into results.sqlite
+```
+
+The workspace holds durable state and provenance; collection is the boundary
+where finished jobs become records in *httk-data*. Re-collecting is safe and
+deduplicated. Run `httk workflow precheck WORKSPACE` before starting managers
+to report missing settings, runner references, and machine readiness.
+
+For a remote, add and configure the machine, initialize its workspace, then
+transfer jobs and run a manager there:
+
+```console
+httk workflow remote add kappa --template ssh-slurm
+httk workflow remote configure kappa \
+    --set host=kappa.example.org --set username=rar \
+    --set check_connectivity=yes
+httk workflow remote install kappa
+httk workflow workspace init kappa:/scratch/rar/httk/runs
+httk workflow workspace settings set kappa:runs slurm.partition batch
+httk workflow workspace settings set kappa:runs vasp.command "srun -n 32 vasp_std"
+httk workflow transfer default kappa:runs --job JOB-ID
+httk workflow precheck kappa:runs
+httk workflow run kappa:runs --workers 8
+httk workflow transfer kappa:runs default --state succeeded --state failed
+```
+
+The remote workspace owns scheduler settings. A large campaign partitions
+ordinary workspaces, assigns root jobs by hash, round-robin, or explicit name,
+and keeps spawned children with their parent's partition. Use
+`campaign init`, `campaign submit`, `campaign start-managers`, and
+`campaign collect` to manage those partitions one at a time or together.
+
+Author reusable workflows as packages with an `httk_workflow.toml` manifest.
+The nine-language SDK family gives the same runner surface from Python, Bash,
+C, Fortran, Rust, Perl, Ada, C++, and Java. jobflow/atomate2, CWL, PWD, and
+httk-v1 documents are also normal workflow language realizations.
+
+Compiled packages declare `[workflow.build]`. Publication carries sources-only
+digests; `httk workflow build` builds and registers a binary per machine, so
+managers execute registered artifacts and never compile jobs themselves.
+
+## Read next
+
+- [Workflow quickstart](https://docs.httk.org/httk-workflow/dev/main/quickstart/), [campaigns](https://docs.httk.org/httk-workflow/dev/main/campaigns/), [collecting](https://docs.httk.org/httk-workflow/dev/main/collecting/), and [CLI](https://docs.httk.org/httk-workflow/dev/main/workflow_cli/).
+- [Workflow package authoring](https://docs.httk.org/httk-workflow/dev/main/workflow_packages/), [languages](https://docs.httk.org/httk-workflow/dev/main/workflow_languages/), and [SDKs](https://docs.httk.org/httk-workflow/dev/main/sdks/).
+- [CLI details](https://docs.httk.org/httk-workflow/dev/main/details/workflow_cli/), [package details](https://docs.httk.org/httk-workflow/dev/main/details/workflow_packages/), and [task-manager details](https://docs.httk.org/httk-workflow/dev/main/details/taskmanager/).

docs/conf.py

@@ -142,6 +142,13 @@
     ("py:obj", "ase.Atoms"),
     # The workflow CLI exposes argparse's private parser-action type.
     ("py:class", "argparse._SubParsersAction"),
+    # pymatgen is an optional dependency surfaced by the integrations layer.
+    ("py:class", "pymatgen.core.Structure"),
+    ("py:obj", "pymatgen.core.Structure"),
+    # AutoAPI renders this TYPE_CHECKING-only record annotation without its
+    # module qualification; the concrete record remains documented in the
+    # storage records module.
+    ("py:class", "NormalizedCompositionRecord"),
     # SQLAlchemy is optional, and these internal-facing signatures have no vendored
     # external inventory.
     ("py:class", "sqlalchemy.Engine"),
@@ -163,6 +170,13 @@
     ("py:class", "FilterAst"),
     ("py:class", "_Context"),
     ("py:class", "_BackingPlan"),
+    # PyMongo publishes no usable intersphinx target for its client class; this
+    # targeted ignore follows the sanctioned external-type precedent in httk-core.
+    ("py:class", "pymongo.MongoClient"),
+    # The parallel bulk-ingest merge entry points (bulk_parallel.merge and
+    # ParallelController) carry the module-private worker-manifest type in their
+    # signatures; AutoAPI intentionally omits that implementation dataclass.
+    ("py:class", "_WorkerManifest"),
     # StoredEntrySource is lazily re-exported from httk.data.db; AutoAPI keeps
     # the public annotation but indexes its defining module instead.
     ("py:class", "httk.data.db.StoredEntrySource"),
@@ -193,19 +207,18 @@
     "adapter_runtime",
     "cli",
     "workflow_cli",
-    # Compatibility internals: the engines are public, their runners and the
-    # v1 CLI alias and shared import tail are not.
-    "compat._integration",
-    "compat.cwl.cwl_runner",
-    "compat.pwd.pwd_runner",
-    "compat.v1._runner",
-    "compat.v1.cli",
+    # Language runners are internal; the language registration modules are
+    # public.
+    "languages.cwl.cwl_runner",
+    "languages.pwd.pwd_runner",
+    "languages.jobflow.jobflow_runner",
+    "languages.httk_v1.v1_runner",
     # The VASP facade is public; the cohesive modules it re-exports are not.
     "vasp.inputs",
     "vasp.diagnostics",
     "vasp.remedies",
     "vasp.reports",
-    "vasp.templates",
+    "vasp.workflows",
 )
 nitpick_ignore_regex = [
     # AutoAPI renders these imported helper names as bare _common.* targets in
@@ -255,10 +268,12 @@
         "httk.workflow.sdk",
         "httk.workflow.runtime",
         "httk.workflow.runtime_utils",
+        "httk.workflow.hookapi",
         "httk.workflow.scaffold",
-        "httk.workflow.backends",
+        "httk.workflow.executors",
         "httk.workflow.shell_bridge",
-        "httk.workflow.harvesting",
+        "httk.workflow.collecting",
+        "httk.workflow.provenance",
         "httk.workflow.supervision",
         "httk.workflow.transfers",
         "httk.workflow.manifests",
@@ -270,8 +285,11 @@
         "httk.workflow.vasp",
         "httk.workflow.compat",
         "httk.workflow.compat.v1",
-        "httk.workflow.compat.cwl",
-        "httk.workflow.compat.pwd",
+        "httk.workflow.languages",
+        "httk.workflow.languages.cwl",
+        "httk.workflow.languages.pwd",
+        "httk.workflow.languages.jobflow",
+        "httk.workflow.languages.httk_v1",
     }
 )
 _workflow_exports_cache: dict[str, frozenset[str] | None] = {}

docs/data.md

@@ -0,0 +1,60 @@
+# Storing, querying, and serving data
+
+*httk₂* keeps data models separate from storage. Plain frozen dataclasses can
+be stored in SQLite or DuckDB through `SqlStore`, or in MongoDB through
+`MongoStore`; the same records and neutral query protocols travel across those
+backends. Content addressing deduplicates equal records while a local `sid`
+identifies a row in one store.
+
+```python
+from dataclasses import dataclass
+from tempfile import TemporaryDirectory
+
+from httk.data.db import Database, SqlStore
+
+@dataclass(frozen=True)
+class Result:
+    formula: str
+
+record = Result("NaCl")
+with TemporaryDirectory() as directory:
+    db = Database.sqlite(f"{directory}/results.sqlite")
+    store = SqlStore(db, entry_records={})
+    with store.transaction():
+        sid = store.save(record)
+    assert store.fetch(type(record), sid) == record
+```
+
+The search DSL binds a record class to a variable, adds comparisons or
+collection predicates such as `has`, `has_any`, and `has_only`, then returns a
+lazy result set. `bulk_ingest(workers=N)` is the faster path for building a
+large store; use ordinary `save()` for a small increment.
+
+MongoDB uses the same model and store surface when MongoDB is already the
+operational data service:
+
+```python
+from httk.data.mongo import MongoDatabase, MongoStore
+
+with MongoDatabase.connect(uri, database="materials") as database:
+    store = MongoStore(database, entry_records={})
+```
+
+Federation presents existing stores as one read-only, source-major union. A
+provider turns a store or in-memory records into the neutral entry-provider
+contract, and *httk-serve* can expose one or more providers through OPTIMADE:
+
+```python
+from httk.serve.optimade import adapter_from_providers, serve
+
+serve(adapter_from_providers([provider]), port=8080)
+```
+
+Construct `adapter_from_providers([provider])` first when testing or embedding the adapter; `serve(...)` is the quick development-server path.
+For deployment, use `create_asgi_app` as the interface to any ASGI server.
+
+## Read next
+
+- [Data management](https://docs.httk.org/httk-data/dev/main/data/), [database storage](https://docs.httk.org/httk-data/dev/main/db/), [MongoDB](https://docs.httk.org/httk-data/dev/main/mongo/), and [federation](https://docs.httk.org/httk-data/dev/main/federation/).
+- [Database details](https://docs.httk.org/httk-data/dev/main/details/db/) and [MongoDB details](https://docs.httk.org/httk-data/dev/main/details/mongo/).
+- [OPTIMADE serving](https://docs.httk.org/httk-serve/dev/main/optimade/serving_providers/) and [core definition details](https://docs.httk.org/httk-core/dev/main/details/optimade_definitions/).

docs/index.md

@@ -20,13 +20,19 @@ publishes its own subsite under [docs.httk.org](https://docs.httk.org); the
 ```{admonition} Quick links
 :class: tip
 
+- **Working with httk₂**:
+  {doc}`structures`, {doc}`data`, {doc}`campaigns`, and {doc}`analysis` — the
+  ecosystem-level path from input files to results and analysis.
 - **Module directory**: {doc}`modules` — every *httk₂* module and where its docs live.
 - **API reference**: {doc}`reference/index` — the aggregate reference for the published modules on this site.
 - **Tutorial**: {doc}`tutorial/index` — the original short
   example sequence translated to current APIs, including the remaining gaps.
 - **Example notebooks**: {doc}`notebooks/index` — runnable tours of the core and cross-module APIs.
 ```
 
+The topic pages are short and practical; they link onward to the module
+documentation for the complete guides and API details.
+
 ## Install
 
 Preferably work in a Python virtual environment.
@@ -80,6 +86,10 @@ The public namespace is split across independently installable distributions:
 
 modules
 architecture
+structures
+data
+campaigns
+analysis
 authoring
 tutorial/index
 reference/index

docs/modules.md

@@ -12,6 +12,7 @@ Shared contracts, exact vector and datastream models, OPTIMADE definitions,
 registries, and dependency-free record models. Import path: `httk.core`.
 
 - API reference: {doc}`httk.core </reference/autoapi/httk/core/index>`
+- Module documentation: <https://docs.httk.org/httk-core/dev/main/>
 - Repository: <https://github.com/httk/httk-core>
 
 ### httk-atomistic
@@ -20,6 +21,7 @@ Exact crystal-structure and symmetry representations. Import path:
 `httk.atomistic`.
 
 - API reference: {doc}`httk.atomistic </reference/autoapi/httk/atomistic/index>`
+- Module documentation: <https://docs.httk.org/httk-atomistic/dev/main/>
 - Repository: <https://github.com/httk/httk-atomistic>
 
 ### httk-analyse
@@ -28,6 +30,7 @@ Generic lower-convex-hull construction and materials-science phase-diagram
 analysis. Import path: `httk.analyse`.
 
 - API reference: {doc}`httk.analyse </reference/autoapi/httk/analyse/index>`
+- Module documentation: <https://docs.httk.org/httk-analyse/dev/main/>
 - Repository: <https://github.com/httk/httk-analyse>
 
 ### httk-io
@@ -36,6 +39,7 @@ File-format parsing and writing capabilities, including CIF and POSCAR. Import
 path: `httk.io`.
 
 - API reference: {doc}`httk.io </reference/autoapi/httk/io/index>`
+- Module documentation: <https://docs.httk.org/httk-io/dev/main/>
 - Repository: <https://github.com/httk/httk-io>
 
 ### httk-data
@@ -44,6 +48,7 @@ Data-management capabilities and validation over core's provider and definition
 contracts. Import path: `httk.data`.
 
 - API reference: {doc}`httk.data </reference/autoapi/httk/data/index>`
+- Module documentation: <https://docs.httk.org/httk-data/dev/main/>
 - Repository: <https://github.com/httk/httk-data>
 
 ### httk-serve
@@ -52,6 +57,7 @@ Web-facing application and OPTIMADE protocol serving capabilities. Import paths:
 `httk.serve.web` and `httk.serve.optimade`.
 
 - API reference: {doc}`httk.serve </reference/autoapi/httk/serve/index>`
+- Module documentation: <https://docs.httk.org/httk-serve/dev/main/>
 - Repository: <https://github.com/httk/httk-serve>
 
 ### httk-workflow
@@ -60,6 +66,7 @@ Workflow and calculation orchestration capabilities. Import path:
 `httk.workflow`.
 
 - API reference: {doc}`httk.workflow </reference/autoapi/httk/workflow/index>`
+- Module documentation: <https://docs.httk.org/httk-workflow/dev/main/>
 - Repository: <https://github.com/httk/httk-workflow>
 
 ## Selected snapshot versions