docs: cover readiness, verification, and recursive discovery

docs/commands.md

@@ -6,25 +6,29 @@
 
 The commands available to you are:
 
-- `clear`: This removes all files belonging to the 'scope' and 'dataset', only
-  available for the local and canfar sites.
-- `config`: Edit the `.datatrail/config.yaml` configuration file.
-- `inventory`: Recursively discover datasets and write their file replica URIs
-  to a resumable JSON manifest.
-- `list`: This list either the 'scopes' available or all of the datasets
-  belonging to the given dataset.
-- `ps`: This provides detailed information for the given 'scope' and 'dataset' combination.
-- `pull`: This allows you to download all files belonging to the 'scope' and
-  'dataset' provided.
-- `pull-manifest`: Download Minoc files from a resumable inventory manifest.
-- `scout`: This command provides an overview of what the Datatrail database
-  thinks is the current number of files for a given dataset at each storage
-  element, compared to what is observed. If a discrepancy is found at Minoc,
-  the user can choose to create the file replicas missing for Minoc.
-- `unregistered`: This provides insight into datasets which failed to register
-  with Datatrail, either summarised across the whole unregistered bucket
-  (`summary`) or for a single event (`search`).
-- `version`: List the CLI and server version.
+- [`clear`](clear.md): Remove a dataset's files at the local or CANFAR site.
+- [`config`](cli.md#datatrail-config): Initialize, inspect, or edit the CLI
+  configuration.
+- [`doctor`](doctor.md): Check configuration, server health, certificate validity,
+  and authentication to Minoc and Luskan.
+- [`inventory`](inventory.md): Recursively discover datasets and write their
+  file replica URIs to a resumable JSON manifest.
+- [`list` / `ls`](list.md): Browse scopes and datasets, filter with `--match`,
+  or discover children with `--expand` and `--recursive`.
+- [`ps`](ps.md): Inspect a dataset's files and storage locations. JSON output
+  includes common paths for each storage element.
+- [`pull`](pull.md): Download a dataset's files.
+- [`pull-manifest`](pull-manifest.md): Download Minoc files from an inventory
+  manifest with resumable transfer state.
+- [`scout`](scout.md): Compare registered and observed file counts and, after
+  confirmation, register missing Minoc replicas.
+- [`unregistered`](unregistered.md): Summarize registration failures or search
+  for records of a specific event.
+- [`verify`](verify.md): Compare registered Minoc files with Minoc and Luskan
+  metadata, reporting missing files, size or checksum mismatches, and
+  unavailable metadata.
+- [`version`](cli.md#datatrail-version): Show CLI and server version information.
 
-Detailed information on all of the CLI commands can be found on the
-[Reference](cli.md) page.
+For practical examples, start with the [User Guide](user_guide.md) or the
+[recursive discovery walkthrough](discovery.md). The
+[Reference](cli.md) lists every command and its options.

docs/discovery.md

@@ -0,0 +1,73 @@
+# Discover datasets recursively
+
+Use `datatrail list` (or its alias `datatrail ls`) when you know part of a
+dataset name but not where it sits in the hierarchy. Choose how far to explore:
+
+- `--match`: Select larger datasets using comma-separated, case-insensitive
+  terms. Every term must appear in the combined scope and dataset name.
+- `--expand`: Open each selected larger dataset one level and list its children.
+- `--recursive`: Follow descendants to terminal datasets and record the path
+  to each result.
+
+## Start with a bounded search
+
+For example, find gain datasets within one scope:
+
+```shell
+datatrail ls gbo.acquisition.processed --match gains
+```
+
+To inspect one level of children, add `--expand`:
+
+```shell
+datatrail ls gbo.acquisition.processed --match gains --expand
+```
+
+To continue through nested datasets, use `--recursive` instead:
+
+```shell
+datatrail ls gbo.acquisition.processed --match gains --recursive
+```
+
+You can omit the scope to search across scopes, but `--expand` and `--recursive`
+then require `--match` to bound the search. Matching selects the starting larger
+datasets; descendants are followed even if their own names do not contain the
+search terms.
+
+The recursive walk visits each dataset once, so shared descendants are not
+repeated and cycles cannot loop forever. It retains the first path found in
+sorted order. A dataset with an answered empty child list is terminal.
+
+## Save discovery results for a script
+
+```shell
+datatrail ls gbo.acquisition.processed --match gains --recursive --json > datasets.json
+```
+
+The JSON map contains `results` and `failed`. Each recursive result records its
+`scope`, `dataset`, `parent`, and `path`. See the [list guide](list.md) for output
+examples and ordinary scope or child-dataset queries. A top-level failure can
+return an `error` object instead of a map and exits `1`, so check the process
+status before reading the result fields.
+
+!!! warning "Check for incomplete discovery"
+
+    An unanswered branch is retained as a partial row and recorded in `failed`.
+    A partial map with rows still exits `0`; no rows plus unanswered queries exits
+    `1`. Scripts that require complete results must also check that `failed` is
+    empty.
+
+## Turn discovery into a resumable download
+
+`list --recursive` maps datasets. Use [`inventory`](inventory.md) when you also
+need their file replica URIs saved in a durable manifest:
+
+```shell
+datatrail inventory gbo.acquisition.processed --match gains --output gains-inventory.json
+datatrail pull-manifest gains-inventory.json --directory ./gains --cores 4
+```
+
+An inventory can resume unfinished queries. [`pull-manifest`](pull-manifest.md)
+tracks completed transfers in a separate state file so a later run can continue
+without repeating completed downloads. Read both guides for incomplete results,
+state-file ownership, and transfer behavior.

docs/doctor.md

@@ -0,0 +1,55 @@
+# Check readiness with `doctor`
+
+Use `doctor` when a command cannot connect, authentication fails, or you want to
+check your setup before downloading data:
+
+```bash
+datatrail doctor
+```
+
+The command reads your existing CLI configuration and certificate, then checks
+Datatrail, Minoc, and Luskan. See [initial setup](initialising.md) if you have not
+configured the CLI or obtained a CADC certificate.
+
+## Understand the checks
+
+The text report prints one line per check, labeled `OK` or `FAILED`:
+
+| Check | What must pass |
+| --- | --- |
+| `config` | The configuration loads and contains a server URL, certificate path, site, and root mount for that site. |
+| `server` | The configured Datatrail server returns a valid, healthy `/health/check` report, including database and API status. |
+| `certificate` | The configured certificate is readable PEM and its validity period includes the current time. |
+| `minoc` | Minoc's capabilities endpoint responds successfully and confirms authentication with the certificate. |
+| `luskan` | Luskan's capabilities endpoint responds successfully and confirms authentication with the certificate. |
+
+If configuration fails, the remaining readiness checks are skipped. If the
+certificate fails, the Minoc and Luskan checks are skipped. Skipped checks appear
+as `FAILED` with a message explaining why they were not checked; they do not
+independently establish that a service is down.
+
+A successful report confirms these readiness checks at that moment. It does not
+prove permission to read every dataset, verify file contents, or continuously
+monitor registration workers. The command does not change configuration, renew
+certificates, or download datasets.
+
+## Use the result in a script
+
+```bash
+datatrail doctor --json > doctor.json
+```
+
+`--json` replaces the text report with a JSON object on standard output. It has a
+top-level `ok` boolean and a `checks` object containing `config`, `server`,
+`certificate`, `minoc`, and `luskan`. Each check contains its own `ok` boolean and
+`message`. The report is still written when a check fails.
+
+| Exit status | Meaning |
+| --- | --- |
+| `0` | Every readiness check passed. |
+| `1` | At least one readiness check failed or was skipped. |
+
+Address the first relevant failure: review [configuration and certificate setup](initialising.md)
+for local problems, or share the failed check and message with the service operator.
+Run `doctor` again after addressing the cause. For dataset metadata checks, use
+[`verify`](verify.md).

docs/index.md

@@ -41,6 +41,10 @@ In order to fully utilise this CLI, you must have an account with
 [CANFAR](https://www.canfar.net) and access to the either of the following
 groups: `chime-frb-ro` or `chime-frb-rw`.
 
+Check your setup with [`doctor`](doctor.md). The [User Guide](user_guide.md)
+covers [recursive dataset discovery](discovery.md), resumable inventory and downloads,
+[`verify`](verify.md), and registration troubleshooting.
+
 ## 🛠️ [Installation](install.md)
 
 ## ⚙️  [Initialise](initialising.md)

docs/initialising.md

@@ -56,13 +56,13 @@ one of the outrigger sites. See below for a guide for each of the sites.
         of 30 days. You must refresh the certificate periodically.
 
     If you do not keep your CADC certificate in the default location, you must
-    update the configuration file to pointt to the correct location.
+    update the configuration file to point to the correct location.
 
     ```shell
     # Updating CADC Certificate location
-    $> datatrail config set vospace_certificate /non/standard/location/cadcproxy.pem
-    Attempting to set vospace_certificate to /non/standard/location/cadcproxy.pem
-    Set vospace_certificate to /non/standard/location/cadcproxy.pem
+    $> datatrail config set vospace_certfile /non/standard/location/cadcproxy.pem
+    Attempting to set vospace_certfile to /non/standard/location/cadcproxy.pem
+    Set vospace_certfile to /non/standard/location/cadcproxy.pem
     ```
 
 === "CANFAR"
@@ -123,13 +123,13 @@ one of the outrigger sites. See below for a guide for each of the sites.
         of 30 days. You must refresh the certificate periodically.
 
     If you do not keep your CADC certificate in the default location, you must
-    update the configuration file to pointt to the correct location.
+    update the configuration file to point to the correct location.
 
     ```shell
     # Updating CADC Certificate location
-    $> datatrail config set vospace_certificate /non/standard/location/cadcproxy.pem
-    Attempting to set vospace_certificate to /non/standard/location/cadcproxy.pem
-    Set vospace_certificate to /non/standard/location/cadcproxy.pem
+    $> datatrail config set vospace_certfile /non/standard/location/cadcproxy.pem
+    Attempting to set vospace_certfile to /non/standard/location/cadcproxy.pem
+    Set vospace_certfile to /non/standard/location/cadcproxy.pem
     ```
 
 === "CHIME"
@@ -171,3 +171,15 @@ one of the outrigger sites. See below for a guide for each of the sites.
     # Ensure valid CADC Certificate exists
     cadc-get-cert -u [username]
     ```
+
+## Check your setup
+
+After initializing the configuration and obtaining your certificate, run
+[`doctor`](doctor.md) to check configuration, server health, certificate validity,
+and authentication to Minoc and Luskan:
+
+```shell
+datatrail doctor
+```
+
+The [doctor guide](doctor.md) explains failed checks and JSON output for scripts.

docs/user_guide.md

@@ -4,30 +4,55 @@
 
 <h1 align="center">User Guide</h1>
 
-## 🛠️ [Installation](install.md)
-
-How to install Datatrail CLI.
-
-## ⚙️  [Initialise](initialising.md)
-
-Performing the initial setup in order to use the Datatrail CLI.
-
-## 🗑️ [clear](clear.md)
-
-Deleting a dataset.
-
-## 🗒️ [list](list.md)
-
-Searching the Datatrail database for scopes and datasets.
-
-## 🔍 [ps](ps.md)
-
-Querying the Datatrail database for details about a dataset.
-
-## ⬇️  [pull](pull.md)
-
-Downloading a dataset.
-
-## 🕵️  [scout](scout.md)
-
-Investigating number of files for a dataset across storage elements.
+## Set up and check your connection
+
+[Install the CLI](install.md), then [initialize its configuration](initialising.md).
+Use [`doctor`](doctor.md) to check your configuration, Datatrail server health,
+CANFAR certificate, and authentication to Minoc and Luskan before working with
+files. The [`config` reference](cli.md#datatrail-config) covers inspecting and
+editing configuration values.
+
+## Find and inspect datasets
+
+Start with [Recursive discovery](discovery.md) for a walkthrough of `--match`,
+`--expand`, and `--recursive`, followed by a resumable inventory and download.
+
+- [`list` / `ls`](list.md): Browse scopes and datasets. Use `--match` to narrow
+  a search, `--expand` to see one level of children, and `--recursive` to follow
+  descendants to terminal datasets.
+- [`ps`](ps.md): Inspect a dataset's files and storage locations, including
+  per-storage-element common paths in JSON output.
+- [`inventory`](inventory.md): Save recursive discovery and file replica URIs
+  in a JSON manifest that can be resumed after an interruption.
+
+## Download and check files
+
+For a large collection, use [`inventory`](inventory.md) to save the discovered
+files, then pass that manifest to [`pull-manifest`](pull-manifest.md) to download
+them with resumable progress.
+
+- [`pull`](pull.md): Download the files in a dataset.
+- [`pull-manifest`](pull-manifest.md): Download Minoc files from an inventory
+  with saved transfer progress and bounded concurrency.
+- [`verify`](verify.md): Compare registered Minoc files with Minoc and Luskan
+  metadata to identify missing files, size or checksum mismatches, and
+  unavailable metadata.
+- [`clear`](clear.md): Remove a dataset's local or CANFAR files.
+
+## Investigate registration problems
+
+- [`scout`](scout.md): Compare registered and observed file counts and review
+  discrepancies before confirming repairs.
+- [`unregistered`](unregistered.md): Summarize registration failures or inspect
+  records for a specific event, with optional scope and partial-name filters.
+
+## Use results in scripts
+
+`list`, `ps`, `doctor`, `verify`, and `unregistered search` support `--json`.
+Their guides explain each output format. `inventory` writes a JSON manifest,
+while `pull-manifest` saves a separate JSON transfer-state file. Consult the
+relevant command's guide for exit status and error handling.
+
+The [command overview](commands.md) links to every command, including
+[`version`](cli.md#datatrail-version). The [CLI reference](cli.md) lists all
+arguments and options.

docs/verify.md

@@ -0,0 +1,66 @@
+# Compare registered Minoc files with `verify`
+
+Use `verify` to check whether the Minoc files recorded by Datatrail have consistent
+metadata in Minoc and the Luskan inventory. Datatrail supplies the file list;
+`verify` compares the sizes and MD5 checksums reported by the two CADC services.
+
+```bash
+datatrail verify chime.event.baseband.raw EVENT_ID
+```
+
+Replace the scope and `EVENT_ID` with your dataset's actual identifiers. The CLI
+needs its normal configuration and a valid CADC certificate with service access.
+Use [`doctor`](doctor.md) to troubleshoot configuration or authentication first.
+
+## Read the results
+
+The report starts with the scope, dataset, and `registered` count: the number of
+unique Minoc file URIs returned by Datatrail. It then shows these result counts
+and lists the URIs with problems:
+
+| Result | Meaning |
+| --- | --- |
+| `present` | Both services returned complete, matching size and checksum metadata. |
+| `missing` | A successful metadata lookup did not return the registered URI from Minoc, Luskan, or both. |
+| `size-mismatch` | Both sizes are available, but their byte counts differ. |
+| `checksum-mismatch` | Both MD5 values are available, but they differ. |
+| `unavailable` | A service request failed, a response was invalid, or required metadata was incomplete. The comparison could not be completed. |
+
+One file can have more than one problem, so problem counts need not add up to
+`registered`. An unavailable lookup is not reported as proof that a file is
+missing.
+
+!!! note "An empty check can succeed"
+
+    If Datatrail returns no registered Minoc files, the command reports
+    `registered: 0` and succeeds. This does not establish that the dataset has
+    been replicated. Inspect its recorded locations with [`ps`](ps.md).
+
+This command reads metadata; it does not download files, recompute checksums from
+their contents, or repair registrations. It checks only the registered Minoc
+URIs supplied by Datatrail, so it does not discover unregistered files or verify
+other storage elements. Matching service metadata is not a fresh integrity check
+of the stored bytes.
+
+## Save a detailed report
+
+```bash
+datatrail verify chime.event.baseband.raw EVENT_ID --json > verification.json
+```
+
+`--json` writes a JSON object on standard output, including on verification
+failure. It contains `scope`, `dataset`, `registered`, `ok`, `summary`, and
+`results`. The category keys use underscores (`size_mismatch` and
+`checksum_mismatch`). Detailed results identify the URI, affected services, or
+the differing Minoc and Luskan values as appropriate.
+
+| Exit status | Meaning |
+| --- | --- |
+| `0` | No missing files, mismatches, or unavailable metadata were reported; this includes an empty registered file list. |
+| `1` | Missing files or metadata mismatches were found, with no unavailable results. |
+| `2` | At least one result was unavailable, even if other files also had mismatches. |
+
+For unavailable results, check service readiness and retry after the underlying
+problem is resolved. For missing files or mismatches, retain the detailed report
+and investigate the dataset with [`ps`](ps.md) and [`scout`](scout.md) before
+requesting corrective work.