diff --git a/docs/commands.md b/docs/commands.md index 9104b3d..ae79d1a 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -6,25 +6,29 @@ The commands available to you are: -- `clear`: This removes all files belonging to the 'scope' and 'dataset', only - available for the local and canfar sites. -- `config`: Edit the `.datatrail/config.yaml` configuration file. -- `inventory`: Recursively discover datasets and write their file replica URIs - to a resumable JSON manifest. -- `list`: This list either the 'scopes' available or all of the datasets - belonging to the given dataset. -- `ps`: This provides detailed information for the given 'scope' and 'dataset' combination. -- `pull`: This allows you to download all files belonging to the 'scope' and - 'dataset' provided. -- `pull-manifest`: Download Minoc files from a resumable inventory manifest. -- `scout`: This command provides an overview of what the Datatrail database - thinks is the current number of files for a given dataset at each storage - element, compared to what is observed. If a discrepancy is found at Minoc, - the user can choose to create the file replicas missing for Minoc. -- `unregistered`: This provides insight into datasets which failed to register - with Datatrail, either summarised across the whole unregistered bucket - (`summary`) or for a single event (`search`). -- `version`: List the CLI and server version. +- [`clear`](clear.md): Remove a dataset's files at the local or CANFAR site. +- [`config`](cli.md#datatrail-config): Initialize, inspect, or edit the CLI + configuration. +- [`doctor`](doctor.md): Check configuration, server health, certificate validity, + and authentication to Minoc and Luskan. +- [`inventory`](inventory.md): Recursively discover datasets and write their + file replica URIs to a resumable JSON manifest. +- [`list` / `ls`](list.md): Browse scopes and datasets, filter with `--match`, + or discover children with `--expand` and `--recursive`. +- [`ps`](ps.md): Inspect a dataset's files and storage locations. JSON output + includes common paths for each storage element. +- [`pull`](pull.md): Download a dataset's files. +- [`pull-manifest`](pull-manifest.md): Download Minoc files from an inventory + manifest with resumable transfer state. +- [`scout`](scout.md): Compare registered and observed file counts and, after + confirmation, register missing Minoc replicas. +- [`unregistered`](unregistered.md): Summarize registration failures or search + for records of a specific event. +- [`verify`](verify.md): Compare registered Minoc files with Minoc and Luskan + metadata, reporting missing files, size or checksum mismatches, and + unavailable metadata. +- [`version`](cli.md#datatrail-version): Show CLI and server version information. -Detailed information on all of the CLI commands can be found on the -[Reference](cli.md) page. +For practical examples, start with the [User Guide](user_guide.md) or the +[recursive discovery walkthrough](discovery.md). The +[Reference](cli.md) lists every command and its options. diff --git a/docs/discovery.md b/docs/discovery.md new file mode 100644 index 0000000..678282f --- /dev/null +++ b/docs/discovery.md @@ -0,0 +1,73 @@ +# Discover datasets recursively + +Use `datatrail list` (or its alias `datatrail ls`) when you know part of a +dataset name but not where it sits in the hierarchy. Choose how far to explore: + +- `--match`: Select larger datasets using comma-separated, case-insensitive + terms. Every term must appear in the combined scope and dataset name. +- `--expand`: Open each selected larger dataset one level and list its children. +- `--recursive`: Follow descendants to terminal datasets and record the path + to each result. + +## Start with a bounded search + +For example, find gain datasets within one scope: + +```shell +datatrail ls gbo.acquisition.processed --match gains +``` + +To inspect one level of children, add `--expand`: + +```shell +datatrail ls gbo.acquisition.processed --match gains --expand +``` + +To continue through nested datasets, use `--recursive` instead: + +```shell +datatrail ls gbo.acquisition.processed --match gains --recursive +``` + +You can omit the scope to search across scopes, but `--expand` and `--recursive` +then require `--match` to bound the search. Matching selects the starting larger +datasets; descendants are followed even if their own names do not contain the +search terms. + +The recursive walk visits each dataset once, so shared descendants are not +repeated and cycles cannot loop forever. It retains the first path found in +sorted order. A dataset with an answered empty child list is terminal. + +## Save discovery results for a script + +```shell +datatrail ls gbo.acquisition.processed --match gains --recursive --json > datasets.json +``` + +The JSON map contains `results` and `failed`. Each recursive result records its +`scope`, `dataset`, `parent`, and `path`. See the [list guide](list.md) for output +examples and ordinary scope or child-dataset queries. A top-level failure can +return an `error` object instead of a map and exits `1`, so check the process +status before reading the result fields. + +!!! warning "Check for incomplete discovery" + + An unanswered branch is retained as a partial row and recorded in `failed`. + A partial map with rows still exits `0`; no rows plus unanswered queries exits + `1`. Scripts that require complete results must also check that `failed` is + empty. + +## Turn discovery into a resumable download + +`list --recursive` maps datasets. Use [`inventory`](inventory.md) when you also +need their file replica URIs saved in a durable manifest: + +```shell +datatrail inventory gbo.acquisition.processed --match gains --output gains-inventory.json +datatrail pull-manifest gains-inventory.json --directory ./gains --cores 4 +``` + +An inventory can resume unfinished queries. [`pull-manifest`](pull-manifest.md) +tracks completed transfers in a separate state file so a later run can continue +without repeating completed downloads. Read both guides for incomplete results, +state-file ownership, and transfer behavior. diff --git a/docs/doctor.md b/docs/doctor.md new file mode 100644 index 0000000..607fa24 --- /dev/null +++ b/docs/doctor.md @@ -0,0 +1,55 @@ +# Check readiness with `doctor` + +Use `doctor` when a command cannot connect, authentication fails, or you want to +check your setup before downloading data: + +```bash +datatrail doctor +``` + +The command reads your existing CLI configuration and certificate, then checks +Datatrail, Minoc, and Luskan. See [initial setup](initialising.md) if you have not +configured the CLI or obtained a CADC certificate. + +## Understand the checks + +The text report prints one line per check, labeled `OK` or `FAILED`: + +| Check | What must pass | +| --- | --- | +| `config` | The configuration loads and contains a server URL, certificate path, site, and root mount for that site. | +| `server` | The configured Datatrail server returns a valid, healthy `/health/check` report, including database and API status. | +| `certificate` | The configured certificate is readable PEM and its validity period includes the current time. | +| `minoc` | Minoc's capabilities endpoint responds successfully and confirms authentication with the certificate. | +| `luskan` | Luskan's capabilities endpoint responds successfully and confirms authentication with the certificate. | + +If configuration fails, the remaining readiness checks are skipped. If the +certificate fails, the Minoc and Luskan checks are skipped. Skipped checks appear +as `FAILED` with a message explaining why they were not checked; they do not +independently establish that a service is down. + +A successful report confirms these readiness checks at that moment. It does not +prove permission to read every dataset, verify file contents, or continuously +monitor registration workers. The command does not change configuration, renew +certificates, or download datasets. + +## Use the result in a script + +```bash +datatrail doctor --json > doctor.json +``` + +`--json` replaces the text report with a JSON object on standard output. It has a +top-level `ok` boolean and a `checks` object containing `config`, `server`, +`certificate`, `minoc`, and `luskan`. Each check contains its own `ok` boolean and +`message`. The report is still written when a check fails. + +| Exit status | Meaning | +| --- | --- | +| `0` | Every readiness check passed. | +| `1` | At least one readiness check failed or was skipped. | + +Address the first relevant failure: review [configuration and certificate setup](initialising.md) +for local problems, or share the failed check and message with the service operator. +Run `doctor` again after addressing the cause. For dataset metadata checks, use +[`verify`](verify.md). diff --git a/docs/index.md b/docs/index.md index 018a52a..913d3dd 100644 --- a/docs/index.md +++ b/docs/index.md @@ -41,6 +41,10 @@ In order to fully utilise this CLI, you must have an account with [CANFAR](https://www.canfar.net) and access to the either of the following groups: `chime-frb-ro` or `chime-frb-rw`. +Check your setup with [`doctor`](doctor.md). The [User Guide](user_guide.md) +covers [recursive dataset discovery](discovery.md), resumable inventory and downloads, +[`verify`](verify.md), and registration troubleshooting. + ## 🛠️ [Installation](install.md) ## ⚙️ [Initialise](initialising.md) diff --git a/docs/initialising.md b/docs/initialising.md index 372d8d2..a926dc3 100644 --- a/docs/initialising.md +++ b/docs/initialising.md @@ -56,13 +56,13 @@ one of the outrigger sites. See below for a guide for each of the sites. of 30 days. You must refresh the certificate periodically. If you do not keep your CADC certificate in the default location, you must - update the configuration file to pointt to the correct location. + update the configuration file to point to the correct location. ```shell # Updating CADC Certificate location - $> datatrail config set vospace_certificate /non/standard/location/cadcproxy.pem - Attempting to set vospace_certificate to /non/standard/location/cadcproxy.pem - Set vospace_certificate to /non/standard/location/cadcproxy.pem + $> datatrail config set vospace_certfile /non/standard/location/cadcproxy.pem + Attempting to set vospace_certfile to /non/standard/location/cadcproxy.pem + Set vospace_certfile to /non/standard/location/cadcproxy.pem ``` === "CANFAR" @@ -123,13 +123,13 @@ one of the outrigger sites. See below for a guide for each of the sites. of 30 days. You must refresh the certificate periodically. If you do not keep your CADC certificate in the default location, you must - update the configuration file to pointt to the correct location. + update the configuration file to point to the correct location. ```shell # Updating CADC Certificate location - $> datatrail config set vospace_certificate /non/standard/location/cadcproxy.pem - Attempting to set vospace_certificate to /non/standard/location/cadcproxy.pem - Set vospace_certificate to /non/standard/location/cadcproxy.pem + $> datatrail config set vospace_certfile /non/standard/location/cadcproxy.pem + Attempting to set vospace_certfile to /non/standard/location/cadcproxy.pem + Set vospace_certfile to /non/standard/location/cadcproxy.pem ``` === "CHIME" @@ -171,3 +171,15 @@ one of the outrigger sites. See below for a guide for each of the sites. # Ensure valid CADC Certificate exists cadc-get-cert -u [username] ``` + +## Check your setup + +After initializing the configuration and obtaining your certificate, run +[`doctor`](doctor.md) to check configuration, server health, certificate validity, +and authentication to Minoc and Luskan: + +```shell +datatrail doctor +``` + +The [doctor guide](doctor.md) explains failed checks and JSON output for scripts. diff --git a/docs/user_guide.md b/docs/user_guide.md index ddb6d9b..84f9cea 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -4,30 +4,55 @@