Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 24 additions & 21 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,40 +4,43 @@ Instructions for AI coding agents (OpenAI Codex, Cursor, Gemini CLI, and any too

## Prerequisites

1. `apimetrics project show` — confirm a project is active. Every project-scoped command fails without one. If none is set, run `apimetrics project select`.
2. `apimetrics login` handles authentication (OAuth). **There is no `--api-key` flag.**
1. `apimetrics project show` — confirm a project is active. Every project-scoped command fails without one unless you pass `--project-id <id>`. If none is set, run `apimetrics project select`.
2. Commands log in automatically when needed; `apimetrics login` does it up front (OAuth). For headless use, pass `--service-account <file>`.

## Core facts about this CLI

- **Commands are flat, generated from the platform's live OpenAPI description**: `list-calls`, `create-call`, `run-call` — not `calls create`. The set can change with the server spec, so treat `apimetrics --help` and `apimetrics <command> --help` as authoritative and inspect a command's help before assuming an option or body field.
- **Create/update commands read JSON from stdin** via heredoc. There is no `--body`, `--data`, or `-d`:
- **Commands are flat, generated from the platform's live OpenAPI description**: `list-calls`, `create-call`, `run-monitor`. The set can change with the server spec, so treat `apimetrics --help` and `apimetrics <command> --help` as authoritative and inspect a command's help before assuming an option or body field. `run-monitor <id>` is the single command that starts a run for any monitor type — API call, browser, or MCP; it requires a body, so send at least `{}` on stdin, and returns the `result_id` to poll.
- **Deprecated commands still run but warn.** Where the server's spec deprecates a command, the CLI hides it from `--help` and prints `Command "X" is deprecated, do not use` on stderr (stdout JSON is unaffected). On servers that expose the `/api/3/monitors/` commands, `list-calls`/`create-call`/`get-call`/`update-call`/`delete-call` are deprecated in favour of `list-api-monitors` (or `list-monitors` for every type), `create-api-monitor`, `get-api-monitor`, `patch-api-monitor`/`update-api-monitor`, and `delete-api-monitor`. The new create/update commands take a flat body (`name`, `method`, `url`, `headers`, …) rather than `meta`/`request`; monitor IDs are the same, so `set-call-conditions`, `get-call-passfail-*`, and `list-results-by-call` keep working. Use whichever set your `--help` lists.
- **Create/update commands read their JSON body from stdin** via heredoc:
```bash
apimetrics create-call <<'EOF'
{ "meta": { "name": "Health" }, "request": { "method": "GET", "url": "https://example.com/health" } }
EOF
```
CLI Shorthand is also accepted (`apimetrics create-call meta.name: "Health", request.method: GET, request.url: https://example.com/health`).
- **Output**: `-o json` for machine use; `-f` projects with a shorthand query; `-q` adds confirmed query params. The command spec is cached ~24h and refreshes automatically; `--rsh-no-cache` forces a refresh.
- **List envelopes are not uniform.** `list-calls`, `list-results`, `list-call-results`, and `list-auth-settings` return `{ "meta": ..., "results": [...] }`; `list-schedules` returns `{ "data": [...] }`; `list-slos`, `list-browser-monitors`, and `list-mcp-monitors` return a bare `{ "results": [...] }`. Inspect each command's own output before writing an `-f` path.
- **`get-result` is a summary** (`result`, `http_code`, `response_time` ms, `location_id`, `test`, `created`; `result` ∈ `PASS`/`FAIL`/`WARN`/`ERROR`/`TIMEOUT`/`QUEUED`). Deeper data comes from `get-result-content`, `get-result-screenshot`, the `query-*-performance`/`query-*-dns-diagnostics` commands, and `conformance-results`.
- **Percentiles are server-computed** by the `query-*-performance` commands (`measures`: `mean`/`p50`/`p90`/`p95`/`p99`; API `metrics`: `total`/`dns`/`connect`/`tls`/`ttfb`/`response`). Do not hand-average result summaries.
- **API assertions** are set with `set-call-conditions <call-id>` (read with `get-call-conditions`), not in the `create-call` body.
- **Bulk config-as-code** lives under the hidden `apimetrics bulk` group (`init <url>` e.g. `apimetrics:/monitors`, `status`, `diff [--remote]`, `pull`, `reset`, `push`).
- **Never print secrets.** Reference auth settings by ID (`auth_id`/`token_id`); never paste tokens, cookies, keys, or certificate contents.
CLI Shorthand is also accepted. Quote any value containing `//`, or the shorthand parser rejects it, and redirect stdin from `/dev/null` when running non-interactively — shorthand also reads stdin to merge with the arguments, so an open stdin pipe blocks: `apimetrics create-call meta.name: Health, request.method: GET, request.url: '"https://example.com/health"' </dev/null`.
- **Output**: `-o json` for machine use; it prints the response body. `-f` projects with a shorthand query rooted at the whole response (`status`, `headers`, `body`), so every path starts with `body.` — single-quote it so the shell doesn't glob `[]`: `-f 'body.results[].meta.name'`. A path that matches nothing prints nothing and exits 0. `-q key=value` appends a raw, unvalidated query parameter; prefer the command's own `--option` flags. The command spec is cached ~24h and refreshes automatically; `--rsh-no-cache` forces a refresh.
- **List envelopes are not uniform.** `list-calls`, `list-api-monitors`, `list-results`, `list-results-by-call`, and `list-auth-settings` return `{ "meta": ..., "results": [...] }`; `list-schedules` and `list-monitors` return `{ "data": [...], "meta": ... }`; `list-browser-monitors` and `list-mcp-monitors` return a bare `{ "results": [...] }`. Paged lists return one page per call — repeat with `--cursor <meta.next_cursor>` while `meta.more` is true. Inspect each command's own output before writing an `-f` path.
- **Result rows** (`list-results`, `list-results-by-call`) are `{ id, meta, timing }`: `meta.result_category`, `meta.result`, `meta.http_code`, `meta.location_id`, `meta.call_id` (the monitor ID), `meta.start_time`, and `timing.total` in ms with `dns`/`tcp`/`handshake`/`processing`/`download`/`upload` components. `result_category` ∈ `PASS`/`SLOW`/`WARNING`/`FAIL` (HTTP 4xx lands in `WARNING`, connection errors in `FAIL`) and is empty while the run is queued; `result` carries the detail — `QUEUED` while in flight, then e.g. `COMPLETE`, `HTTP_CLIENT_ERROR`, `CONNECTION_ERROR`. `--result-category` takes those four values case-insensitively and silently ignores any other value, returning every category. `--from`/`--time` are the window start and end as full ISO-8601 date-times (a bare date is rejected) and are minute-granular; always pass `--from`, as the default window is short.
- **`get-result` detail depends on caller role.** A below-ANALYST caller gets a summary only — `meta` and `timing` as in a result row, with no `request`, `response`, or `context`. An ANALYST-or-above caller (most project members) gets the full object by default — response headers and body, request method/URL/headers, `meta.dns_details`, `meta.tls_version`, and the `timing` breakdown. Add `--mode full` for the request body, `messages`/`log_messages`, and certificate detail. An oversized response body is replaced by `"<< LARGE DATA >>"` with a `response.content_path`. Check `apimetrics get-result --help` and what actually comes back before reaching for `get-result-content`/`get-result-screenshot`/`query-*` commands.
- **Percentiles are server-computed** by the `query-*-performance` commands (`measures`: `mean`/`p50`/`p90`/`p95`/`p99`; API `metrics`: `total`/`dns`/`connect`/`tls`/`ttfb`/`response`/`upload`/`network`/`server`/`total_no_dns`; browser and MCP metric names differ — check `--help`). Without a `status` filter, failed runs are included; pass `"status": ["PASS"]` for the latency of successful runs. Do not hand-average result rows.
- **API assertions** are set with `set-call-conditions <call-id>` (read with `get-call-conditions`), as a separate step after `create-call`.
- **SLOs are one per project.** Use `get-project-slo`/`update-project-slo`/`delete-project-slo` (update-project-slo creates one if none exists); scope objectives with `include_tags`/`exclude_tags`. `get-project-slo` returns the definition — or platform defaults, with `created`/`last_update` null, when none is stored; measure attainment with the server-side aggregates — `get-call-passfail-range`/`get-call-passfail-total` per scoped monitor for availability (sum the per-category `stat.<CATEGORY>.total` values from `get-call-passfail-total`; its top-level `stat.total` counts only uncategorised results), `query-*-performance` for latency.
- **Bulk config-as-code** lives under the hidden `apimetrics bulk` group (`init <url>`, `list`, `status`, `diff [--remote]`, `pull`, `reset`, `push`). `init` needs a real, resolvable URL — a full host+path against the CLI's configured API server (e.g. `qc-client.apimetrics.io/api/2/calls/`). The checkout does not record its project, so pass the same `--project-id` (or keep the same selected project) on every `bulk` command. Files hold the full GET representation, which `PUT /api/2/calls/{id}` rejects (read-only `id`/`meta.*` fields), so reduce a changed file to just the writable fields being changed before `bulk push` — that endpoint applies partial updates. `bulk push` prints `Push complete.` and exits 0 even when an upload fails, and does not check for remote drift: treat any `Error uploading` line as a failure, and confirm `bulk status` is clean afterwards.
- **Never print secrets.** Reference auth settings by ID (`auth_id`/`token_id`); never paste tokens, cookies, keys, or certificate contents. `list-auth-settings` returns credential values in clear text, and monitor `request.headers`/`request.body` can hold literal tokens, so project those lists with `-f` to the fields you need rather than printing full JSON — e.g. `-f 'body.results.{id, name: meta.name, auth_type: settings.auth_type}'`.

## Workflows (skills)

Each entry names when to reach for it and the key commands. Full step-by-step workflows — including safety gates, evidence rules, and output templates — are in `skills/claude-skills/<name>/SKILL.md`; read that file before executing the workflow. Run `apimetrics onboard` to print every workflow to stdout for context injection.
Each entry names when to reach for it and the key commands. Full step-by-step workflows — including safety gates, evidence rules, and output templates — are in `skills/claude-skills/<name>/SKILL.md`; read that file before executing the workflow. Run `apimetrics onboard` to print the three monitor-setup guides and all eight workflows to stdout for context injection.

| Workflow | Use when | Key commands |
|---|---|---|
| **project-bootstrap** | setting up monitoring / onboarding a service | `project select`, `create-call`/`create-browser-monitor`/`create-mcp-monitor`, `set-call-conditions`, `create-schedule`, `add-call-to-schedule`, `run-call`/`run-monitor` |
| **weekly-health-review** | reliability report for the last 7 complete days | `list-results --since --before`, `query-api-performance`, `list-calls` |
| **failure-investigation** | root-causing one failing result or a series | `get-result`, `list-call-results --since --before`, `get-result-content`, `query-api-monitor-dns-diagnostics`, `get-call-conditions` |
| **incident-triage** | blast radius / common cause across many monitors | `list-results --since --before`, `query-api-dns-diagnostics`, `query-api-performance` |
| **monitoring-estate-audit** | coverage & hygiene governance review | `list-calls`, `list-schedules`, `list-slos`, `list-auth-settings`, `list-calls-by-auth` |
| **slo-review** | SLO design & attainment (no attainment endpoint — derive it) | `list-slos`, `get-slo`, `list-results`, `query-api-performance` |
| **config-as-code** | reviewed bulk edits with rollback | `bulk init apimetrics:/monitors`, `bulk status`, `bulk diff`, `bulk push`, `bulk reset` |
| **project-bootstrap** | setting up monitoring / onboarding a service | `project select`, `create-project-in-org` + `create-project-access`, `create-call`/`create-browser-monitor`/`create-mcp-monitor`, `set-call-conditions`, `create-schedule`, `add-call-to-schedule`, `run-monitor` |
| **weekly-health-review** | reliability report for the last 7 complete days | `get-call-passfail-range`, `list-results --from --time`, `query-api-performance`, `list-calls` |
| **failure-investigation** | root-causing one failing result or a series | `get-result`, `list-results-by-call --from --result-category`, `get-result-content`, `query-api-monitor-dns-diagnostics`, `get-call-conditions` |
| **incident-triage** | blast radius / common cause across many monitors | `list-results --from --time`, `query-api-dns-diagnostics`, `query-api-performance` |
| **monitoring-estate-audit** | coverage & hygiene governance review | `list-calls`, `list-schedules`, `get-project-slo`, `list-auth-settings`, `list-calls-by-auth` |
| **slo-review** | SLO design & attainment (one SLO per project; attainment from server-side aggregates) | `get-project-slo`, `update-project-slo`, `get-call-passfail-range`, `query-api-performance` |
| **config-as-code** | reviewed bulk edits with rollback | `bulk init <resolvable-url>`, `bulk status`, `bulk diff`, `bulk push`, `bulk reset` |
| **performance-analytics** | latency percentiles, component timing, DNS drift | `query-api-performance`, `query-api-monitor-performance`, `query-api-dns-diagnostics`, `query-browser-performance`, `query-mcp-performance` |

## Safety
Expand All @@ -55,4 +58,4 @@ The eight workflows above are shipped to Claude Code as native skills:
apimetrics skills install --claude-skills # writes .claude/skills/<name>/SKILL.md
```

The same content backs this `AGENTS.md` and `apimetrics onboard`, so Claude Code and other agents operate from one source of truth.
`apimetrics onboard` prints the same eight `SKILL.md` files that `skills install --claude-skills` installs, and this `AGENTS.md` indexes them, so Claude Code and other agents work from the same workflows.
14 changes: 9 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,9 +160,11 @@ EOF
You can also use **CLI Shorthand** as a concise alternative to JSON:

```bash
apimetrics create-call meta.name: "My API Check", request.method: GET, request.url: https://api.example.com/health
apimetrics create-call meta.name: "My API Check", request.method: GET, request.url: '"https://api.example.com/health"'
```

Quote any shorthand value that contains `//` (such as a URL), as above; unquoted, the shorthand parser rejects it. Shorthand also reads stdin and merges it with the arguments, so in scripts and other non-interactive shells redirect stdin from `/dev/null` (`... </dev/null`) or the command waits for input.

Or pipe from a file:

```bash
Expand All @@ -183,16 +185,18 @@ Use `-o` / `--rsh-output-format` to control output:
| `gron` | Grep-friendly flattened format |

```bash
# Table output
apimetrics list-calls -o table
# Table output (list bodies are objects, so project an array first)
apimetrics list-calls -o table -f 'body.results[].{id: id, name: meta.name}'

# Raw JSON for scripting
apimetrics list-calls -o json

# Filter results with a query expression
apimetrics list-calls -f body[0].meta.name
apimetrics list-calls -f 'body.results[0].meta.name'
```

`-f` paths are rooted at the whole response (`status`, `headers`, `body`), so they start with `body.`; single-quote them so the shell doesn't expand `[]`. A path that matches nothing prints nothing.

When output is redirected to a pipe or file, color is disabled and only the body is printed as JSON automatically.

## Global Flags
Expand Down Expand Up @@ -279,7 +283,7 @@ Read-only skills load automatically when a request matches; the two mutation ski

### Other agents (OpenAI Codex, Cursor, Gemini CLI, …)

The repo root ships an [`AGENTS.md`](AGENTS.md) — the cross-agent standard read by OpenAI Codex, Cursor, Gemini CLI, and others — that carries the same core CLI facts and workflow index. Any agent can also pull the full workflows on demand with `apimetrics onboard`.
The repo root ships an [`AGENTS.md`](AGENTS.md) — the cross-agent standard read by OpenAI Codex, Cursor, Gemini CLI, and others — that carries the same core CLI facts and workflow index. Any agent can also pull the full workflows on demand with `apimetrics onboard`, which prints the three setup skills followed by all eight workflow skills.

## Configuration

Expand Down
2 changes: 1 addition & 1 deletion bulk/commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ func Init(cmd *cobra.Command) {
Hidden: true,
Use: "bulk",
Short: "Client-side bulk resource management",
Example: " " + os.Args[0] + " bulk init apimetrics:/monitors\n " + os.Args[0] + " bulk list -m 'name contains checkout'\n " + os.Args[0] + " bulk status",
Example: " " + os.Args[0] + " bulk init api.example.com/users\n " + os.Args[0] + " bulk list -m 'id contains abc'\n " + os.Args[0] + " bulk status",
}

bulk.AddGroup(
Expand Down
2 changes: 1 addition & 1 deletion cli/cli.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ func Init(name string, version string) {
$ %s list-monitors

# Create a monitor
$ %s create-monitor name: "Checkout flow", url: https://example.com/checkout`, name, name),
$ %s create-call meta.name: "Checkout flow", request.method: GET, request.url: '"https://example.com/checkout"'`, name, name),
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
settings := viper.AllSettings()
LogDebug("Configuration: %v", settings)
Expand Down
Loading
Loading