Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
0d09499
Address Copilot review on PR #16
ndenny Sep 1, 2026
f7dff71
Bump shamaton/msgpack to v3
ndenny Sep 18, 2026
50388b0
Merge branch 'develop' into fix/copilot-review-pr16
ndenny Sep 18, 2026
e6cd7f3
Fix resolveTarget test signature after claude-skills merge
ndenny Sep 18, 2026
5903541
Address Copilot review: test isolation and error-path assertions
ndenny Sep 18, 2026
289ecc2
Pin goreleaser to the triggering tag to fix version mislabeling
ndenny Sep 18, 2026
e9ff5e9
Fix stale/incorrect CLI claims in skills and AGENTS.md
ndenny Sep 19, 2026
3cd1113
Address Copilot review: consistent result_category naming, cleaner bu…
ndenny Sep 19, 2026
0df8cc0
Merge pull request #25 from APImetrics/bump-msgpack-v3
ndenny Sep 21, 2026
6989381
Merge pull request #24 from APImetrics/fix/copilot-review-pr16
ndenny Sep 21, 2026
2f3df1d
Rewrite negative-space phrasing as standalone positive documentation
ndenny Sep 21, 2026
6dadac7
Fix flag name in slo-review: --project-id, not --apimetrics-project-id
ndenny Sep 21, 2026
a655802
Align bulk top-level help example with bulk init's own examples
ndenny Sep 21, 2026
99f7c0b
Merge pull request #26 from APImetrics/fix/goreleaser-current-tag-race
ndenny Sep 21, 2026
571824a
Address review: drop corrective phrasing, use server-side SLO aggregates
ndenny Sep 23, 2026
f072935
Align skills and onboard text with AGENTS.md
ndenny Sep 24, 2026
561242e
Merge remote-tracking branch 'origin/develop' into fix/skill-docs-liv…
ndenny Sep 24, 2026
88c9716
Scope cross-monitor hard rule to run-monitor; list-results-by-call is…
ndenny Sep 30, 2026
53deef7
list-results-by-call covers every monitor type
ndenny Oct 9, 2026
b7d4d6f
Correct skills and docs against live QC; make onboard print all workf…
ndenny Oct 9, 2026
f72acf3
Clarify create-project-access grantee and passfail stat.total semantics
ndenny Oct 9, 2026
dc8f6ab
Request offline_access so browser logins get a refresh token
ndenny Oct 16, 2026
652503e
Add device code login for machines without a local browser
ndenny Oct 16, 2026
02af57f
Share token request and parsing between the device flow and requestToken
ndenny Oct 16, 2026
69010f9
Fix flaky TestRequestRetryTimeout
ndenny Oct 16, 2026
f805a51
Document full run-monitor response and bare-date rejection in more sk…
ndenny Oct 16, 2026
bc1b015
Give each retry its own timeout and keep it alive while the body is read
ndenny Oct 16, 2026
674aa98
Merge pull request #30 from APImetrics/feat/device-code-login
ndenny Oct 19, 2026
be7055e
Merge pull request #27 from APImetrics/fix/skill-docs-live-cli-review
ndenny Oct 19, 2026
03abafd
Merge pull request #31 from APImetrics/fix/flaky-retry-timeout-test
ndenny Oct 19, 2026
c28fa72
Merge develop into feat/refresh-tokens
ndenny Oct 19, 2026
ad0c94c
Merge pull request #29 from APImetrics/feat/refresh-tokens
ndenny Oct 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -193,10 +193,23 @@ jobs:
echo "Building artifacts without publishing; GitHub release and GCS upload deferred to finalization"
fi

# All four .goreleaser configs sort tags with `-version:refname`, so
# when several environment tags point at the same commit (e.g.
# qc/beta/qc-stable cut from the same develop HEAD), GoReleaser's own
# git-based tag detection can pick a sibling tag instead of the one
# that triggered this run — silently mislabeling main.version.
# GORELEASER_CURRENT_TAG pins it to the actual triggering tag.
DOCKER_ENV_ARGS=(-e GITHUB_TOKEN)
if [[ "$GITHUB_REF" == refs/tags/* ]]; then
export GORELEASER_CURRENT_TAG="$GITHUB_REF_NAME"
DOCKER_ENV_ARGS+=(-e GORELEASER_CURRENT_TAG)
echo "Pinning goreleaser to the triggering tag: $GORELEASER_CURRENT_TAG"
fi

docker run --rm \
-v "${PWD}:/workspace" \
-w /workspace \
-e GITHUB_TOKEN \
"${DOCKER_ENV_ARGS[@]}" \
ghcr.io/goreleaser/goreleaser-cross:v1.25-v2.12.7 \
release $EXTRA --config "$CONFIG" --clean --parallelism 2

Expand Down
45 changes: 24 additions & 21 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,40 +4,43 @@ Instructions for AI coding agents (OpenAI Codex, Cursor, Gemini CLI, and any too

## Prerequisites

1. `apimetrics project show` — confirm a project is active. Every project-scoped command fails without one. If none is set, run `apimetrics project select`.
2. `apimetrics login` handles authentication (OAuth). **There is no `--api-key` flag.**
1. `apimetrics project show` — confirm a project is active. Every project-scoped command fails without one unless you pass `--project-id <id>`. If none is set, run `apimetrics project select`.
2. Commands log in automatically when needed; `apimetrics login` does it up front (OAuth). For headless use, pass `--service-account <file>`.

## Core facts about this CLI

- **Commands are flat, generated from the platform's live OpenAPI description**: `list-calls`, `create-call`, `run-call` — not `calls create`. The set can change with the server spec, so treat `apimetrics --help` and `apimetrics <command> --help` as authoritative and inspect a command's help before assuming an option or body field.
- **Create/update commands read JSON from stdin** via heredoc. There is no `--body`, `--data`, or `-d`:
- **Commands are flat, generated from the platform's live OpenAPI description**: `list-calls`, `create-call`, `run-monitor`. The set can change with the server spec, so treat `apimetrics --help` and `apimetrics <command> --help` as authoritative and inspect a command's help before assuming an option or body field. `run-monitor <id>` is the single command that starts a run for any monitor type — API call, browser, or MCP; it requires a body, so send at least `{}` on stdin, and returns the `result_id` to poll.
- **Deprecated commands still run but warn.** Where the server's spec deprecates a command, the CLI hides it from `--help` and prints `Command "X" is deprecated, do not use` on stderr (stdout JSON is unaffected). On servers that expose the `/api/3/monitors/` commands, `list-calls`/`create-call`/`get-call`/`update-call`/`delete-call` are deprecated in favour of `list-api-monitors` (or `list-monitors` for every type), `create-api-monitor`, `get-api-monitor`, `patch-api-monitor`/`update-api-monitor`, and `delete-api-monitor`. The new create/update commands take a flat body (`name`, `method`, `url`, `headers`, …) rather than `meta`/`request`; monitor IDs are the same, so `set-call-conditions`, `get-call-passfail-*`, and `list-results-by-call` keep working. Use whichever set your `--help` lists.
- **Create/update commands read their JSON body from stdin** via heredoc:
```bash
apimetrics create-call <<'EOF'
{ "meta": { "name": "Health" }, "request": { "method": "GET", "url": "https://example.com/health" } }
EOF
```
CLI Shorthand is also accepted (`apimetrics create-call meta.name: "Health", request.method: GET, request.url: https://example.com/health`).
- **Output**: `-o json` for machine use; `-f` projects with a shorthand query; `-q` adds confirmed query params. The command spec is cached ~24h and refreshes automatically; `--rsh-no-cache` forces a refresh.
- **List envelopes are not uniform.** `list-calls`, `list-results`, `list-call-results`, and `list-auth-settings` return `{ "meta": ..., "results": [...] }`; `list-schedules` returns `{ "data": [...] }`; `list-slos`, `list-browser-monitors`, and `list-mcp-monitors` return a bare `{ "results": [...] }`. Inspect each command's own output before writing an `-f` path.
- **`get-result` is a summary** (`result`, `http_code`, `response_time` ms, `location_id`, `test`, `created`; `result` ∈ `PASS`/`FAIL`/`WARN`/`ERROR`/`TIMEOUT`/`QUEUED`). Deeper data comes from `get-result-content`, `get-result-screenshot`, the `query-*-performance`/`query-*-dns-diagnostics` commands, and `conformance-results`.
- **Percentiles are server-computed** by the `query-*-performance` commands (`measures`: `mean`/`p50`/`p90`/`p95`/`p99`; API `metrics`: `total`/`dns`/`connect`/`tls`/`ttfb`/`response`). Do not hand-average result summaries.
- **API assertions** are set with `set-call-conditions <call-id>` (read with `get-call-conditions`), not in the `create-call` body.
- **Bulk config-as-code** lives under the hidden `apimetrics bulk` group (`init <url>` e.g. `apimetrics:/monitors`, `status`, `diff [--remote]`, `pull`, `reset`, `push`).
- **Never print secrets.** Reference auth settings by ID (`auth_id`/`token_id`); never paste tokens, cookies, keys, or certificate contents.
CLI Shorthand is also accepted. Quote any value containing `//`, or the shorthand parser rejects it, and redirect stdin from `/dev/null` when running non-interactively — shorthand also reads stdin to merge with the arguments, so an open stdin pipe blocks: `apimetrics create-call meta.name: Health, request.method: GET, request.url: '"https://example.com/health"' </dev/null`.
- **Output**: `-o json` for machine use; it prints the response body. `-f` projects with a shorthand query rooted at the whole response (`status`, `headers`, `body`), so every path starts with `body.` — single-quote it so the shell doesn't glob `[]`: `-f 'body.results[].meta.name'`. A path that matches nothing prints nothing and exits 0. `-q key=value` appends a raw, unvalidated query parameter; prefer the command's own `--option` flags. The command spec is cached ~24h and refreshes automatically; `--rsh-no-cache` forces a refresh.
- **List envelopes are not uniform.** `list-calls`, `list-api-monitors`, `list-results`, `list-results-by-call`, and `list-auth-settings` return `{ "meta": ..., "results": [...] }`; `list-schedules` and `list-monitors` return `{ "data": [...], "meta": ... }`; `list-browser-monitors` and `list-mcp-monitors` return a bare `{ "results": [...] }`. Paged lists return one page per call — repeat with `--cursor <meta.next_cursor>` while `meta.more` is true. Inspect each command's own output before writing an `-f` path.
- **Result rows** (`list-results`, `list-results-by-call`) are `{ id, meta, timing }`: `meta.result_category`, `meta.result`, `meta.http_code`, `meta.location_id`, `meta.call_id` (the monitor ID), `meta.start_time`, and `timing.total` in ms with `dns`/`tcp`/`handshake`/`processing`/`download`/`upload` components. `result_category` ∈ `PASS`/`SLOW`/`WARNING`/`FAIL` (HTTP 4xx lands in `WARNING`, connection errors in `FAIL`) and is empty while the run is queued; `result` carries the detail — `QUEUED` while in flight, then e.g. `COMPLETE`, `HTTP_CLIENT_ERROR`, `CONNECTION_ERROR`. `--result-category` takes those four values case-insensitively and silently ignores any other value, returning every category. `--from`/`--time` are the window start and end as full ISO-8601 date-times (a bare date is rejected) and are minute-granular; always pass `--from`, as the default window is short.
- **`get-result` detail depends on caller role.** A below-ANALYST caller gets a summary only — `meta` and `timing` as in a result row, with no `request`, `response`, or `context`. An ANALYST-or-above caller (most project members) gets the full object by default — response headers and body, request method/URL/headers, `meta.dns_details`, `meta.tls_version`, and the `timing` breakdown. Add `--mode full` for the request body, `messages`/`log_messages`, and certificate detail. An oversized response body is replaced by `"<< LARGE DATA >>"` with a `response.content_path`. Check `apimetrics get-result --help` and what actually comes back before reaching for `get-result-content`/`get-result-screenshot`/`query-*` commands.
- **Percentiles are server-computed** by the `query-*-performance` commands (`measures`: `mean`/`p50`/`p90`/`p95`/`p99`; API `metrics`: `total`/`dns`/`connect`/`tls`/`ttfb`/`response`/`upload`/`network`/`server`/`total_no_dns`; browser and MCP metric names differ — check `--help`). Without a `status` filter, failed runs are included; pass `"status": ["PASS"]` for the latency of successful runs. Do not hand-average result rows.
- **API assertions** are set with `set-call-conditions <call-id>` (read with `get-call-conditions`), as a separate step after `create-call`.
- **SLOs are one per project.** Use `get-project-slo`/`update-project-slo`/`delete-project-slo` (update-project-slo creates one if none exists); scope objectives with `include_tags`/`exclude_tags`. `get-project-slo` returns the definition — or platform defaults, with `created`/`last_update` null, when none is stored; measure attainment with the server-side aggregates — `get-call-passfail-range`/`get-call-passfail-total` per scoped monitor for availability (sum the per-category `stat.<CATEGORY>.total` values from `get-call-passfail-total`; its top-level `stat.total` counts only uncategorised results), `query-*-performance` for latency.
- **Bulk config-as-code** lives under the hidden `apimetrics bulk` group (`init <url>`, `list`, `status`, `diff [--remote]`, `pull`, `reset`, `push`). `init` needs a real, resolvable URL — a full host+path against the CLI's configured API server (e.g. `qc-client.apimetrics.io/api/2/calls/`). The checkout does not record its project, so pass the same `--project-id` (or keep the same selected project) on every `bulk` command. Files hold the full GET representation, which `PUT /api/2/calls/{id}` rejects (read-only `id`/`meta.*` fields), so reduce a changed file to just the writable fields being changed before `bulk push` — that endpoint applies partial updates. `bulk push` prints `Push complete.` and exits 0 even when an upload fails, and does not check for remote drift: treat any `Error uploading` line as a failure, and confirm `bulk status` is clean afterwards.
- **Never print secrets.** Reference auth settings by ID (`auth_id`/`token_id`); never paste tokens, cookies, keys, or certificate contents. `list-auth-settings` returns credential values in clear text, and monitor `request.headers`/`request.body` can hold literal tokens, so project those lists with `-f` to the fields you need rather than printing full JSON — e.g. `-f 'body.results.{id, name: meta.name, auth_type: settings.auth_type}'`.

## Workflows (skills)

Each entry names when to reach for it and the key commands. Full step-by-step workflows — including safety gates, evidence rules, and output templates — are in `skills/claude-skills/<name>/SKILL.md`; read that file before executing the workflow. Run `apimetrics onboard` to print every workflow to stdout for context injection.
Each entry names when to reach for it and the key commands. Full step-by-step workflows — including safety gates, evidence rules, and output templates — are in `skills/claude-skills/<name>/SKILL.md`; read that file before executing the workflow. Run `apimetrics onboard` to print the three monitor-setup guides and all eight workflows to stdout for context injection.

| Workflow | Use when | Key commands |
|---|---|---|
| **project-bootstrap** | setting up monitoring / onboarding a service | `project select`, `create-call`/`create-browser-monitor`/`create-mcp-monitor`, `set-call-conditions`, `create-schedule`, `add-call-to-schedule`, `run-call`/`run-monitor` |
| **weekly-health-review** | reliability report for the last 7 complete days | `list-results --since --before`, `query-api-performance`, `list-calls` |
| **failure-investigation** | root-causing one failing result or a series | `get-result`, `list-call-results --since --before`, `get-result-content`, `query-api-monitor-dns-diagnostics`, `get-call-conditions` |
| **incident-triage** | blast radius / common cause across many monitors | `list-results --since --before`, `query-api-dns-diagnostics`, `query-api-performance` |
| **monitoring-estate-audit** | coverage & hygiene governance review | `list-calls`, `list-schedules`, `list-slos`, `list-auth-settings`, `list-calls-by-auth` |
| **slo-review** | SLO design & attainment (no attainment endpoint — derive it) | `list-slos`, `get-slo`, `list-results`, `query-api-performance` |
| **config-as-code** | reviewed bulk edits with rollback | `bulk init apimetrics:/monitors`, `bulk status`, `bulk diff`, `bulk push`, `bulk reset` |
| **project-bootstrap** | setting up monitoring / onboarding a service | `project select`, `create-project-in-org` + `create-project-access`, `create-call`/`create-browser-monitor`/`create-mcp-monitor`, `set-call-conditions`, `create-schedule`, `add-call-to-schedule`, `run-monitor` |
| **weekly-health-review** | reliability report for the last 7 complete days | `get-call-passfail-range`, `list-results --from --time`, `query-api-performance`, `list-calls` |
| **failure-investigation** | root-causing one failing result or a series | `get-result`, `list-results-by-call --from --result-category`, `get-result-content`, `query-api-monitor-dns-diagnostics`, `get-call-conditions` |
| **incident-triage** | blast radius / common cause across many monitors | `list-results --from --time`, `query-api-dns-diagnostics`, `query-api-performance` |
| **monitoring-estate-audit** | coverage & hygiene governance review | `list-calls`, `list-schedules`, `get-project-slo`, `list-auth-settings`, `list-calls-by-auth` |
| **slo-review** | SLO design & attainment (one SLO per project; attainment from server-side aggregates) | `get-project-slo`, `update-project-slo`, `get-call-passfail-range`, `query-api-performance` |
| **config-as-code** | reviewed bulk edits with rollback | `bulk init <resolvable-url>`, `bulk status`, `bulk diff`, `bulk push`, `bulk reset` |
| **performance-analytics** | latency percentiles, component timing, DNS drift | `query-api-performance`, `query-api-monitor-performance`, `query-api-dns-diagnostics`, `query-browser-performance`, `query-mcp-performance` |

## Safety
Expand All @@ -55,4 +58,4 @@ The eight workflows above are shipped to Claude Code as native skills:
apimetrics skills install --claude-skills # writes .claude/skills/<name>/SKILL.md
```

The same content backs this `AGENTS.md` and `apimetrics onboard`, so Claude Code and other agents operate from one source of truth.
`apimetrics onboard` prints the same eight `SKILL.md` files that `skills install --claude-skills` installs, and this `AGENTS.md` indexes them, so Claude Code and other agents work from the same workflows.
32 changes: 22 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,27 +9,27 @@ For more information, please visit our main documentation site, available at [ht

## Installation

## Manual installation
### Manual installation

The binaries for the APImetrics CLI can be downloaded from the [releases page](https://github.com/APImetrics/APImetrics-cli/releases). Choose the appropriate version for your operating system and put the executable in your path for easy access from the command line.

## MacOS
### MacOS

You can install the APImetrics CLI using Homebrew:

```bash
brew install apimetrics/tap/apimetrics
```

## Windows
### Windows

You can install the APImetrics CLI using the Windows Package Manager (winget):

```powershell
winget install APIContext.APImetricsCLI
```

## Linux
### Linux

You can install the APImetrics CLI by downloading the binary from the [releases page](https://github.com/APImetrics/APImetrics-cli/releases).

Expand All @@ -50,6 +50,14 @@ apimetrics login

This opens your browser for OAuth2 authentication. Your credentials are cached locally and refreshed automatically.

On a machine whose browser can't reach it — over SSH, in a container, or on a remote desktop — use the device flow instead:

```bash
apimetrics login --device
```

The CLI prints a URL and a short code. Open the URL in a browser on any device, check the code matches, and approve; the CLI finishes logging in once you do.

### 2. Select a project

All commands require an active project. Run the following to choose one:
Expand Down Expand Up @@ -141,7 +149,7 @@ by `project select`. To see which projects a service account can reach, run

## Passing Input

All create and update commands read a JSON body from **stdin** using a heredoc. There is no `--body`, `--data`, or `-d` flag.
Create and update commands take their body from **stdin** (a heredoc is the clearest way to write one) or from **CLI Shorthand** arguments. There is no `--body`, `--data`, or `-d` flag.

```bash
apimetrics create-call <<'EOF'
Expand All @@ -160,9 +168,11 @@ EOF
You can also use **CLI Shorthand** as a concise alternative to JSON:

```bash
apimetrics create-call meta.name: "My API Check", request.method: GET, request.url: https://api.example.com/health
apimetrics create-call meta.name: "My API Check", request.method: GET, request.url: '"https://api.example.com/health"'
```

Quote any shorthand value that contains `//` (such as a URL), as above; unquoted, the shorthand parser rejects it. Shorthand also reads stdin and merges it with the arguments, so in scripts and other non-interactive shells redirect stdin from `/dev/null` (`... </dev/null`) or the command waits for input.

Or pipe from a file:

```bash
Expand All @@ -183,16 +193,18 @@ Use `-o` / `--rsh-output-format` to control output:
| `gron` | Grep-friendly flattened format |

```bash
# Table output
apimetrics list-calls -o table
# Table output (list bodies are objects, so project an array first)
apimetrics list-calls -o table -f 'body.results[].{id: id, name: meta.name}'

# Raw JSON for scripting
apimetrics list-calls -o json

# Filter results with a query expression
apimetrics list-calls -f body[0].meta.name
apimetrics list-calls -f 'body.results[0].meta.name'
```

`-f` paths are rooted at the whole response (`status`, `headers`, `body`), so they start with `body.`; single-quote them so the shell doesn't expand `[]`. A path that matches nothing prints nothing.

When output is redirected to a pipe or file, color is disabled and only the body is printed as JSON automatically.

## Global Flags
Expand Down Expand Up @@ -279,7 +291,7 @@ Read-only skills load automatically when a request matches; the two mutation ski

### Other agents (OpenAI Codex, Cursor, Gemini CLI, …)

The repo root ships an [`AGENTS.md`](AGENTS.md) — the cross-agent standard read by OpenAI Codex, Cursor, Gemini CLI, and others — that carries the same core CLI facts and workflow index. Any agent can also pull the full workflows on demand with `apimetrics onboard`.
The repo root ships an [`AGENTS.md`](AGENTS.md) — the cross-agent standard read by OpenAI Codex, Cursor, Gemini CLI, and others — that carries the same core CLI facts and workflow index. Any agent can also pull the full workflows on demand with `apimetrics onboard`, which prints the three setup skills followed by all eight workflow skills.

## Configuration

Expand Down
2 changes: 1 addition & 1 deletion bench_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
"apicontext.com/apimetrics/openapi"
"github.com/amzn/ion-go/ion"
"github.com/fxamacker/cbor/v2"
"github.com/shamaton/msgpack/v2"
"github.com/shamaton/msgpack/v3"
"github.com/spf13/cobra"
)

Expand Down
Loading
Loading