From 06700d2991e74c811373aae2281381088f857225 Mon Sep 17 00:00:00 2001 From: Siddhant Sinha Date: Tue, 27 Oct 2026 18:06:54 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20policies=20and=20verdicts=20=E2=80=94?= =?UTF-8?q?=20user=20guide=20page=20and=20skill=20reference?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add docs/user-guide/policies.md: the policy YAML format, the built-in and kane-cli metrics, lint-policy, evidence verdict, --policy/--policy-id on testmd run, testrun run and evidence merge, verdict files and exit codes, policies in Test Manager (policy create|list|metrics-list), and a CI example. Link it from the guide index, evidence, testrun, testmd running, cicd, running-tests and test-manager-integration pages. Add the agent reference skill-installer/skills/references/policy.md, the flags in the evidence, testrun and testmd references, and routing lines in SKILL.md; the .claude and .agents copies are kept in sync. Marked as the release after 0.10.0-rc.3. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/kane-cli/SKILL.md | 4 +- .../skills/kane-cli/references/evidence.md | 15 +- .agents/skills/kane-cli/references/policy.md | 132 +++++ .agents/skills/kane-cli/references/testmd.md | 1 + .agents/skills/kane-cli/references/testrun.md | 6 + .claude/skills/kane-cli/SKILL.md | 4 +- .../skills/kane-cli/references/evidence.md | 15 +- .claude/skills/kane-cli/references/policy.md | 132 +++++ .claude/skills/kane-cli/references/testmd.md | 1 + .claude/skills/kane-cli/references/testrun.md | 6 + docs/user-guide/README.md | 2 + docs/user-guide/cicd.md | 3 + docs/user-guide/evidence.md | 21 +- docs/user-guide/policies.md | 457 ++++++++++++++++++ docs/user-guide/running-tests.md | 3 + docs/user-guide/test-manager-integration.md | 5 + docs/user-guide/testmd/running.md | 5 + docs/user-guide/testrun.md | 6 + skill-installer/skills/SKILL.md | 4 +- skill-installer/skills/references/evidence.md | 15 +- skill-installer/skills/references/policy.md | 132 +++++ skill-installer/skills/references/testmd.md | 1 + skill-installer/skills/references/testrun.md | 6 + 23 files changed, 964 insertions(+), 12 deletions(-) create mode 100644 .agents/skills/kane-cli/references/policy.md create mode 100644 .claude/skills/kane-cli/references/policy.md create mode 100644 docs/user-guide/policies.md create mode 100644 skill-installer/skills/references/policy.md diff --git a/.agents/skills/kane-cli/SKILL.md b/.agents/skills/kane-cli/SKILL.md index 1aebb55..20d7f87 100644 --- a/.agents/skills/kane-cli/SKILL.md +++ b/.agents/skills/kane-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: kane-cli -description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover. Work in a team's shared Test Manager project from a linked folder: link it with --project, save sessions, run the designed tests on their Test Manager cases, and read the project's coverage. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share a local assurance store (.context/) through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." +description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover, and judge a pack against a policy of quality gates (kane-cli evidence verdict, --policy) to pass or fail a CI run. Work in a team's shared Test Manager project from a linked folder: link it with --project, save sessions, run the designed tests on their Test Manager cases, and read the project's coverage. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share a local assurance store (.context/) through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." --- # Kane CLI — End-to-End Test Generation and Execution Skill @@ -188,6 +188,7 @@ When the user's request involves a browser — or writing test cases: - Share a **local** context store through a repository, a bucket or a folder, join one, keep two stores level, or resolve a sync conflict → Read `references/context-sync.md` first (kane-cli 0.8.14+). A team in Test Manager uses a shared project instead (`references/shared-projects.md`); a linked folder refuses `kane-cli context sync`, `kane-cli context push` and `kane-cli context clone` (`REMOTE_UNSUPPORTED`), and a location pull (a location name, `--rebase`, `--yes` or `--answer`: `PULL_LOCATION_LINKED`); a plain `kane-cli context pull --mode agent` there brings the project's tests (`references/shared-projects.md` §7); the store is shared through a location, never by copying or git-merging `.context/` - Multiple independent browser tasks → Read `references/parallel.md` first - View, share, or validate run evidence (`.evidence` packs) → Read `references/evidence.md` +- Judge a run or pack against quality gates (a policy YAML, `--policy`, `--policy-id`, `kane-cli evidence verdict`, `kane-cli policy …`), or gate CI on more than pass/fail → Read `references/policy.md` first (after 0.10.0-rc.3) - Debug a failed run → Read `references/debug.md` - Configure kane-cli or check directory layout → Read `references/setup-and-config.md` - Browse / create / pick a Test Manager project or folder for `kane-cli run`, or interpret the auto-default event → Read `references/test-manager.md`. Linking an assurance folder to a project is different: `references/shared-projects.md` §2–3 @@ -382,6 +383,7 @@ A draft in chat cannot be run, and a hand-written `_test.md` carries no requirem | Share the context store with a team, join one, keep stores level, or resolve a sync conflict (kane-cli 0.8.14+) | `references/context-sync.md` | | Run failed, need to diagnose | `references/debug.md` | | View, share, validate, or merge evidence packs | `references/evidence.md` | +| Judge a pack against a policy, run with `--policy`, or store/list policies in Test Manager | `references/policy.md` | | Multiple independent browser tasks | `references/parallel.md` | | Need full NDJSON event schema (`run`) | `references/parsing.md` | | Browse / create projects or folders, or parse the auto-default event | `references/test-manager.md` | diff --git a/.agents/skills/kane-cli/references/evidence.md b/.agents/skills/kane-cli/references/evidence.md index fbe6643..ba86a3e 100644 --- a/.agents/skills/kane-cli/references/evidence.md +++ b/.agents/skills/kane-cli/references/evidence.md @@ -1,4 +1,4 @@ - + # Evidence Packs — Agent Surface @@ -69,7 +69,18 @@ Exit `0` valid · `1` invalid · `2` not found. Use it when a pack won't open in kane-cli evidence merge --run-id ``` -Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/.evidence`. `--run-id` is required. Exit `0` merged · `1` policy abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · `2` usage. Full policy detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. +Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/.evidence`. `--run-id` is required. Exit `0` merged · `1` merge-rules abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · `2` usage. Full merge-rules detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. + +After 0.10.0-rc.3, `--policy ` / `--policy-id ` (repeatable) judge the sealed merged pack: exit `1` then also means a failing verdict (pack and verdict files written), `3` an errored verdict; `--json` adds `verdict {…}` to the report; `--no-finalize` with a policy is exit 2. See `references/policy.md`. + +## Verdict — judge a pack against a policy (after 0.10.0-rc.3) + +```bash +kane-cli evidence lint-policy --json +kane-cli evidence verdict --policy [--policy-id ] --json +``` + +Writes `.verdicts/.verdict.json` beside the pack (never inside it). Exit `0` pass · `1` fail · `3` errored · `2` usage or I/O. Policy format, metrics, the `verdict` event, run-time `--policy` and Test Manager policies: `references/policy.md`. ## Debugging with a pack diff --git a/.agents/skills/kane-cli/references/policy.md b/.agents/skills/kane-cli/references/policy.md new file mode 100644 index 0000000..0fcd70b --- /dev/null +++ b/.agents/skills/kane-cli/references/policy.md @@ -0,0 +1,132 @@ + + +# Policies and Verdicts — Agent Surface + +**Version.** Needs the kane-cli release after 0.10.0-rc.3. If `kane-cli policy metrics-list` is an unknown command, or `--policy` is an unknown option, the installed build predates policies: say so and stop; do not fake a verdict by reading the pack yourself. + +A **policy** is a YAML file of **gates** over a sealed evidence pack's metrics. A **verdict** is one policy judged against one pack: `pass`, `fail` or `errored`. The pack is never modified; each verdict is a JSON file beside it at `.verdicts/.verdict.json`. + +Use a policy when the person wants a run judged on more than pass/fail: "no broken tests", "pass rate at least 95% on checkout", "every test must assert something", "fail CI if a test needed a heal". + +## 1. Commands + +| Command | What it does | +|---|---| +| `kane-cli policy metrics-list [--pack ] [--env prod\|stage] [--agent]` | Every metric a policy can gate on: built-in metrics, then the kane-cli catalog as `metrics.`. `--pack` = what one pack supports (the catalog it was sealed with). | +| `kane-cli evidence lint-policy [--json] [--env ]` | Check policy files against the format and the kane-cli catalog. Prints every problem. | +| `kane-cli evidence verdict (--policy \| --policy-id )... [--json] [--env ]` | Judge one sealed pack. `` = execution id, `.evidence` path, or `latest` (newest sealed pack in `/.testmuai/evidence/` by mtime). | +| `kane-cli testmd run --policy / --policy-id ` | Lint before the run, judge the test's own pack after it seals. | +| `kane-cli testrun run … --policy / --policy-id < /dev/null` | Lint after the plan (dry run too), judge the suite's pack. `--remote`: the laptop judges the consolidated pack; the grid never gets the flag. | +| `kane-cli evidence merge --run-id --policy / --policy-id ` | Judge the sealed merged pack. Files are read but **not** linted first (a bad policy = an errored verdict). `--no-finalize` with a policy = exit 2. | +| `kane-cli policy list [--project \| --all-projects] [--search ] [--limit 1-200] [--page ] [--agent]` | Policies in Test Manager (latest versions). Default scope: the profile's project. | +| `kane-cli policy create [--project ] [--agent]` | Lint, then store in Test Manager. New content under an existing `policy:` id = a new version; identical content = the stored version (`created: false`). Project: `--project`, else the profile's (`kane-cli config project `); none = exit 2. | + +`policy list` / `policy create` also take `--profile`, `--env`, `--username`, `--access-key`. `evidence verdict --policy-id` signs in with the active profile; `testmd run` / `testrun run` use the run's `--username` / `--access-key` when given. + +**Flag rules.** `--policy` and `--policy-id` are repeatable and combine. Repeat the flag per policy; never comma-join (`--policy a.yaml,b.yaml` is one file name). Two policies with the same `policy:` id in one command are refused. `--policy-id` takes the id `policy list` prints (the `ID` column / `uid`), not the YAML name, and always judges the **latest version**. + +## 2. Policy YAML + +```yaml +policy: release-gate # required; names the verdict file; ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ +requires: # optional: refuse packs with another catalog / version + catalog: kane-cli + version: ">=1" +gates: # required, at least one; at most 100; file at most 64 KB + - { id: no-failed, metric: totals.failed, equals: 0, severity: blocking } + - { id: checkout-rate, metric: pass_rate, where: "'checkout' in test.tags", min: 0.95, severity: blocking } + - { id: every-test-checks, metric: metrics.tests_without_checks, equals: 0, severity: blocking } + - { id: no-heals, metric: metrics.heal.healed_tests, max: 0, severity: warning, message: Review the heal } + - { id: no-flaky-critical, expr: "tests.filter(t, 'critical' in t.tags).all(t, !t.flaky)", severity: warning } +``` + +- A gate is **either** `metric` + exactly one of `equals` / `min` / `max` / `range: [lo, hi]` / `in: [...]`, **or** `expr` (CEL over `run`, `tests`, `totals`, `failures`, must be boolean) with no metric/operator/where. +- `severity`: `blocking` fails the verdict; `warning` reports (`status: warn`) without failing. An **errored** gate errors the verdict whatever its severity. +- `where` (CEL over one `test`) only on per-test metrics. +- Unknown keys are refused. +- Operators by type: counts/integers/ms/bytes/currency → all five; ratio/percentage/decimal → `min`/`max`/`range`; string/enum → `equals`/`in`. + +**Metrics.** Built-in (no prefix): `totals.tests`, `totals.passed`, `totals.failed`, `totals.broken`, `totals.skipped`, `pass_rate` (ratio; errors when every test skipped), `steps.total`, `steps.failed`, `steps.skipped` (all per test), `run.duration_ms`, `run.status` (enum `running|finalized|aborted`), `failures.count`. kane-cli catalog v1 (always `metrics.` prefix, all per test): `tests_without_checks`, `heal.healed_tests`, `heal.exhausted_tests`, `heal.reauthored_steps`, `non_replay_tests` (counts), `flaky_rate` (ratio). Run `kane-cli policy metrics-list --agent` for the installed truth before writing a gate; never invent a metric name. + +**Values.** A missing value is never zero: no value for the metric, or a pack older than the metric needs, **errors** the gate. A stored number that disagrees with its recompute also errors it. + +## 3. Exit codes + +| Command | 0 | 1 | 2 | 3 | +|---|---|---|---|---| +| `evidence verdict` | all pass | a verdict failed (none errored) | usage / I/O: no policy given, no pack, unreadable or unfetchable policy, a policy that doesn't check clean against the pack's catalog, unreadable pack, verdict files not written | a verdict errored | +| `evidence lint-policy` | clean (warnings allowed) | any error | file unreadable or lint threw | — | +| `testmd run` / `testrun run` with a policy | passed and verdict pass | tests failed, **or** tests passed and verdict failed | bad policy before anything ran (lint error, unknown `--policy-id`, sign-in failure), plus the command's usual 2s | verdict errored (tests passed), **or** cancelled / timed out | +| `evidence merge` with a policy | merged, verdict pass | merge-rules abort (no pack) or verdict failed (pack + verdicts written) | usage (incl. `--no-finalize`), verdict files not written | verdict errored | +| `policy metrics-list` | ok | — | unknown pack / catalog won't load | — | +| `policy list` | ok | Test Manager failure | bad flag / sign-in | — | +| `policy create` | created or already stored | Test Manager failure | no project, unreadable, lint errors, rejected by Test Manager (problems printed), sign-in | — | + +A verdict only changes a **clean 0**: failing → 1, errored → 3. A run that already failed or was cancelled keeps its own code. `test_md_done.overall_status` / `testrun_done.overall_status` stay the **tests'** status: read the `verdict` event for the policy result. Exit 3 is ambiguous on runs (errored verdict vs cancel/timeout): the `verdict` event decides it. + +## 4. The `verdict` event (agent mode) + +`testmd run`, `testrun run` (local and `--remote`) emit one typed line on stdout after the pack is judged and **before** the terminal event (`test_md_done`; `testrun_summary`/`testrun_done`; `remote_done`, whose `exit` already includes the verdict): + +```json +{"type":"verdict","evidence_id":"","result":"fail","exit_code":1, + "policies":[{"id":"release-gate","sha256":"…","result":"fail","exit_code":1, + "counts":{"total":5,"failed":1,"warned":0,"errored":0}, + "failed_gate_ids":["no-failed"],"errored_gate_ids":[], + "file":"/abs/path/.evidence.verdicts/release-gate.verdict.json"}], + "v":1,"ts":"…"} +``` + +- Combined `result`: errored if any policy errored, else fail if any failed, else pass. +- `error` (top level): nothing could be judged (e.g. the pack did not seal) — `policies` is empty. +- `policies[].error`: that verdict errored as a whole — `{code, message, hint}`. +- `files_error`: verdicts were made but files could not be written; the run warns and continues (verdicts still upload with the evidence). +- In a terminal the same information is one stderr line: `verdict: fail — release-gate fail (no-failed)`. + +`evidence verdict --json` prints the same object without `type`, plus `verdicts: []`. `evidence merge --json` adds it to the merge report as `verdict: {…}`. + +Parsing: + +```text +if type === "verdict": + policy_result = result # pass | fail | errored + for p in policies: note p.id, p.result, p.failed_gate_ids, p.errored_gate_ids, p.file + if error: the pack could not be judged — report error, not a gate failure +``` + +## 5. The verdict file + +`.verdicts/.verdict.json` (the `policy:` value). Fields: `result`, `exit_code`, `evaluated_at`, `evaluator`, `pack {run_id, sha256, contract}`, `policy {id, sha256}`, `catalog`, `error`, `failed_gates[]`, `warnings[]`, `errored_gates[]`, and `gates[]` with `{id, metric | expr, where, op, expected, actual, severity, status: pass|fail|warn|errored, message, error}`. To explain a failure, read `gates[]` where `status` is `fail` or `errored` and give `expected` vs `actual`. + +**Where the folder is.** Always beside the pack that was judged; trust `policies[].file` over guessing. + +| Surface | Pack judged | +|---|---| +| `evidence verdict` | the pack you named (`latest` → `/.testmuai/evidence/.evidence`) | +| `testmd run` | the session-directory pack `~/.testmuai/kaneai/sessions//evidence/.evidence` (not the `.testmuai/evidence/` copy) | +| `testrun run` (local or `--remote`) | the suite's pack in `/.testmuai/evidence/` | +| `evidence merge` | the merged pack (`-o`, default `/.testmuai/evidence/.evidence`) | + +Judging again overwrites that policy's file; the pack never changes. + +## 6. Presenting results + +Never show event or field names. One row in the result card: + +```markdown +| 🔴 **Policy** | release-gate failed: 2 tests failed (expected 0) · 1 warning: a test needed a heal | +``` + +Use 🟢 pass, 🔴 fail, 🟡 errored (say *why* it couldn't be judged — a missing value, a pack that didn't seal — not "failed"). When the tests passed but the policy failed, say both plainly: "All 12 tests passed, but the release-gate policy failed: …". Offer to open the evidence (`references/evidence.md`). + +## 7. Common mistakes + +- **`kane-cli run` has no `--policy`.** Judge a one-off run afterwards: `kane-cli evidence verdict --policy ` (the pack path is in the stderr evidence hint). +- **Built-in metrics never take `metrics.`** (`totals.failed`, not `metrics.totals.failed`); kane-cli catalog metrics always do (`metrics.flaky_rate`). Lint reports `policy.builtin_prefixed` / `policy.unknown_metric`. +- **Two operators on one gate**, or `where` on an expression gate / a non-per-test metric: lint error. +- **`min`/`max` on a ratio means 0–1**, not percent: `min: 0.95`, not `95`. +- **Judge sealed packs only.** An unsealed pack directory cannot be judged (the verdict errors), and `evidence merge --no-finalize` with a policy is refused. +- **Lint before you run** a newly written policy: `kane-cli evidence lint-policy --json`, then fix every `diagnostics[]` entry (`location`, `message`, `code`, `hint`). Runs lint for you, but a lint error there costs an exit 2 the person sees. +- **Don't parse `testrun_done.overall_status` for the gate** — it is the tests' status. The policy result is only in the `verdict` event, the exit code, and the files. +- **`--policy-id` needs a sign-in**; a failed sign-in is exit 2 before anything runs. Never ask for an access key in chat (`references/ready-check.md`). +- **Writing a policy for the person**: write the YAML from their words, run `metrics-list` to pick real metrics, lint it, and show them the gates before storing it with `policy create` (which shares it with their Test Manager project). diff --git a/.agents/skills/kane-cli/references/testmd.md b/.agents/skills/kane-cli/references/testmd.md index 919921b..8670745 100644 --- a/.agents/skills/kane-cli/references/testmd.md +++ b/.agents/skills/kane-cli/references/testmd.md @@ -146,6 +146,7 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `--no-adaptive-heal` | healing enabled | Disable adaptive healing after replay failure | | `--author` | off | Force authorable steps to re-author; replay-only steps still replay | | `--bug-detection ` | config (`off`) | Flag suspected product bugs while **authoring** (`stop` halts on a confirmed bug; `continue` records it). Replay failures always investigate regardless. | +| `--policy ` / `--policy-id ` | none | (after 0.10.0-rc.3) Judge the test's sealed pack against a policy YAML / a Test Manager policy (repeatable). Linted before the run (error = exit 2). A passing run exits `1` on a failing verdict, `3` on an errored one; a `verdict` event precedes `test_md_done`. `references/policy.md` | Common flags include `--agent`, `--headless`, `--max-steps`, `--timeout`, and `--variables`. Command-specific flags differ; check `kane-cli testmd run --help`. diff --git a/.agents/skills/kane-cli/references/testrun.md b/.agents/skills/kane-cli/references/testrun.md index 025e62a..948ab72 100644 --- a/.agents/skills/kane-cli/references/testrun.md +++ b/.agents/skills/kane-cli/references/testrun.md @@ -36,6 +36,8 @@ kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON when stdin is not | `--device-name ` | Device for the mobile members: as `kane-cli devices list --target ` prints it locally, or a grid catalog device (`devices list … --remote`) with `--remote`; for real-device members a name or pattern from `devices list --target android\|ios --remote`. Validated before dispatch | members' `device_name:` (real devices: else the hub picks) | | `--os-version ` | OS version for the mobile members (`14`, `17.5`); alone = any device on that version | members' `os_version:` | | `--username` / `--access-key` | Basic auth | active profile | +| `--policy ` | (after 0.10.0-rc.3) Judge the suite's sealed pack against this policy YAML (repeatable; never comma-join). Linted after the plan, dry run too: an error is exit 2 before anything runs. With `--remote` the laptop judges the consolidated pack. `references/policy.md` | — | +| `--policy-id ` | (after 0.10.0-rc.3) The same with a Test Manager policy, latest version (repeatable; id from `kane-cli policy list --agent`) | — | ## Preflight (why members get rejected) @@ -152,6 +154,7 @@ All typed; stdout; one JSON object per line. **Local completion: `testrun_done`. | `testrun_investigations_wait` | `count` | Failed replays left investigations running; the coordinator waits before sealing. Narrate as "investigating N failures". | | `testrun_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` | Pack published to the dashboard. Absent when publish is skipped. | | `testrun_summary` | `totals: {tests, passed, failed, broken, skipped, authored}`, `duration_s`, `upload`, `cancelled`, `execution: {id, status}` | Build the rollup table from this. | +| `verdict` | `evidence_id`, `result: pass\|fail\|errored`, `exit_code`, `policies: [{id, result, failed_gate_ids, errored_gate_ids, file, …}]`, `error?` | Only with `--policy` / `--policy-id`, after the pack is judged and before `testrun_summary`. On `--remote`, before `remote_done`. Schema: `references/policy.md` §4 | | `testrun_done` | `execution_id`, `overall_status: "passed"\|"failed"\|"cancelled"` (`"refused"` for an invalid plan in a linked folder), `coverage?: {status, pct?}` | Local completion; remote runs continue through `remote_done`. `coverage` repeats the run's coverage figure (`pct` only when `status` is `written`). | ### Each test's own log, and `--stream-members` (0.8.17+) @@ -189,6 +192,7 @@ for each line: if type === "warning" && code === "unresolved_variables" → name the variables with no value (references/parsing.md); the run continues; ignore other warning codes if type === "testrun_member_end" or "testrun_authored_member_end" → note per-member outcome if type === "testrun_summary" → capture totals for the rollup + if type === "verdict" → capture the policy result (references/policy.md §4); testrun_done stays the tests' status else → informational; narrate sparingly ``` @@ -212,6 +216,8 @@ Don't list passing tests individually. If the pack published, mention the run is `0` all members passed · `1` at least one failed/broken · `2` usage / invalid plan / auth (nothing ran) · `3` cancelled (Ctrl-C — no new members start, running ones are stopped, and the pack still seals with what finished). +With `--policy` / `--policy-id`: a bad policy is `2` before anything runs; when every member passed, a failing verdict exits `1` and an errored verdict `3`; a run that already failed or was cancelled keeps its code. Tell an errored verdict from a cancel by the `verdict` event. + ## Evidence & debugging The suite produces **one** sealed pack, created directly in `/.testmuai/evidence/`. Offer it to the user per `references/evidence.md`. Members run silently by design; to see per-member output while debugging, set `KANE_TESTRUN_MEMBER_DEBUG=1` (routes member events to stderr, prefixed `[member]`). diff --git a/.claude/skills/kane-cli/SKILL.md b/.claude/skills/kane-cli/SKILL.md index 1aebb55..20d7f87 100644 --- a/.claude/skills/kane-cli/SKILL.md +++ b/.claude/skills/kane-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: kane-cli -description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover. Work in a team's shared Test Manager project from a linked folder: link it with --project, save sessions, run the designed tests on their Test Manager cases, and read the project's coverage. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share a local assurance store (.context/) through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." +description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover, and judge a pack against a policy of quality gates (kane-cli evidence verdict, --policy) to pass or fail a CI run. Work in a team's shared Test Manager project from a linked folder: link it with --project, save sessions, run the designed tests on their Test Manager cases, and read the project's coverage. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share a local assurance store (.context/) through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." --- # Kane CLI — End-to-End Test Generation and Execution Skill @@ -188,6 +188,7 @@ When the user's request involves a browser — or writing test cases: - Share a **local** context store through a repository, a bucket or a folder, join one, keep two stores level, or resolve a sync conflict → Read `references/context-sync.md` first (kane-cli 0.8.14+). A team in Test Manager uses a shared project instead (`references/shared-projects.md`); a linked folder refuses `kane-cli context sync`, `kane-cli context push` and `kane-cli context clone` (`REMOTE_UNSUPPORTED`), and a location pull (a location name, `--rebase`, `--yes` or `--answer`: `PULL_LOCATION_LINKED`); a plain `kane-cli context pull --mode agent` there brings the project's tests (`references/shared-projects.md` §7); the store is shared through a location, never by copying or git-merging `.context/` - Multiple independent browser tasks → Read `references/parallel.md` first - View, share, or validate run evidence (`.evidence` packs) → Read `references/evidence.md` +- Judge a run or pack against quality gates (a policy YAML, `--policy`, `--policy-id`, `kane-cli evidence verdict`, `kane-cli policy …`), or gate CI on more than pass/fail → Read `references/policy.md` first (after 0.10.0-rc.3) - Debug a failed run → Read `references/debug.md` - Configure kane-cli or check directory layout → Read `references/setup-and-config.md` - Browse / create / pick a Test Manager project or folder for `kane-cli run`, or interpret the auto-default event → Read `references/test-manager.md`. Linking an assurance folder to a project is different: `references/shared-projects.md` §2–3 @@ -382,6 +383,7 @@ A draft in chat cannot be run, and a hand-written `_test.md` carries no requirem | Share the context store with a team, join one, keep stores level, or resolve a sync conflict (kane-cli 0.8.14+) | `references/context-sync.md` | | Run failed, need to diagnose | `references/debug.md` | | View, share, validate, or merge evidence packs | `references/evidence.md` | +| Judge a pack against a policy, run with `--policy`, or store/list policies in Test Manager | `references/policy.md` | | Multiple independent browser tasks | `references/parallel.md` | | Need full NDJSON event schema (`run`) | `references/parsing.md` | | Browse / create projects or folders, or parse the auto-default event | `references/test-manager.md` | diff --git a/.claude/skills/kane-cli/references/evidence.md b/.claude/skills/kane-cli/references/evidence.md index fbe6643..ba86a3e 100644 --- a/.claude/skills/kane-cli/references/evidence.md +++ b/.claude/skills/kane-cli/references/evidence.md @@ -1,4 +1,4 @@ - + # Evidence Packs — Agent Surface @@ -69,7 +69,18 @@ Exit `0` valid · `1` invalid · `2` not found. Use it when a pack won't open in kane-cli evidence merge --run-id ``` -Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/.evidence`. `--run-id` is required. Exit `0` merged · `1` policy abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · `2` usage. Full policy detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. +Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/.evidence`. `--run-id` is required. Exit `0` merged · `1` merge-rules abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · `2` usage. Full merge-rules detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. + +After 0.10.0-rc.3, `--policy ` / `--policy-id ` (repeatable) judge the sealed merged pack: exit `1` then also means a failing verdict (pack and verdict files written), `3` an errored verdict; `--json` adds `verdict {…}` to the report; `--no-finalize` with a policy is exit 2. See `references/policy.md`. + +## Verdict — judge a pack against a policy (after 0.10.0-rc.3) + +```bash +kane-cli evidence lint-policy --json +kane-cli evidence verdict --policy [--policy-id ] --json +``` + +Writes `.verdicts/.verdict.json` beside the pack (never inside it). Exit `0` pass · `1` fail · `3` errored · `2` usage or I/O. Policy format, metrics, the `verdict` event, run-time `--policy` and Test Manager policies: `references/policy.md`. ## Debugging with a pack diff --git a/.claude/skills/kane-cli/references/policy.md b/.claude/skills/kane-cli/references/policy.md new file mode 100644 index 0000000..0fcd70b --- /dev/null +++ b/.claude/skills/kane-cli/references/policy.md @@ -0,0 +1,132 @@ + + +# Policies and Verdicts — Agent Surface + +**Version.** Needs the kane-cli release after 0.10.0-rc.3. If `kane-cli policy metrics-list` is an unknown command, or `--policy` is an unknown option, the installed build predates policies: say so and stop; do not fake a verdict by reading the pack yourself. + +A **policy** is a YAML file of **gates** over a sealed evidence pack's metrics. A **verdict** is one policy judged against one pack: `pass`, `fail` or `errored`. The pack is never modified; each verdict is a JSON file beside it at `.verdicts/.verdict.json`. + +Use a policy when the person wants a run judged on more than pass/fail: "no broken tests", "pass rate at least 95% on checkout", "every test must assert something", "fail CI if a test needed a heal". + +## 1. Commands + +| Command | What it does | +|---|---| +| `kane-cli policy metrics-list [--pack ] [--env prod\|stage] [--agent]` | Every metric a policy can gate on: built-in metrics, then the kane-cli catalog as `metrics.`. `--pack` = what one pack supports (the catalog it was sealed with). | +| `kane-cli evidence lint-policy [--json] [--env ]` | Check policy files against the format and the kane-cli catalog. Prints every problem. | +| `kane-cli evidence verdict (--policy \| --policy-id )... [--json] [--env ]` | Judge one sealed pack. `` = execution id, `.evidence` path, or `latest` (newest sealed pack in `/.testmuai/evidence/` by mtime). | +| `kane-cli testmd run --policy / --policy-id ` | Lint before the run, judge the test's own pack after it seals. | +| `kane-cli testrun run … --policy / --policy-id < /dev/null` | Lint after the plan (dry run too), judge the suite's pack. `--remote`: the laptop judges the consolidated pack; the grid never gets the flag. | +| `kane-cli evidence merge --run-id --policy / --policy-id ` | Judge the sealed merged pack. Files are read but **not** linted first (a bad policy = an errored verdict). `--no-finalize` with a policy = exit 2. | +| `kane-cli policy list [--project \| --all-projects] [--search ] [--limit 1-200] [--page ] [--agent]` | Policies in Test Manager (latest versions). Default scope: the profile's project. | +| `kane-cli policy create [--project ] [--agent]` | Lint, then store in Test Manager. New content under an existing `policy:` id = a new version; identical content = the stored version (`created: false`). Project: `--project`, else the profile's (`kane-cli config project `); none = exit 2. | + +`policy list` / `policy create` also take `--profile`, `--env`, `--username`, `--access-key`. `evidence verdict --policy-id` signs in with the active profile; `testmd run` / `testrun run` use the run's `--username` / `--access-key` when given. + +**Flag rules.** `--policy` and `--policy-id` are repeatable and combine. Repeat the flag per policy; never comma-join (`--policy a.yaml,b.yaml` is one file name). Two policies with the same `policy:` id in one command are refused. `--policy-id` takes the id `policy list` prints (the `ID` column / `uid`), not the YAML name, and always judges the **latest version**. + +## 2. Policy YAML + +```yaml +policy: release-gate # required; names the verdict file; ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ +requires: # optional: refuse packs with another catalog / version + catalog: kane-cli + version: ">=1" +gates: # required, at least one; at most 100; file at most 64 KB + - { id: no-failed, metric: totals.failed, equals: 0, severity: blocking } + - { id: checkout-rate, metric: pass_rate, where: "'checkout' in test.tags", min: 0.95, severity: blocking } + - { id: every-test-checks, metric: metrics.tests_without_checks, equals: 0, severity: blocking } + - { id: no-heals, metric: metrics.heal.healed_tests, max: 0, severity: warning, message: Review the heal } + - { id: no-flaky-critical, expr: "tests.filter(t, 'critical' in t.tags).all(t, !t.flaky)", severity: warning } +``` + +- A gate is **either** `metric` + exactly one of `equals` / `min` / `max` / `range: [lo, hi]` / `in: [...]`, **or** `expr` (CEL over `run`, `tests`, `totals`, `failures`, must be boolean) with no metric/operator/where. +- `severity`: `blocking` fails the verdict; `warning` reports (`status: warn`) without failing. An **errored** gate errors the verdict whatever its severity. +- `where` (CEL over one `test`) only on per-test metrics. +- Unknown keys are refused. +- Operators by type: counts/integers/ms/bytes/currency → all five; ratio/percentage/decimal → `min`/`max`/`range`; string/enum → `equals`/`in`. + +**Metrics.** Built-in (no prefix): `totals.tests`, `totals.passed`, `totals.failed`, `totals.broken`, `totals.skipped`, `pass_rate` (ratio; errors when every test skipped), `steps.total`, `steps.failed`, `steps.skipped` (all per test), `run.duration_ms`, `run.status` (enum `running|finalized|aborted`), `failures.count`. kane-cli catalog v1 (always `metrics.` prefix, all per test): `tests_without_checks`, `heal.healed_tests`, `heal.exhausted_tests`, `heal.reauthored_steps`, `non_replay_tests` (counts), `flaky_rate` (ratio). Run `kane-cli policy metrics-list --agent` for the installed truth before writing a gate; never invent a metric name. + +**Values.** A missing value is never zero: no value for the metric, or a pack older than the metric needs, **errors** the gate. A stored number that disagrees with its recompute also errors it. + +## 3. Exit codes + +| Command | 0 | 1 | 2 | 3 | +|---|---|---|---|---| +| `evidence verdict` | all pass | a verdict failed (none errored) | usage / I/O: no policy given, no pack, unreadable or unfetchable policy, a policy that doesn't check clean against the pack's catalog, unreadable pack, verdict files not written | a verdict errored | +| `evidence lint-policy` | clean (warnings allowed) | any error | file unreadable or lint threw | — | +| `testmd run` / `testrun run` with a policy | passed and verdict pass | tests failed, **or** tests passed and verdict failed | bad policy before anything ran (lint error, unknown `--policy-id`, sign-in failure), plus the command's usual 2s | verdict errored (tests passed), **or** cancelled / timed out | +| `evidence merge` with a policy | merged, verdict pass | merge-rules abort (no pack) or verdict failed (pack + verdicts written) | usage (incl. `--no-finalize`), verdict files not written | verdict errored | +| `policy metrics-list` | ok | — | unknown pack / catalog won't load | — | +| `policy list` | ok | Test Manager failure | bad flag / sign-in | — | +| `policy create` | created or already stored | Test Manager failure | no project, unreadable, lint errors, rejected by Test Manager (problems printed), sign-in | — | + +A verdict only changes a **clean 0**: failing → 1, errored → 3. A run that already failed or was cancelled keeps its own code. `test_md_done.overall_status` / `testrun_done.overall_status` stay the **tests'** status: read the `verdict` event for the policy result. Exit 3 is ambiguous on runs (errored verdict vs cancel/timeout): the `verdict` event decides it. + +## 4. The `verdict` event (agent mode) + +`testmd run`, `testrun run` (local and `--remote`) emit one typed line on stdout after the pack is judged and **before** the terminal event (`test_md_done`; `testrun_summary`/`testrun_done`; `remote_done`, whose `exit` already includes the verdict): + +```json +{"type":"verdict","evidence_id":"","result":"fail","exit_code":1, + "policies":[{"id":"release-gate","sha256":"…","result":"fail","exit_code":1, + "counts":{"total":5,"failed":1,"warned":0,"errored":0}, + "failed_gate_ids":["no-failed"],"errored_gate_ids":[], + "file":"/abs/path/.evidence.verdicts/release-gate.verdict.json"}], + "v":1,"ts":"…"} +``` + +- Combined `result`: errored if any policy errored, else fail if any failed, else pass. +- `error` (top level): nothing could be judged (e.g. the pack did not seal) — `policies` is empty. +- `policies[].error`: that verdict errored as a whole — `{code, message, hint}`. +- `files_error`: verdicts were made but files could not be written; the run warns and continues (verdicts still upload with the evidence). +- In a terminal the same information is one stderr line: `verdict: fail — release-gate fail (no-failed)`. + +`evidence verdict --json` prints the same object without `type`, plus `verdicts: []`. `evidence merge --json` adds it to the merge report as `verdict: {…}`. + +Parsing: + +```text +if type === "verdict": + policy_result = result # pass | fail | errored + for p in policies: note p.id, p.result, p.failed_gate_ids, p.errored_gate_ids, p.file + if error: the pack could not be judged — report error, not a gate failure +``` + +## 5. The verdict file + +`.verdicts/.verdict.json` (the `policy:` value). Fields: `result`, `exit_code`, `evaluated_at`, `evaluator`, `pack {run_id, sha256, contract}`, `policy {id, sha256}`, `catalog`, `error`, `failed_gates[]`, `warnings[]`, `errored_gates[]`, and `gates[]` with `{id, metric | expr, where, op, expected, actual, severity, status: pass|fail|warn|errored, message, error}`. To explain a failure, read `gates[]` where `status` is `fail` or `errored` and give `expected` vs `actual`. + +**Where the folder is.** Always beside the pack that was judged; trust `policies[].file` over guessing. + +| Surface | Pack judged | +|---|---| +| `evidence verdict` | the pack you named (`latest` → `/.testmuai/evidence/.evidence`) | +| `testmd run` | the session-directory pack `~/.testmuai/kaneai/sessions//evidence/.evidence` (not the `.testmuai/evidence/` copy) | +| `testrun run` (local or `--remote`) | the suite's pack in `/.testmuai/evidence/` | +| `evidence merge` | the merged pack (`-o`, default `/.testmuai/evidence/.evidence`) | + +Judging again overwrites that policy's file; the pack never changes. + +## 6. Presenting results + +Never show event or field names. One row in the result card: + +```markdown +| 🔴 **Policy** | release-gate failed: 2 tests failed (expected 0) · 1 warning: a test needed a heal | +``` + +Use 🟢 pass, 🔴 fail, 🟡 errored (say *why* it couldn't be judged — a missing value, a pack that didn't seal — not "failed"). When the tests passed but the policy failed, say both plainly: "All 12 tests passed, but the release-gate policy failed: …". Offer to open the evidence (`references/evidence.md`). + +## 7. Common mistakes + +- **`kane-cli run` has no `--policy`.** Judge a one-off run afterwards: `kane-cli evidence verdict --policy ` (the pack path is in the stderr evidence hint). +- **Built-in metrics never take `metrics.`** (`totals.failed`, not `metrics.totals.failed`); kane-cli catalog metrics always do (`metrics.flaky_rate`). Lint reports `policy.builtin_prefixed` / `policy.unknown_metric`. +- **Two operators on one gate**, or `where` on an expression gate / a non-per-test metric: lint error. +- **`min`/`max` on a ratio means 0–1**, not percent: `min: 0.95`, not `95`. +- **Judge sealed packs only.** An unsealed pack directory cannot be judged (the verdict errors), and `evidence merge --no-finalize` with a policy is refused. +- **Lint before you run** a newly written policy: `kane-cli evidence lint-policy --json`, then fix every `diagnostics[]` entry (`location`, `message`, `code`, `hint`). Runs lint for you, but a lint error there costs an exit 2 the person sees. +- **Don't parse `testrun_done.overall_status` for the gate** — it is the tests' status. The policy result is only in the `verdict` event, the exit code, and the files. +- **`--policy-id` needs a sign-in**; a failed sign-in is exit 2 before anything runs. Never ask for an access key in chat (`references/ready-check.md`). +- **Writing a policy for the person**: write the YAML from their words, run `metrics-list` to pick real metrics, lint it, and show them the gates before storing it with `policy create` (which shares it with their Test Manager project). diff --git a/.claude/skills/kane-cli/references/testmd.md b/.claude/skills/kane-cli/references/testmd.md index 919921b..8670745 100644 --- a/.claude/skills/kane-cli/references/testmd.md +++ b/.claude/skills/kane-cli/references/testmd.md @@ -146,6 +146,7 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `--no-adaptive-heal` | healing enabled | Disable adaptive healing after replay failure | | `--author` | off | Force authorable steps to re-author; replay-only steps still replay | | `--bug-detection ` | config (`off`) | Flag suspected product bugs while **authoring** (`stop` halts on a confirmed bug; `continue` records it). Replay failures always investigate regardless. | +| `--policy ` / `--policy-id ` | none | (after 0.10.0-rc.3) Judge the test's sealed pack against a policy YAML / a Test Manager policy (repeatable). Linted before the run (error = exit 2). A passing run exits `1` on a failing verdict, `3` on an errored one; a `verdict` event precedes `test_md_done`. `references/policy.md` | Common flags include `--agent`, `--headless`, `--max-steps`, `--timeout`, and `--variables`. Command-specific flags differ; check `kane-cli testmd run --help`. diff --git a/.claude/skills/kane-cli/references/testrun.md b/.claude/skills/kane-cli/references/testrun.md index 025e62a..948ab72 100644 --- a/.claude/skills/kane-cli/references/testrun.md +++ b/.claude/skills/kane-cli/references/testrun.md @@ -36,6 +36,8 @@ kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON when stdin is not | `--device-name ` | Device for the mobile members: as `kane-cli devices list --target ` prints it locally, or a grid catalog device (`devices list … --remote`) with `--remote`; for real-device members a name or pattern from `devices list --target android\|ios --remote`. Validated before dispatch | members' `device_name:` (real devices: else the hub picks) | | `--os-version ` | OS version for the mobile members (`14`, `17.5`); alone = any device on that version | members' `os_version:` | | `--username` / `--access-key` | Basic auth | active profile | +| `--policy ` | (after 0.10.0-rc.3) Judge the suite's sealed pack against this policy YAML (repeatable; never comma-join). Linted after the plan, dry run too: an error is exit 2 before anything runs. With `--remote` the laptop judges the consolidated pack. `references/policy.md` | — | +| `--policy-id ` | (after 0.10.0-rc.3) The same with a Test Manager policy, latest version (repeatable; id from `kane-cli policy list --agent`) | — | ## Preflight (why members get rejected) @@ -152,6 +154,7 @@ All typed; stdout; one JSON object per line. **Local completion: `testrun_done`. | `testrun_investigations_wait` | `count` | Failed replays left investigations running; the coordinator waits before sealing. Narrate as "investigating N failures". | | `testrun_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` | Pack published to the dashboard. Absent when publish is skipped. | | `testrun_summary` | `totals: {tests, passed, failed, broken, skipped, authored}`, `duration_s`, `upload`, `cancelled`, `execution: {id, status}` | Build the rollup table from this. | +| `verdict` | `evidence_id`, `result: pass\|fail\|errored`, `exit_code`, `policies: [{id, result, failed_gate_ids, errored_gate_ids, file, …}]`, `error?` | Only with `--policy` / `--policy-id`, after the pack is judged and before `testrun_summary`. On `--remote`, before `remote_done`. Schema: `references/policy.md` §4 | | `testrun_done` | `execution_id`, `overall_status: "passed"\|"failed"\|"cancelled"` (`"refused"` for an invalid plan in a linked folder), `coverage?: {status, pct?}` | Local completion; remote runs continue through `remote_done`. `coverage` repeats the run's coverage figure (`pct` only when `status` is `written`). | ### Each test's own log, and `--stream-members` (0.8.17+) @@ -189,6 +192,7 @@ for each line: if type === "warning" && code === "unresolved_variables" → name the variables with no value (references/parsing.md); the run continues; ignore other warning codes if type === "testrun_member_end" or "testrun_authored_member_end" → note per-member outcome if type === "testrun_summary" → capture totals for the rollup + if type === "verdict" → capture the policy result (references/policy.md §4); testrun_done stays the tests' status else → informational; narrate sparingly ``` @@ -212,6 +216,8 @@ Don't list passing tests individually. If the pack published, mention the run is `0` all members passed · `1` at least one failed/broken · `2` usage / invalid plan / auth (nothing ran) · `3` cancelled (Ctrl-C — no new members start, running ones are stopped, and the pack still seals with what finished). +With `--policy` / `--policy-id`: a bad policy is `2` before anything runs; when every member passed, a failing verdict exits `1` and an errored verdict `3`; a run that already failed or was cancelled keeps its code. Tell an errored verdict from a cancel by the `verdict` event. + ## Evidence & debugging The suite produces **one** sealed pack, created directly in `/.testmuai/evidence/`. Offer it to the user per `references/evidence.md`. Members run silently by design; to see per-member output while debugging, set `KANE_TESTRUN_MEMBER_DEBUG=1` (routes member events to stderr, prefixed `[member]`). diff --git a/docs/user-guide/README.md b/docs/user-guide/README.md index 6357064..89a9329 100644 --- a/docs/user-guide/README.md +++ b/docs/user-guide/README.md @@ -61,6 +61,8 @@ Every page below is standalone — start wherever your job starts. ## 3. Prove and maintain - [Evidence packs](./evidence.md) — the sealed proof every run produces: screenshots, logs, results, the viewer. + +- [Policies and verdicts](./policies.md) *(next release after 0.10.0-rc.3)*: judge a pack against quality gates (no broken tests, a pass-rate floor, every test asserting something) and fail CI on a failed verdict. - [Coverage](./assurance/coverage.md) — what execution proved vs what the design still owes. - [Maintaining the suite](./assurance/maintain.md) — reconcile the suite when a requirement document changes. diff --git a/docs/user-guide/cicd.md b/docs/user-guide/cicd.md index 1ee01f5..927b4d2 100644 --- a/docs/user-guide/cicd.md +++ b/docs/user-guide/cicd.md @@ -21,6 +21,9 @@ These patterns apply to every CI system; the recipes below differ only in how th The suite becomes one execution with one exit code (`0` all passed, `1` any failure, `2` invalid plan, `3` cancelled) and one sealed [evidence pack](./evidence.md) — archive `.testmuai/evidence/*.evidence` as a CI artifact and anyone can drop it into the viewer. See [Batch runs with testrun](./testrun.md). + +- Gate on quality rules, not only on pass/fail *(next release after 0.10.0-rc.3)*. Add `--policy ` to `testrun run` or `testmd run` and the job fails when the run's pack breaks a gate (a broken test, a pass rate under a floor, a test with no checks). See [Policies and verdicts](./policies.md#in-ci). + - Run the suite on the cloud grid when the runner can't run it. `kane-cli testrun run … --remote` turns the suite into one HyperExecute job: the grid supplies Chrome on a macOS runner — or, for mobile `_test.md` members, a virtual Android emulator or iOS simulator — so the runner needs no Chrome, Xcode, or Android Studio, and `--parallel N` spreads the members across N grid runners. The recordings and the evidence pack come back to the checkout as if the suite had run locally. ```bash diff --git a/docs/user-guide/evidence.md b/docs/user-guide/evidence.md index 944b033..90eceb7 100644 --- a/docs/user-guide/evidence.md +++ b/docs/user-guide/evidence.md @@ -4,7 +4,7 @@ Every kane-cli run produces an **evidence pack**: a single sealed `.evidence` fi A pack is a self-contained zip. You can open it in the hosted viewer, share it with a teammate, archive it in CI, validate it, or merge several packs into one. -This page covers where packs live, how to view them, and the `kane-cli evidence` command family. For debugging a failed run with its pack, see [Debugging a failed run from its pack](#debugging-a-failed-run-from-its-pack). +This page covers where packs live, how to view them, and the `kane-cli evidence` command family. To judge a pack against quality gates, see [Policies and verdicts](./policies.md). For debugging a failed run with its pack, see [Debugging a failed run from its pack](#debugging-a-failed-run-from-its-pack). ## What's inside a pack @@ -171,10 +171,12 @@ Targets are execution ids or pack paths, and **order matters** (earlier targets | `--title ` | Title for the merged run | first eligible pack's | | `--no-finalize` | Keep the merged pack live instead of sealing it | seals by default | | `--json` | Machine-readable merge report | off | +| `--policy <file>` | *(next release after 0.10.0-rc.3)* Judge the sealed merged pack against this policy (repeatable). See [Policies and verdicts](./policies.md) | — | +| `--policy-id <id>` | *(next release after 0.10.0-rc.3)* Judge it against a policy stored in Test Manager, latest version (repeatable) | — | -`--rules` and `--on-collision` are mutually exclusive. The default policy requires inputs to be sealed and valid, skips duplicate run ids, and refuses to merge packs from different projects or organisations. +`--rules` and `--on-collision` are mutually exclusive. The default rules require inputs to be sealed and valid, skips duplicate run ids, and refuses to merge packs from different projects or organisations. -Exit codes: `0` merged · `1` policy abort · `2` usage error. +Exit codes: `0` merged · `1` merge-rules abort, or a failing verdict with `--policy` · `2` usage error · `3` an errored verdict with `--policy`. ## Debugging a failed run from its pack @@ -187,6 +189,18 @@ The pack is the fastest way to understand a failure, because everything is in on If a pack won't open in the viewer, run `kane-cli evidence validate <pack>` — an unsealed or truncated pack (for example from a run that was killed hard) reports as invalid, and the run's session directory still holds the raw logs. +<!-- release: needs evidence policies --> +## Judging a pack against a policy *(next release after 0.10.0-rc.3)* + +A policy is a YAML file of gates over a pack's metrics. `kane-cli evidence verdict` judges a sealed pack against one or more policies and writes each verdict next to the pack, in `<pack>.verdicts/<policy-id>.verdict.json`. The pack itself is never changed. + +```bash +kane-cli evidence lint-policy release-policy.yaml +kane-cli evidence verdict latest --policy release-policy.yaml +``` + +Exit codes: `0` pass · `1` fail · `3` errored · `2` usage or I/O error. `testmd run`, `testrun run` and `evidence merge` take the same `--policy` and `--policy-id` flags to judge their own pack. The format, the metrics, and storing policies in Test Manager: [Policies and verdicts](./policies.md). + ## Publishing to the dashboard Runs that upload to Test Manager attach their sealed pack automatically. Replayed `testmd` runs and `testrun` executions publish their packs to your project as well, so the dashboard shows execution evidence even for cache-replayed runs that never re-author. @@ -194,6 +208,7 @@ Runs that upload to Test Manager attach their sealed pack automatically. Replaye ## Next steps - [Batch runs with testrun](./testrun.md) — many tests, one execution, one pack. +- [Policies and verdicts](./policies.md) — judge a pack against quality gates. - [Running test.md files](./testmd/running.md) — the testmd command family. - [Troubleshooting](./troubleshooting.md) — debugging flows and escape hatches. diff --git a/docs/user-guide/policies.md b/docs/user-guide/policies.md new file mode 100644 index 0000000..6f918bd --- /dev/null +++ b/docs/user-guide/policies.md @@ -0,0 +1,457 @@ +# Policies and verdicts + +<!-- release: needs evidence policies --> +> Needs the kane-cli release after **0.10.0-rc.3** (`kane-cli --version`). + +A run tells you whether each test passed. A **policy** tells you whether the run as a whole is good enough: no broken tests, a pass rate above a floor, every test asserting something, no healed tests. kane-cli judges a sealed [evidence pack](./evidence.md) against a policy and writes the outcome, the **verdict**, to a file next to the pack. + +- A policy is a small YAML file of **gates**. Each gate checks one metric of the pack. +- A verdict is one policy judged against one pack. It is `pass`, `fail` or `errored`. +- The pack is never changed. You can judge any sealed pack, at any time, against any number of policies. + +You can judge a pack after the fact (`kane-cli evidence verdict`), or ask a run to judge its own pack (`--policy` on `testmd run`, `testrun run` and `evidence merge`). Either way, a failing verdict gives a non-zero exit code, so a CI job can gate on it. + +## A first policy + +Save this as `release-policy.yaml`: + +```yaml +policy: release-gate +gates: + - id: no-failed-tests + metric: totals.failed + equals: 0 + severity: blocking + message: Every test must pass. + - id: no-broken-tests + metric: totals.broken + equals: 0 + severity: blocking + - id: checkout-pass-rate + metric: pass_rate + where: "'checkout' in test.tags" + min: 0.95 + severity: blocking + message: Checkout tests must pass at least 95% of the time. + - id: every-test-checks + metric: metrics.tests_without_checks + equals: 0 + severity: blocking + message: Every test must assert something. + - id: no-healed-tests + metric: metrics.heal.healed_tests + max: 0 + severity: warning + message: A healed test means the UI moved under a recorded step. Review the heal. +``` + +Check it, then judge the newest pack in the project with it: + +```bash +kane-cli evidence lint-policy release-policy.yaml +kane-cli evidence verdict latest --policy release-policy.yaml +``` + +```text +verdict: fail — release-gate fail (no-failed-tests) + /path/to/project/.testmuai/evidence/<execution_id>.evidence.verdicts/release-gate.verdict.json +``` + +The exit code is `1` because a blocking gate failed. + +## The policy format + +A policy is a YAML mapping. Unknown keys are refused, so a typo fails loudly instead of being ignored. + +| Key | Required | Meaning | +|---|---|---| +| `policy` | yes | The policy id. It names the verdict file (`<policy>.verdict.json`), so it must start with a letter or digit and use only letters, digits, `.`, `_` and `-` (at most 128 characters). | +| `gates` | yes | A list of at least one gate. | +| `requires` | no | `catalog: <id>` and an optional `version: "<comparators>"` (for example `">=1"` or `">=1 <4"`). A pack whose metric catalog is a different one, or a version outside the range, is refused. | + +### Gates + +A gate is either a **metric gate** (one metric and exactly one operator) or an **expression gate** (a CEL expression that must be true). + +| Key | Meaning | +|---|---| +| `id` | Gate id, unique within the policy. Same character rule as `policy`. | +| `severity` | `blocking`: a failure fails the verdict. `warning`: a failure is reported but does not fail the verdict. | +| `metric` | A built-in metric (`totals.failed`) or a kane-cli metric (`metrics.tests_without_checks`). See [Metrics](#metrics). | +| `equals` | Pass when the metric equals this number or string. | +| `min` | Pass when the metric is at least this number. | +| `max` | Pass when the metric is at most this number. | +| `range` | `[lo, hi]`: pass when `lo <= metric <= hi`. | +| `in` | A list: pass when the metric is one of these values. | +| `where` | Metric gates on a per-test metric only: a CEL expression over one test (`test`) that narrows the metric to the tests it holds for, for example `"'checkout' in test.tags"`. | +| `expr` | Expression gates only: a CEL expression over the pack (`run`, `tests`, `totals`, `failures`) that evaluates to a boolean. An expression gate takes no `metric`, `where` or operator. | +| `message` | Optional text, copied into the gate's result. | + +Which operators a metric allows depends on its type: + +| Metric type | Operators | +|---|---| +| `count`, `integer`, `ms`, `bytes`, `currency` | `equals`, `min`, `max`, `range`, `in` | +| `ratio`, `percentage`, `decimal` | `min`, `max`, `range` | +| `string`, `enum` | `equals`, `in` | + +An expression gate: + +```yaml + - id: no-flaky-critical + expr: "tests.filter(t, 'critical' in t.tags).all(t, !t.flaky)" + severity: warning +``` + +### Limits + +kane-cli refuses a policy before it judges anything when: + +- the file is over 64 KB, or has more than 100 gates; +- its `policy` id breaks the character rule above; +- two policies given to one command share a `policy` id (each verdict needs its own file name). + +### How values are read + +Values are recomputed from the pack's own files where possible. A stored number that disagrees with its recompute **errors** the gate rather than passing it. A missing value is never read as zero: a test with no value for the metric, or a pack written by a kane-cli older than the metric needs, errors the gate. An errored gate errors the whole verdict, whatever its severity. + +## Metrics + +A policy can gate on two kinds of metric: + +- **Built-in metrics**, which every pack has. Use them by their own name. +- **kane-cli metrics**, from the metric catalog kane-cli seals into every pack (`meta/metrics-catalog.yaml`). Always prefix them with `metrics.`. + +List every metric your kane-cli can gate on: + +```bash +kane-cli policy metrics-list +``` + +```text +METRIC TYPE UNIT LABEL +totals.tests count Tests +totals.failed count Failed tests +... +metrics.tests_without_checks count tests Tests without checks +... + +Built-in metrics: evidence-core 0.1. Producer metrics: kane-cli catalog v1 (controller). +``` + +Add `--pack <execution-id|path|latest>` to list what one pack supports: the built-in metrics plus the catalog that pack was sealed with. Use it when you judge an old pack. A pack with no catalog supports only the built-in metrics. + +### Built-in metrics + +| Metric | Type | Per test | Meaning | +|---|---|---|---| +| `totals.tests` | count | yes | Tests in the pack. | +| `totals.passed` | count | yes | Passed tests. | +| `totals.failed` | count | yes | Failed tests. | +| `totals.broken` | count | yes | Broken tests: the test could not be evaluated. | +| `totals.skipped` | count | yes | Skipped tests. | +| `pass_rate` | ratio | yes | passed ÷ (tests − skipped). Errors when every test was skipped. | +| `steps.total` | count | yes | Steps. | +| `steps.failed` | count | yes | Failed steps. | +| `steps.skipped` | count | yes | Skipped steps. | +| `run.duration_ms` | ms | no | The run's end time minus its start time. | +| `run.status` | enum | no | `running`, `finalized` or `aborted`. | +| `failures.count` | count | no | Rows in the pack's failure index. | + +"Per test" metrics take a `where`. + +### kane-cli metrics (catalog version 1) + +| Metric | Type | Meaning | +|---|---|---| +| `metrics.tests_without_checks` | count | Tests with no assert step. Analyze steps that only read a value do not count as checks. | +| `metrics.heal.healed_tests` | count | Tests that failed on replay and passed after adaptive heal re-authored them. | +| `metrics.heal.exhausted_tests` | count | Tests that still failed after adaptive heal used its retries. | +| `metrics.heal.reauthored_steps` | count | Steps adaptive heal re-authored, summed over the tests where a heal ran. `--author` runs are left out. | +| `metrics.non_replay_tests` | count | Tests whose run was not a replay, so the result rests on an authoring run. | +| `metrics.flaky_rate` | ratio | Tests whose attempts disagreed, divided by all tests. | + +kane-cli downloads the current catalog when it can and falls back to a cached or built-in copy. `kane-cli policy metrics-list` always shows what your installation will use. + +## Checking a policy: `kane-cli evidence lint-policy` + +```bash +kane-cli evidence lint-policy release-policy.yaml smoke-policy.yaml +``` + +```text +release-policy.yaml: ok +error: smoke-policy.yaml#/gates/0/metric: <message> (policy.unknown_metric) +``` + +Lint checks each file against the policy format and the kane-cli metric catalog: YAML that does not parse, unknown keys, a missing operator or two operators on one gate, duplicate gate ids, an unknown metric, a built-in metric written with the `metrics.` prefix, an operator the metric's type does not allow, a threshold of the wrong type, a `where` on a metric that is not per test, a CEL expression that does not parse or is not a boolean, the size and gate limits, and the id rules. It prints every problem in every file, each with its location (`<file>#/gates/0/metric`) and a code such as `policy.unknown_metric`. + +| Flag | Meaning | Default | +|---|---|---| +| `--json` | One JSON object: `{"valid": true\|false, "files": [{"path", "diagnostics": [...]}]}` | off | +| `--env <name>` | Environment whose metric catalog to check against (`prod` or `stage`) | your default environment | + +| Exit | Meaning | +|---|---| +| `0` | Every file is clean (warnings are allowed). | +| `1` | At least one file has an error. | +| `2` | A file could not be read, or the lint itself failed. | + +You rarely need to run lint by hand before a run: `testmd run`, `testrun run` and `policy create` lint every policy first and stop with exit `2` on an error. + +## Judging a pack: `kane-cli evidence verdict` + +```bash +kane-cli evidence verdict <pack> --policy <file> [--policy <file> ...] [--policy-id <id> ...] +``` + +`<pack>` is an execution id (looked up in `.testmuai/evidence/`), the path to a sealed `.evidence` file, or `latest` (the newest sealed pack in `.testmuai/evidence/`, by modification time). You need at least one `--policy` or `--policy-id`. + +| Flag | Meaning | +|---|---| +| `--policy <file>` | Policy YAML to judge against. Repeatable. | +| `--policy-id <id>` | A policy stored in Test Manager, at its latest version. Repeatable. See [Policies in Test Manager](#policies-in-test-manager). | +| `--json` | Print one JSON object: the summary and the full verdict documents. | +| `--env <name>` | Environment a `--policy-id` is read from (`prod` or `stage`). Defaults to your default environment. | + +Repeat a flag for each policy (`--policy a.yaml --policy b.yaml`); a comma-separated list is read as one file name. + +A `--policy-id` signs in with your active profile, the same credentials `kane-cli projects list` uses. + +| Exit | Meaning | +|---|---| +| `0` | Every verdict passed. | +| `1` | At least one verdict failed, and none errored. | +| `3` | At least one verdict errored. | +| `2` | Usage or I/O error: no policy given, no such pack, a policy that cannot be read or fetched, a policy that does not check clean against the pack's catalog, a pack that cannot be read, or verdict files that cannot be written. | + +When several policies are given, the combined result is `errored` if any verdict errored, else `fail` if any failed, else `pass`. + +## Judging at run time + +Pass `--policy` or `--policy-id` to a run and it judges its own sealed pack when it ends: + +```bash +kane-cli testmd run tests/checkout_test.md --policy release-policy.yaml +kane-cli testrun run --tags smoke --policy release-policy.yaml --policy-id <id> +kane-cli evidence merge <targets...> --run-id nightly --policy release-policy.yaml +``` + +Both flags are repeatable and can be combined. They work the same on all three commands, with these rules: + +- **Before anything runs**, `testmd run` and `testrun run` read and lint every policy. A policy error, an unknown `--policy-id` or a failed sign-in stops the run with exit `2`, so a policy mistake never costs you a run. Lint warnings print on stderr as `warn: policy …` and the run goes ahead. `testrun run --dry-run` checks the policies too. +- **After the pack seals**, the run judges it and writes the verdict files next to it. In a terminal, one `verdict: …` line prints on stderr. In agent mode, a `verdict` event goes out on stdout before the run's last event. +- **The exit code**: a failing verdict turns a passing run's `0` into `1`, and an errored verdict turns it into `3`. A run that already failed or was cancelled keeps its own exit code. The tests' own status (`test_md_done`, `testrun_done`) is not changed by the verdict. +- **A pack that did not seal** cannot be judged, so the verdict is `errored`. + +Command by command: + +| Command | Which pack is judged | Notes | +|---|---|---| +| `kane-cli testmd run` | The test's pack, in the session directory (`~/.testmuai/kaneai/sessions/<session-id>/evidence/<execution_id>.evidence`) | A replay's evidence upload to Test Manager carries the policies and verdicts. A run that authors or commits a test writes the verdict files and sets the exit code but uploads no verdict. | +| `kane-cli testrun run` | The suite's pack in `.testmuai/evidence/`, the one the run publishes | With `--remote`, the policies are not sent to the grid. Your machine judges the suite's pack once it is back. The policies are never passed on to the individual tests. | +| `kane-cli evidence merge` | The sealed merged pack | Policies are read before the merge but not linted: a policy that does not check clean is an `errored` verdict. `--policy` with `--no-finalize` is a usage error (exit `2`). `--json` adds a `verdict` object to the merge report. Exit `1` means a merge-rules abort or a failing verdict, `3` an errored verdict. | + +`kane-cli run` (a one-off objective) does not take `--policy`. Judge its pack afterwards with `kane-cli evidence verdict`. + +Exit `3` also means "cancelled or timed out" on run commands. When it matters which one happened, read the `verdict` event, or run `evidence verdict` on the pack yourself. + +## Verdict files + +Each verdict is its own JSON file in a folder next to the pack: + +```text +<pack>.evidence +<pack>.evidence.verdicts/ + release-gate.verdict.json + smoke.verdict.json +``` + +The file name is the policy's `policy` id. The verdict file is never written inside the pack, so judging a pack again overwrites the verdict file for that policy and leaves the pack as it was. + +A verdict file looks like this (shortened): + +```json +{ + "result": "fail", + "exit_code": 1, + "evaluated_at": "2026-10-27T10:15:02.000Z", + "evaluator": "evidence-cli 0.2.0", + "pack": { "run_id": "<execution_id>", "sha256": "…", "contract": "…" }, + "policy": { "id": "release-gate", "sha256": "…" }, + "catalog": { "...": "the catalog the pack was judged with" }, + "error": null, + "failed_gates": ["no-failed-tests"], + "warnings": ["no-healed-tests"], + "errored_gates": [], + "gates": [ + { + "id": "no-failed-tests", + "metric": "totals.failed", + "where": null, + "op": "equals", + "expected": 0, + "actual": 2, + "severity": "blocking", + "status": "fail", + "message": "Every test must pass.", + "error": null + } + ] +} +``` + +| Field | Meaning | +|---|---| +| `result` | `pass`, `fail` or `errored`. | +| `exit_code` | `0`, `1` or `3`, matching `result`. | +| `gates[]` | One entry per gate: `status` is `pass`, `fail`, `warn` (a warning gate that failed) or `errored`; `expected` and `actual` show the comparison. | +| `failed_gates`, `warnings`, `errored_gates` | Gate ids by outcome. | +| `error` | Set when the whole verdict errored: `{code, message, hint}`. | +| `policy.sha256` | Hash of the exact policy bytes, so you can tell which version judged the pack. | + +### `--json` output + +`kane-cli evidence verdict --json` prints one object: + +```json +{ + "evidence_id": "<execution_id>", + "result": "fail", + "exit_code": 1, + "policies": [ + { + "id": "release-gate", + "sha256": "…", + "result": "fail", + "exit_code": 1, + "counts": { "total": 5, "failed": 1, "warned": 0, "errored": 0 }, + "failed_gate_ids": ["no-failed-tests"], + "errored_gate_ids": [], + "file": "/path/to/<execution_id>.evidence.verdicts/release-gate.verdict.json" + } + ], + "verdicts": [ { "...": "the full verdict documents" } ] +} +``` + +The `verdict` event a run prints in agent mode has the same fields as this object, with `"type": "verdict"` and without `verdicts`. An `error` field appears when nothing could be judged, and `files_error` when the verdicts were made but their files could not be written (a run warns and goes on; `evidence verdict` and `evidence merge` exit `2`). + +## Policies in Test Manager + +Store a policy in Test Manager once and the whole team judges with the same one, by id, without copying the YAML around. + +### Create a policy + +```bash +kane-cli policy create release-policy.yaml +``` + +```text +Created policy "release-gate" version 1 (<id>). +Judge with it: --policy-id <id> +``` + +`policy create` lints the file first and refuses it (exit `2`) on any error. Creating it again with new content stores a new version under the same id. Creating it again with the same content changes nothing and prints `Test Manager already has this content: …`. + +The policy is scoped to a project: `--project <id>`, else the project your profile is set to (`kane-cli config project <id>`). With neither, the command stops with exit `2`. + +### List policies + +```bash +kane-cli policy list +``` + +```text +ID NAME VERSION GATES LAST RESULT +<id> release-gate 2 5 pass + +1 of 1 +Judge with one: --policy-id <ID> +``` + +The list shows each policy's latest version, for your profile's project. + +| Flag | Meaning | Default | +|---|---|---| +| `--project <id>` | Project whose policies to list | your profile's project | +| `--all-projects` | List every policy in the organization | off | +| `--search <term>` | Search by name | — | +| `--limit <n>` | Policies per page, 1 to 200 | `10` | +| `--page <n>` | Page number | `1` | + +### Judge with a stored policy + +Use `--policy-id <id>` anywhere `--policy <file>` works: `evidence verdict`, `testmd run`, `testrun run` and `evidence merge`. kane-cli fetches the **latest version** of that policy, checks that its content matches the hash Test Manager stored, and judges with it exactly like a file. + +### Shared flags + +`policy list` and `policy create` take the same sign-in flags as `kane-cli projects list`: + +| Flag | Meaning | +|---|---| +| `--profile <name>` | Profile to use (default: the active one) | +| `--env <name>` | Environment (`prod` or `stage`) | +| `--username <user>` / `--access-key <key>` | Basic auth, skipping stored credentials | +| `--agent` | NDJSON output. On by default when stdin or stdout is not a terminal. | + +`policy metrics-list` takes `--pack`, `--env` and `--agent`. + +| Command | Exit `0` | Exit `1` | Exit `2` | +|---|---|---|---| +| `policy metrics-list` | listed | — | unknown pack, or a catalog that cannot be loaded | +| `policy list` | listed | Test Manager request failed | bad flag, or sign-in failed | +| `policy create` | created, or already stored | Test Manager request failed | no project, file unreadable, lint errors, sign-in failed, or Test Manager rejected the policy (its reasons are printed) | + +## In CI + +A run with `--policy` already fails the job on a failing verdict. Keep the verdict files with the packs so you can read them later: + +```yaml + - name: Run the smoke suite and judge it + env: + LT_USERNAME: ${{ secrets.LT_USERNAME }} + LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }} + run: | + kane-cli testrun run --tags smoke --headless \ + --policy .github/kane/release-policy.yaml \ + --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY" < /dev/null + + - name: Keep the evidence and verdicts + if: always() + uses: actions/upload-artifact@v4 + with: + name: kane-evidence + path: .testmuai/evidence/ +``` + +`testrun run` writes both the pack and its `.verdicts/` folder in `.testmuai/evidence/`, so one artifact path keeps both. The step fails on exit `1` (a test failed, or a verdict failed), `2` (a bad policy, before anything ran) or `3` (cancelled, or a verdict errored). + +To keep "the tests failed" and "the policy failed" apart, run the suite in one step and judge it in another: + +```yaml + - name: Run the smoke suite + env: + LT_USERNAME: ${{ secrets.LT_USERNAME }} + LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }} + run: kane-cli testrun run --tags smoke --headless --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY" < /dev/null + + - name: Judge the suite + if: always() + run: kane-cli evidence verdict latest --policy .github/kane/release-policy.yaml +``` + +Lint policy files in a pull request that changes them: + +```bash +kane-cli evidence lint-policy .github/kane/*.yaml +``` + +More pipeline recipes: [CI/CD](./cicd.md). + +## Next steps + +- [Evidence packs](./evidence.md): what a pack holds, and how to view, validate and merge packs. +- [Batch runs with testrun](./testrun.md): one pack for a whole suite. +- [Running test.md files](./testmd/running.md): single-test runs and their flags. +- [Test Manager integration](./test-manager-integration.md): projects, folders and uploads. diff --git a/docs/user-guide/running-tests.md b/docs/user-guide/running-tests.md index e05def9..c948390 100644 --- a/docs/user-guide/running-tests.md +++ b/docs/user-guide/running-tests.md @@ -268,6 +268,9 @@ When stdin is not a TTY, kane-cli automatically switches to plain NDJSON mode (t Use these codes to gate downstream CI steps. +<!-- release: needs evidence policies --> +To judge a run's evidence pack against quality gates *(next release after 0.10.0-rc.3)*, see [Policies and verdicts](./policies.md). + ## What you see at the end of a run When a run finishes, kane-cli prints a result summary to the terminal: diff --git a/docs/user-guide/test-manager-integration.md b/docs/user-guide/test-manager-integration.md index e99e901..8ab1772 100644 --- a/docs/user-guide/test-manager-integration.md +++ b/docs/user-guide/test-manager-integration.md @@ -107,6 +107,11 @@ Saving a design session makes one test case per scenario in the folder's project A designed test's first run adds its KaneAI test to that case, and every later run is recorded on it. Coverage counts those runs. If you edit a case's steps in Test Manager, the next save doesn't overwrite them: it lists the case as one it couldn't update. +<!-- release: needs evidence policies --> +## Policies *(next release after 0.10.0-rc.3)* + +Test Manager also stores evidence **policies**: YAML files of quality gates a run's pack is judged against. `kane-cli policy create <file>` stores one in a project, `kane-cli policy list` lists them, and `--policy-id <id>` on `testmd run`, `testrun run`, `evidence verdict` and `evidence merge` judges with the latest version. A replay or a suite run that is judged sends its verdicts to Test Manager with its evidence. See [Policies and verdicts](./policies.md#policies-in-test-manager). + ## Code export Code export converts a completed test case into runnable Playwright code that you can check into your repository or extend by hand. diff --git a/docs/user-guide/testmd/running.md b/docs/user-guide/testmd/running.md index 703f67a..fd70c06 100644 --- a/docs/user-guide/testmd/running.md +++ b/docs/user-guide/testmd/running.md @@ -66,6 +66,8 @@ Every flag accepted by `kane-cli testmd run`: | `--no-adaptive-heal` | flag | healing enabled | Disable adaptive healing after replay failure | | `--author` | flag | off | Re-author authorable steps; replay-only marked steps still replay. | | `--bug-detection <mode>` | `off` \| `stop` \| `continue` | config value (`off`) | Detect product bugs while authoring: `stop` halts the run on a confirmed bug; `continue` records it and keeps going. Overrides `config set-bug-detection`. Authoring steps only — replay failures always investigate. See [Configuration](../configuration.md#bug-detection). | +| `--policy <file>` | path, repeatable | none | *(next release after 0.10.0-rc.3)* Judge the test's sealed evidence pack against this policy YAML. Linted before the run. See [Policies and verdicts](../policies.md). | +| `--policy-id <id>` | string, repeatable | none | *(next release after 0.10.0-rc.3)* Judge it against a policy stored in Test Manager, latest version (`kane-cli policy list`). | Most flags have a frontmatter counterpart with the same name (with underscores). Where both are set, the CLI flag wins — except for `variables`, which the file owns; see [overview.md](./overview.md#variables). @@ -200,6 +202,9 @@ When an `@import` step fails, the badge points at the failing leaf inside the he Use these to gate downstream CI steps. +<!-- release: needs evidence policies --> +With `--policy` or `--policy-id` *(next release after 0.10.0-rc.3)*, a bad policy is exit `2` before any step runs, and a run that passed exits `1` on a failing verdict and `3` on an errored one. See [Policies and verdicts](../policies.md#judging-at-run-time). + ## Lock conflicts Each test in Test Manager has a single-writer lock so two engineers cannot author the same test at the same time. When you run a test and someone else already holds the lock, `--on-lock-conflict` decides what to do: diff --git a/docs/user-guide/testrun.md b/docs/user-guide/testrun.md index 6e0b04e..4a09e7b 100644 --- a/docs/user-guide/testrun.md +++ b/docs/user-guide/testrun.md @@ -114,6 +114,8 @@ kane-cli testrun run --match '^tests/devices/' --parallel 2 | `--device-name <name>` | Device for the suite's mobile members: as `kane-cli devices list` prints it locally, or a grid catalog device with `--remote` | member's `device_name:` | | `--os-version <version>` | OS version for the mobile members (`14`, `17.5`); alone = any device on that version | member's `os_version:` | | `--username <user>` / `--access-key <key>` | Basic auth (skips OAuth) | — | +| `--policy <file>` | *(next release after 0.10.0-rc.3)* Judge the suite's sealed pack against this policy YAML (repeatable). Linted before the run. See [Policies and verdicts](./policies.md) | — | +| `--policy-id <id>` | *(next release after 0.10.0-rc.3)* Judge it against a policy stored in Test Manager, latest version (repeatable; `kane-cli policy list`) | — | Each desktop worker gets its **own isolated Chrome** with a fresh temporary profile, so parallel members never share cookies, logins, or tabs — and never fight over your real browser profile. @@ -151,6 +153,9 @@ In a terminal, kane-cli offers to open the pack in the [evidence viewer](./evide | `2` | Usage error, invalid plan (preflight failures), or auth error. Nothing ran. | | `3` | Cancelled (Ctrl-C). | +<!-- release: needs evidence policies --> +With `--policy` or `--policy-id` *(next release after 0.10.0-rc.3)*, a bad policy is exit `2` before anything runs, and after the run a failing verdict turns `0` into `1` and an errored verdict turns it into `3`. A run that already failed keeps its code. With `--remote`, your machine judges the suite's pack once it is back. See [Policies and verdicts](./policies.md#judging-at-run-time). + ## Designed tests in a shared project In a folder linked to a [shared project](./assurance/shared-projects.md), each designed test runs on its scenario's **test case** in Test Manager, so every run is recorded on that case and counts toward the project's coverage. @@ -232,6 +237,7 @@ In agent / non-TTY mode, `testrun run` emits its own typed NDJSON events on stdo - [Remote runs on the cloud grid](./remote-execution.md) — `--remote`, mobile suites from any machine. - [Evidence packs](./evidence.md) — what's in the pack and how to view it. +- [Policies and verdicts](./policies.md) — gate the suite's pack on quality rules. - [Writing test.md files](./testmd/overview.md) — the file format, including `tags:`. - [Running test.md files](./testmd/running.md) — single-test runs, replay, and flags. - [CI/CD recipes](./cicd.md) — pipeline patterns. diff --git a/skill-installer/skills/SKILL.md b/skill-installer/skills/SKILL.md index 1aebb55..20d7f87 100644 --- a/skill-installer/skills/SKILL.md +++ b/skill-installer/skills/SKILL.md @@ -1,6 +1,6 @@ --- name: kane-cli -description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover. Work in a team's shared Test Manager project from a linked folder: link it with --project, save sessions, run the designed tests on their Test Manager cases, and read the project's coverage. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share a local assurance store (.context/) through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." +description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover, and judge a pack against a policy of quality gates (kane-cli evidence verdict, --policy) to pass or fail a CI run. Work in a team's shared Test Manager project from a linked folder: link it with --project, save sessions, run the designed tests on their Test Manager cases, and read the project's coverage. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share a local assurance store (.context/) through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." --- # Kane CLI — End-to-End Test Generation and Execution Skill @@ -188,6 +188,7 @@ When the user's request involves a browser — or writing test cases: - Share a **local** context store through a repository, a bucket or a folder, join one, keep two stores level, or resolve a sync conflict → Read `references/context-sync.md` first (kane-cli 0.8.14+). A team in Test Manager uses a shared project instead (`references/shared-projects.md`); a linked folder refuses `kane-cli context sync`, `kane-cli context push` and `kane-cli context clone` (`REMOTE_UNSUPPORTED`), and a location pull (a location name, `--rebase`, `--yes` or `--answer`: `PULL_LOCATION_LINKED`); a plain `kane-cli context pull --mode agent` there brings the project's tests (`references/shared-projects.md` §7); the store is shared through a location, never by copying or git-merging `.context/` - Multiple independent browser tasks → Read `references/parallel.md` first - View, share, or validate run evidence (`.evidence` packs) → Read `references/evidence.md` +- Judge a run or pack against quality gates (a policy YAML, `--policy`, `--policy-id`, `kane-cli evidence verdict`, `kane-cli policy …`), or gate CI on more than pass/fail → Read `references/policy.md` first (after 0.10.0-rc.3) - Debug a failed run → Read `references/debug.md` - Configure kane-cli or check directory layout → Read `references/setup-and-config.md` - Browse / create / pick a Test Manager project or folder for `kane-cli run`, or interpret the auto-default event → Read `references/test-manager.md`. Linking an assurance folder to a project is different: `references/shared-projects.md` §2–3 @@ -382,6 +383,7 @@ A draft in chat cannot be run, and a hand-written `_test.md` carries no requirem | Share the context store with a team, join one, keep stores level, or resolve a sync conflict (kane-cli 0.8.14+) | `references/context-sync.md` | | Run failed, need to diagnose | `references/debug.md` | | View, share, validate, or merge evidence packs | `references/evidence.md` | +| Judge a pack against a policy, run with `--policy`, or store/list policies in Test Manager | `references/policy.md` | | Multiple independent browser tasks | `references/parallel.md` | | Need full NDJSON event schema (`run`) | `references/parsing.md` | | Browse / create projects or folders, or parse the auto-default event | `references/test-manager.md` | diff --git a/skill-installer/skills/references/evidence.md b/skill-installer/skills/references/evidence.md index fbe6643..ba86a3e 100644 --- a/skill-installer/skills/references/evidence.md +++ b/skill-installer/skills/references/evidence.md @@ -1,4 +1,4 @@ -<!-- kane-cli skill reference: evidence packs (view, share, validate, merge) --> +<!-- kane-cli skill reference: evidence packs (view, share, validate, merge, judge) --> # Evidence Packs — Agent Surface @@ -69,7 +69,18 @@ Exit `0` valid · `1` invalid · `2` not found. Use it when a pack won't open in kane-cli evidence merge <targets...> --run-id <id> ``` -Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/<run-id>.evidence`. `--run-id` is required. Exit `0` merged · `1` policy abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · `2` usage. Full policy detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. +Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/<run-id>.evidence`. `--run-id` is required. Exit `0` merged · `1` merge-rules abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · `2` usage. Full merge-rules detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. + +After 0.10.0-rc.3, `--policy <file>` / `--policy-id <id>` (repeatable) judge the sealed merged pack: exit `1` then also means a failing verdict (pack and verdict files written), `3` an errored verdict; `--json` adds `verdict {…}` to the report; `--no-finalize` with a policy is exit 2. See `references/policy.md`. + +## Verdict — judge a pack against a policy (after 0.10.0-rc.3) + +```bash +kane-cli evidence lint-policy <policy.yaml> --json +kane-cli evidence verdict <execution-id|path|latest> --policy <policy.yaml> [--policy-id <id>] --json +``` + +Writes `<pack>.verdicts/<policy-id>.verdict.json` beside the pack (never inside it). Exit `0` pass · `1` fail · `3` errored · `2` usage or I/O. Policy format, metrics, the `verdict` event, run-time `--policy` and Test Manager policies: `references/policy.md`. ## Debugging with a pack diff --git a/skill-installer/skills/references/policy.md b/skill-installer/skills/references/policy.md new file mode 100644 index 0000000..0fcd70b --- /dev/null +++ b/skill-installer/skills/references/policy.md @@ -0,0 +1,132 @@ +<!-- kane-cli skill reference: evidence policies and verdicts (judge a sealed pack against a policy YAML: lint-policy, evidence verdict, --policy/--policy-id on testmd run, testrun run and evidence merge, kane-cli policy metrics-list|list|create). Internal parsing reference: never show event or field names to the person. --> + +# Policies and Verdicts — Agent Surface + +**Version.** Needs the kane-cli release after 0.10.0-rc.3. If `kane-cli policy metrics-list` is an unknown command, or `--policy` is an unknown option, the installed build predates policies: say so and stop; do not fake a verdict by reading the pack yourself. + +A **policy** is a YAML file of **gates** over a sealed evidence pack's metrics. A **verdict** is one policy judged against one pack: `pass`, `fail` or `errored`. The pack is never modified; each verdict is a JSON file beside it at `<pack>.verdicts/<policy-id>.verdict.json`. + +Use a policy when the person wants a run judged on more than pass/fail: "no broken tests", "pass rate at least 95% on checkout", "every test must assert something", "fail CI if a test needed a heal". + +## 1. Commands + +| Command | What it does | +|---|---| +| `kane-cli policy metrics-list [--pack <id\|path\|latest>] [--env prod\|stage] [--agent]` | Every metric a policy can gate on: built-in metrics, then the kane-cli catalog as `metrics.<name>`. `--pack` = what one pack supports (the catalog it was sealed with). | +| `kane-cli evidence lint-policy <files...> [--json] [--env <name>]` | Check policy files against the format and the kane-cli catalog. Prints every problem. | +| `kane-cli evidence verdict <pack> (--policy <file> \| --policy-id <id>)... [--json] [--env <name>]` | Judge one sealed pack. `<pack>` = execution id, `.evidence` path, or `latest` (newest sealed pack in `<cwd>/.testmuai/evidence/` by mtime). | +| `kane-cli testmd run <file> --policy <f> / --policy-id <id>` | Lint before the run, judge the test's own pack after it seals. | +| `kane-cli testrun run … --policy <f> / --policy-id <id> < /dev/null` | Lint after the plan (dry run too), judge the suite's pack. `--remote`: the laptop judges the consolidated pack; the grid never gets the flag. | +| `kane-cli evidence merge <targets...> --run-id <id> --policy <f> / --policy-id <id>` | Judge the sealed merged pack. Files are read but **not** linted first (a bad policy = an errored verdict). `--no-finalize` with a policy = exit 2. | +| `kane-cli policy list [--project <id> \| --all-projects] [--search <t>] [--limit 1-200] [--page <n>] [--agent]` | Policies in Test Manager (latest versions). Default scope: the profile's project. | +| `kane-cli policy create <file> [--project <id>] [--agent]` | Lint, then store in Test Manager. New content under an existing `policy:` id = a new version; identical content = the stored version (`created: false`). Project: `--project`, else the profile's (`kane-cli config project <id>`); none = exit 2. | + +`policy list` / `policy create` also take `--profile`, `--env`, `--username`, `--access-key`. `evidence verdict --policy-id` signs in with the active profile; `testmd run` / `testrun run` use the run's `--username` / `--access-key` when given. + +**Flag rules.** `--policy` and `--policy-id` are repeatable and combine. Repeat the flag per policy; never comma-join (`--policy a.yaml,b.yaml` is one file name). Two policies with the same `policy:` id in one command are refused. `--policy-id` takes the id `policy list` prints (the `ID` column / `uid`), not the YAML name, and always judges the **latest version**. + +## 2. Policy YAML + +```yaml +policy: release-gate # required; names the verdict file; ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ +requires: # optional: refuse packs with another catalog / version + catalog: kane-cli + version: ">=1" +gates: # required, at least one; at most 100; file at most 64 KB + - { id: no-failed, metric: totals.failed, equals: 0, severity: blocking } + - { id: checkout-rate, metric: pass_rate, where: "'checkout' in test.tags", min: 0.95, severity: blocking } + - { id: every-test-checks, metric: metrics.tests_without_checks, equals: 0, severity: blocking } + - { id: no-heals, metric: metrics.heal.healed_tests, max: 0, severity: warning, message: Review the heal } + - { id: no-flaky-critical, expr: "tests.filter(t, 'critical' in t.tags).all(t, !t.flaky)", severity: warning } +``` + +- A gate is **either** `metric` + exactly one of `equals` / `min` / `max` / `range: [lo, hi]` / `in: [...]`, **or** `expr` (CEL over `run`, `tests`, `totals`, `failures`, must be boolean) with no metric/operator/where. +- `severity`: `blocking` fails the verdict; `warning` reports (`status: warn`) without failing. An **errored** gate errors the verdict whatever its severity. +- `where` (CEL over one `test`) only on per-test metrics. +- Unknown keys are refused. +- Operators by type: counts/integers/ms/bytes/currency → all five; ratio/percentage/decimal → `min`/`max`/`range`; string/enum → `equals`/`in`. + +**Metrics.** Built-in (no prefix): `totals.tests`, `totals.passed`, `totals.failed`, `totals.broken`, `totals.skipped`, `pass_rate` (ratio; errors when every test skipped), `steps.total`, `steps.failed`, `steps.skipped` (all per test), `run.duration_ms`, `run.status` (enum `running|finalized|aborted`), `failures.count`. kane-cli catalog v1 (always `metrics.` prefix, all per test): `tests_without_checks`, `heal.healed_tests`, `heal.exhausted_tests`, `heal.reauthored_steps`, `non_replay_tests` (counts), `flaky_rate` (ratio). Run `kane-cli policy metrics-list --agent` for the installed truth before writing a gate; never invent a metric name. + +**Values.** A missing value is never zero: no value for the metric, or a pack older than the metric needs, **errors** the gate. A stored number that disagrees with its recompute also errors it. + +## 3. Exit codes + +| Command | 0 | 1 | 2 | 3 | +|---|---|---|---|---| +| `evidence verdict` | all pass | a verdict failed (none errored) | usage / I/O: no policy given, no pack, unreadable or unfetchable policy, a policy that doesn't check clean against the pack's catalog, unreadable pack, verdict files not written | a verdict errored | +| `evidence lint-policy` | clean (warnings allowed) | any error | file unreadable or lint threw | — | +| `testmd run` / `testrun run` with a policy | passed and verdict pass | tests failed, **or** tests passed and verdict failed | bad policy before anything ran (lint error, unknown `--policy-id`, sign-in failure), plus the command's usual 2s | verdict errored (tests passed), **or** cancelled / timed out | +| `evidence merge` with a policy | merged, verdict pass | merge-rules abort (no pack) or verdict failed (pack + verdicts written) | usage (incl. `--no-finalize`), verdict files not written | verdict errored | +| `policy metrics-list` | ok | — | unknown pack / catalog won't load | — | +| `policy list` | ok | Test Manager failure | bad flag / sign-in | — | +| `policy create` | created or already stored | Test Manager failure | no project, unreadable, lint errors, rejected by Test Manager (problems printed), sign-in | — | + +A verdict only changes a **clean 0**: failing → 1, errored → 3. A run that already failed or was cancelled keeps its own code. `test_md_done.overall_status` / `testrun_done.overall_status` stay the **tests'** status: read the `verdict` event for the policy result. Exit 3 is ambiguous on runs (errored verdict vs cancel/timeout): the `verdict` event decides it. + +## 4. The `verdict` event (agent mode) + +`testmd run`, `testrun run` (local and `--remote`) emit one typed line on stdout after the pack is judged and **before** the terminal event (`test_md_done`; `testrun_summary`/`testrun_done`; `remote_done`, whose `exit` already includes the verdict): + +```json +{"type":"verdict","evidence_id":"<execution_id>","result":"fail","exit_code":1, + "policies":[{"id":"release-gate","sha256":"…","result":"fail","exit_code":1, + "counts":{"total":5,"failed":1,"warned":0,"errored":0}, + "failed_gate_ids":["no-failed"],"errored_gate_ids":[], + "file":"/abs/path/<execution_id>.evidence.verdicts/release-gate.verdict.json"}], + "v":1,"ts":"…"} +``` + +- Combined `result`: errored if any policy errored, else fail if any failed, else pass. +- `error` (top level): nothing could be judged (e.g. the pack did not seal) — `policies` is empty. +- `policies[].error`: that verdict errored as a whole — `{code, message, hint}`. +- `files_error`: verdicts were made but files could not be written; the run warns and continues (verdicts still upload with the evidence). +- In a terminal the same information is one stderr line: `verdict: fail — release-gate fail (no-failed)`. + +`evidence verdict --json` prints the same object without `type`, plus `verdicts: [<full verdict documents>]`. `evidence merge --json` adds it to the merge report as `verdict: {…}`. + +Parsing: + +```text +if type === "verdict": + policy_result = result # pass | fail | errored + for p in policies: note p.id, p.result, p.failed_gate_ids, p.errored_gate_ids, p.file + if error: the pack could not be judged — report error, not a gate failure +``` + +## 5. The verdict file + +`<pack>.verdicts/<policy-id>.verdict.json` (the `policy:` value). Fields: `result`, `exit_code`, `evaluated_at`, `evaluator`, `pack {run_id, sha256, contract}`, `policy {id, sha256}`, `catalog`, `error`, `failed_gates[]`, `warnings[]`, `errored_gates[]`, and `gates[]` with `{id, metric | expr, where, op, expected, actual, severity, status: pass|fail|warn|errored, message, error}`. To explain a failure, read `gates[]` where `status` is `fail` or `errored` and give `expected` vs `actual`. + +**Where the folder is.** Always beside the pack that was judged; trust `policies[].file` over guessing. + +| Surface | Pack judged | +|---|---| +| `evidence verdict` | the pack you named (`latest` → `<cwd>/.testmuai/evidence/<id>.evidence`) | +| `testmd run` | the session-directory pack `~/.testmuai/kaneai/sessions/<session-id>/evidence/<id>.evidence` (not the `.testmuai/evidence/` copy) | +| `testrun run` (local or `--remote`) | the suite's pack in `<cwd>/.testmuai/evidence/` | +| `evidence merge` | the merged pack (`-o`, default `<cwd>/.testmuai/evidence/<run-id>.evidence`) | + +Judging again overwrites that policy's file; the pack never changes. + +## 6. Presenting results + +Never show event or field names. One row in the result card: + +```markdown +| 🔴 **Policy** | release-gate failed: 2 tests failed (expected 0) · 1 warning: a test needed a heal | +``` + +Use 🟢 pass, 🔴 fail, 🟡 errored (say *why* it couldn't be judged — a missing value, a pack that didn't seal — not "failed"). When the tests passed but the policy failed, say both plainly: "All 12 tests passed, but the release-gate policy failed: …". Offer to open the evidence (`references/evidence.md`). + +## 7. Common mistakes + +- **`kane-cli run` has no `--policy`.** Judge a one-off run afterwards: `kane-cli evidence verdict <pack> --policy <f>` (the pack path is in the stderr evidence hint). +- **Built-in metrics never take `metrics.`** (`totals.failed`, not `metrics.totals.failed`); kane-cli catalog metrics always do (`metrics.flaky_rate`). Lint reports `policy.builtin_prefixed` / `policy.unknown_metric`. +- **Two operators on one gate**, or `where` on an expression gate / a non-per-test metric: lint error. +- **`min`/`max` on a ratio means 0–1**, not percent: `min: 0.95`, not `95`. +- **Judge sealed packs only.** An unsealed pack directory cannot be judged (the verdict errors), and `evidence merge --no-finalize` with a policy is refused. +- **Lint before you run** a newly written policy: `kane-cli evidence lint-policy <file> --json`, then fix every `diagnostics[]` entry (`location`, `message`, `code`, `hint`). Runs lint for you, but a lint error there costs an exit 2 the person sees. +- **Don't parse `testrun_done.overall_status` for the gate** — it is the tests' status. The policy result is only in the `verdict` event, the exit code, and the files. +- **`--policy-id` needs a sign-in**; a failed sign-in is exit 2 before anything runs. Never ask for an access key in chat (`references/ready-check.md`). +- **Writing a policy for the person**: write the YAML from their words, run `metrics-list` to pick real metrics, lint it, and show them the gates before storing it with `policy create` (which shares it with their Test Manager project). diff --git a/skill-installer/skills/references/testmd.md b/skill-installer/skills/references/testmd.md index 919921b..8670745 100644 --- a/skill-installer/skills/references/testmd.md +++ b/skill-installer/skills/references/testmd.md @@ -146,6 +146,7 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `--no-adaptive-heal` | healing enabled | Disable adaptive healing after replay failure | | `--author` | off | Force authorable steps to re-author; replay-only steps still replay | | `--bug-detection <off\|stop\|continue>` | config (`off`) | Flag suspected product bugs while **authoring** (`stop` halts on a confirmed bug; `continue` records it). Replay failures always investigate regardless. | +| `--policy <file>` / `--policy-id <id>` | none | (after 0.10.0-rc.3) Judge the test's sealed pack against a policy YAML / a Test Manager policy (repeatable). Linted before the run (error = exit 2). A passing run exits `1` on a failing verdict, `3` on an errored one; a `verdict` event precedes `test_md_done`. `references/policy.md` | Common flags include `--agent`, `--headless`, `--max-steps`, `--timeout`, and `--variables`. Command-specific flags differ; check `kane-cli testmd run --help`. diff --git a/skill-installer/skills/references/testrun.md b/skill-installer/skills/references/testrun.md index 025e62a..948ab72 100644 --- a/skill-installer/skills/references/testrun.md +++ b/skill-installer/skills/references/testrun.md @@ -36,6 +36,8 @@ kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON when stdin is not | `--device-name <name>` | Device for the mobile members: as `kane-cli devices list --target <kind>` prints it locally, or a grid catalog device (`devices list … --remote`) with `--remote`; for real-device members a name or pattern from `devices list --target android\|ios --remote`. Validated before dispatch | members' `device_name:` (real devices: else the hub picks) | | `--os-version <version>` | OS version for the mobile members (`14`, `17.5`); alone = any device on that version | members' `os_version:` | | `--username` / `--access-key` | Basic auth | active profile | +| `--policy <file>` | (after 0.10.0-rc.3) Judge the suite's sealed pack against this policy YAML (repeatable; never comma-join). Linted after the plan, dry run too: an error is exit 2 before anything runs. With `--remote` the laptop judges the consolidated pack. `references/policy.md` | — | +| `--policy-id <id>` | (after 0.10.0-rc.3) The same with a Test Manager policy, latest version (repeatable; id from `kane-cli policy list --agent`) | — | ## Preflight (why members get rejected) @@ -152,6 +154,7 @@ All typed; stdout; one JSON object per line. **Local completion: `testrun_done`. | `testrun_investigations_wait` | `count` | Failed replays left investigations running; the coordinator waits before sealing. Narrate as "investigating N failures". | | `testrun_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` | Pack published to the dashboard. Absent when publish is skipped. | | `testrun_summary` | `totals: {tests, passed, failed, broken, skipped, authored}`, `duration_s`, `upload`, `cancelled`, `execution: {id, status}` | Build the rollup table from this. | +| `verdict` | `evidence_id`, `result: pass\|fail\|errored`, `exit_code`, `policies: [{id, result, failed_gate_ids, errored_gate_ids, file, …}]`, `error?` | Only with `--policy` / `--policy-id`, after the pack is judged and before `testrun_summary`. On `--remote`, before `remote_done`. Schema: `references/policy.md` §4 | | `testrun_done` | `execution_id`, `overall_status: "passed"\|"failed"\|"cancelled"` (`"refused"` for an invalid plan in a linked folder), `coverage?: {status, pct?}` | Local completion; remote runs continue through `remote_done`. `coverage` repeats the run's coverage figure (`pct` only when `status` is `written`). | ### Each test's own log, and `--stream-members` (0.8.17+) @@ -189,6 +192,7 @@ for each line: if type === "warning" && code === "unresolved_variables" → name the variables with no value (references/parsing.md); the run continues; ignore other warning codes if type === "testrun_member_end" or "testrun_authored_member_end" → note per-member outcome if type === "testrun_summary" → capture totals for the rollup + if type === "verdict" → capture the policy result (references/policy.md §4); testrun_done stays the tests' status else → informational; narrate sparingly ``` @@ -212,6 +216,8 @@ Don't list passing tests individually. If the pack published, mention the run is `0` all members passed · `1` at least one failed/broken · `2` usage / invalid plan / auth (nothing ran) · `3` cancelled (Ctrl-C — no new members start, running ones are stopped, and the pack still seals with what finished). +With `--policy` / `--policy-id`: a bad policy is `2` before anything runs; when every member passed, a failing verdict exits `1` and an errored verdict `3`; a run that already failed or was cancelled keeps its code. Tell an errored verdict from a cancel by the `verdict` event. + ## Evidence & debugging The suite produces **one** sealed pack, created directly in `<cwd>/.testmuai/evidence/`. Offer it to the user per `references/evidence.md`. Members run silently by design; to see per-member output while debugging, set `KANE_TESTRUN_MEMBER_DEBUG=1` (routes member events to stderr, prefixed `[member]`).