diff --git a/docs/agent-delivery-loop.md b/docs/agent-delivery-loop.md index 101d3ac0b..9ec551473 100644 --- a/docs/agent-delivery-loop.md +++ b/docs/agent-delivery-loop.md @@ -122,6 +122,18 @@ Before a merge, an orchestrator or operator calls `merge_precondition` (`POST /a | `head_moved` | The pull request's head changed since CI ran. The story goes back to `implementing` over `base_moved`, and the recorded head is cleared, so the gate is not called again until the story is back at `ci`. | | `unevaluated` | 503 with `Retry-After`. After repeated `unevaluated` answers the story escalates. | +### Gate B on the repositories that gate the loop + +Gate B reads one trigger list per repository from `DELIVERY_GATES_CONFIG` (`Loopctl.DeliveryGates.Triggers`, loaded by `Loopctl.DeliveryGates.Config`). A repository with no entry there escalates EVERY change, at triage and at the merge gate, with `{:unknown_repo, repo}` (`Loopctl.DeliveryGates.GateB`, `repo_triggers/2`); `test/loopctl/delivery_gates/gate_b_test.exs` pins it. + +`mkreyman/claude-config` and `mkreyman/chg` have no entry today. Every change to them therefore goes to a human, which is what issue #986 asks for: the loop must not be able to change, unreviewed, the code that lands, reverts or gates its own work, or the hooks that stop a session from doing what it must not. + +**The rule: when either repository gets an entry, its `human_paths` MUST name that repository's self-modification code**, which the loop must never change unreviewed: +- for chg, its gate, the code that advances `verified/master`, culprit and revert, its adapters, and its CI; +- for claude-config, the deny lists and hook wiring, the hooks that block or grant, the plugins that gate commits, merges and secrets, its git hooks (the commit gate) and its CI. + +The exact glob lists live in the operator's knowledge base (article `4a86157b`), not here: both repositories are private, and a list of their safety code does not belong in a public document. That article is a floor. Add to it, never remove from it, and re-check each glob against the repository before pasting it, because a pattern that matches no file is a stale trigger and escalates every change. + ## 7. After the merge - **`deployed` to `verified`, or to `escalated`:** `PostDeployVerification` compares the merge commit with the repository's deployment records. diff --git a/docs/measurements/README.md b/docs/measurements/README.md index bd2d8192d..95bc60aa7 100644 --- a/docs/measurements/README.md +++ b/docs/measurements/README.md @@ -18,7 +18,7 @@ SHIPPED gate code over history and count. |---|---| | `gate_b_.json` | the machine-readable Gate B result: outcome counts, three strata with their rates, the false-negative rows | | `gate_b_.md` | the same run's human summary, including every bias | -| `gate_a_.json` | the Gate A result: two strata, the sensitivity run, the escalated issue numbers | +| `gate_a_.json` | the Gate A result: four strata, the sensitivity run over the same four, the escalated issue numbers | | `gate_a_.md` | that run's human summary | Add a new dated pair per run rather than overwriting: the point of these is a TRACK RECORD, and @@ -168,13 +168,20 @@ Every Gate B artifact carries `scope_note` saying so. ## Reading a Gate A result -Two strata plus a sensitivity run: +Four strata plus a sensitivity run over the same four: - **intake** — the tickets Gate A is actually for: the ones the in-app chat filed with a `[Bug] : ` / `[Feature] : ` prefix. The headline, because that split was decided at FILING time and carries no hindsight. - **all** — every issue in the repository, classified by label. Wider, and biased by labels applied after the outcome was known. +- **chg_loop** — the tickets carrying the `chg-loop` label, which chg stamps on every issue it + files: machine-written, with a failure fingerprint, occurrences, a root cause or "unknown, + investigate", and acceptance criteria written as checks (US-47.11, issue #986). Measured apart + because such a ticket is expected to escalate far less than one a person wrote. It is replayed + exactly like any other ticket; the label only decides which stratum counts it. +- **human** — every ticket WITHOUT the `chg-loop` label. `chg_loop` and `human` partition `all`, + so their totals add up to it. `human` is not `intake`: it holds the engineering issues too. - **sensitivity** — the same corpus with `workflow_change_not_defect_fix` suppressed. Gate A's rate turns almost entirely on whether a feature request counts as a workflow change, so the result is a RANGE and the report says so rather than picking one end. @@ -186,6 +193,12 @@ the `[Feature]` share of the corpus, and the harness has measured the intake cha mix rather than any judgement Gate A made. The field is computed, not asserted, so a later run where the inversion trigger also fires reports `false` and the comparison regains its force. +`chg_loop` and `human` were added in US-47.11 BESIDE `intake` and `all`, which kept their +definitions, so an artifact from before then still compares with one from after on those two. +An artifact from before then has no `chg_loop`/`human` keys at all; that is the absence of the +split, not an empty `chg_loop` stratum. An EMPTY `chg_loop` stratum (rate `null`) means chg filed +nothing in the window. + The `all` stratum is the whole corpus and therefore MIXES the two classifiers: it contains the intake tickets, which keep the filing-time prefix reading, plus everything else, classified by label. It is not a second, disjoint corpus. diff --git a/docs/measurements/gate_a_2026-10-19.json b/docs/measurements/gate_a_2026-10-19.json new file mode 100644 index 000000000..32b627b3a --- /dev/null +++ b/docs/measurements/gate_a_2026-10-19.json @@ -0,0 +1,1715 @@ +{ + "biases": [ + "Trio DISAGREEMENT, the design's primary signal, is unobservable in a replay: every ticket is presented as three identical outputs. The measured rate is a strict LOWER BOUND.", + "The contradicts field is empty by construction, not by measurement. Lower bound again.", + "workflow_change_not_defect_fix stands in as the intake chat's [Feature] prefix for an INTAKE ticket and as an enhancement/idea label with no bug label for everything else. `intake` and `all` are NOT two corpora: `all` is the whole corpus and therefore MIXES both classifiers, because it contains the intake tickets too. Only the label half carries hindsight (a label is often applied after the outcome was known); the prefix is stamped at filing time.", + "inverts_or_removes_deliberate_behaviour stands in as an inversion phrase in the title or body, so an inversion described in other words is missed. Conservative.", + "confidence is sent as 0.0. Gate A requires the field and never reads it; the column is a placeholder, not a measurement.", + "Read `intake_is_feature_share` before comparing the intake rate with the design's 54%. Where every intake escalation is a [Feature] ticket firing on workflow_change alone, the rate is ARITHMETICALLY the [Feature] share of the corpus — the harness is measuring the corpus split, not Gate A's judgement, and the comparison is weaker than the bare number reads.", + "`chg_loop` and `human` PARTITION `all` on the `chg-loop` label chg stamps at filing time: `human` is every ticket WITHOUT that label, so it holds the intake tickets AND the engineering issues, and is not the `intake` stratum. The label only STRATIFIES: a `chg_loop` ticket is replayed exactly like any other, by the intake prefix if its title carries one and by label otherwise. An empty `chg_loop` stratum means chg filed nothing in the window, not that its tickets never escalate." + ], + "design_reference": { + "collapsed_predicate_rate": 0.54, + "note": "Design section 5: collapsing Gate A and Gate B into one escalation produced a 54% escalation rate on 80 first-parent commits, and a rate resembling it is a design failure rather than a tuning question." + }, + "escalations": [ + { + "chg_loop": false, + "created_at": "2026-10-19T04:26:42Z", + "intake": false, + "issue": 2653, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-19T03:54:28Z", + "intake": false, + "issue": 2650, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-19T01:40:47Z", + "intake": false, + "issue": 2626, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-19T01:40:02Z", + "intake": false, + "issue": 2589, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-19T01:39:14Z", + "intake": false, + "issue": 2553, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-18T23:45:10Z", + "intake": false, + "issue": 2540, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-17T02:03:18Z", + "intake": false, + "issue": 2497, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-16T20:04:47Z", + "intake": false, + "issue": 2472, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-16T18:45:45Z", + "intake": false, + "issue": 2466, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-16T18:43:01Z", + "intake": false, + "issue": 2465, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-12T00:33:14Z", + "intake": false, + "issue": 2401, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-10T02:51:12Z", + "intake": true, + "issue": 2369, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-10-08T21:20:57Z", + "intake": true, + "issue": 2325, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-10-08T21:18:54Z", + "intake": true, + "issue": 2324, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-10-02T07:21:05Z", + "intake": false, + "issue": 2056, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-10-01T03:09:31Z", + "intake": false, + "issue": 1987, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-09-30T19:25:26Z", + "intake": true, + "issue": 1970, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-28T19:08:41Z", + "intake": true, + "issue": 1869, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-28T18:41:11Z", + "intake": true, + "issue": 1868, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-28T18:40:00Z", + "intake": true, + "issue": 1867, + "reasons": [ + "workflow_change", + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-28T11:10:06Z", + "intake": true, + "issue": 1862, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-28T11:09:08Z", + "intake": true, + "issue": 1861, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-28T10:44:07Z", + "intake": true, + "issue": 1860, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-28T04:44:29Z", + "intake": true, + "issue": 1853, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-28T02:13:13Z", + "intake": false, + "issue": 1849, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-09-22T03:04:30Z", + "intake": false, + "issue": 1742, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-09-20T21:56:29Z", + "intake": false, + "issue": 1687, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-09-20T21:39:33Z", + "intake": false, + "issue": 1686, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-09-18T00:19:07Z", + "intake": false, + "issue": 1598, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-09-17T19:24:41Z", + "intake": false, + "issue": 1591, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-09-17T18:24:20Z", + "intake": false, + "issue": 1587, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-17T18:22:08Z", + "intake": false, + "issue": 1586, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-17T18:18:32Z", + "intake": false, + "issue": 1585, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-16T07:43:46Z", + "intake": true, + "issue": 1563, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-09-10T16:04:28Z", + "intake": true, + "issue": 1494, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-08-31T15:50:14Z", + "intake": false, + "issue": 1400, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-08-28T14:28:07Z", + "intake": false, + "issue": 1392, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-08-28T01:59:20Z", + "intake": false, + "issue": 1385, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-08-21T00:39:19Z", + "intake": false, + "issue": 1243, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-08-18T04:54:24Z", + "intake": false, + "issue": 1175, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-08-13T20:28:23Z", + "intake": false, + "issue": 1140, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-08-12T02:48:57Z", + "intake": true, + "issue": 1104, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-27T15:26:05Z", + "intake": true, + "issue": 956, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-24T19:21:27Z", + "intake": true, + "issue": 926, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-23T00:07:30Z", + "intake": false, + "issue": 913, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-22T20:00:17Z", + "intake": true, + "issue": 910, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-21T04:42:34Z", + "intake": false, + "issue": 892, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-07-19T02:01:51Z", + "intake": false, + "issue": 866, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-18T16:13:36Z", + "intake": false, + "issue": 842, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-07-17T20:58:26Z", + "intake": false, + "issue": 830, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-07-17T20:58:12Z", + "intake": false, + "issue": 816, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-07-17T20:58:10Z", + "intake": false, + "issue": 815, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-07-17T20:58:06Z", + "intake": false, + "issue": 811, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T23:10:16Z", + "intake": true, + "issue": 665, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:18:31Z", + "intake": false, + "issue": 661, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:18:12Z", + "intake": false, + "issue": 660, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:17:54Z", + "intake": false, + "issue": 659, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:17:37Z", + "intake": false, + "issue": 658, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:17:16Z", + "intake": false, + "issue": 657, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:16:57Z", + "intake": false, + "issue": 656, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:16:37Z", + "intake": false, + "issue": 655, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:16:22Z", + "intake": false, + "issue": 654, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:16:09Z", + "intake": false, + "issue": 653, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:15:54Z", + "intake": false, + "issue": 652, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:15:39Z", + "intake": false, + "issue": 651, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-08T01:00:15Z", + "intake": false, + "issue": 650, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-07T23:07:02Z", + "intake": false, + "issue": 647, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": true, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-07-07T23:06:45Z", + "intake": false, + "issue": 646, + "reasons": [ + "workflow_change", + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-07T23:06:24Z", + "intake": false, + "issue": 645, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-07T23:05:59Z", + "intake": false, + "issue": 644, + "reasons": [ + "workflow_change", + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-07T20:13:40Z", + "intake": false, + "issue": 643, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-07-04T01:13:43Z", + "intake": false, + "issue": 630, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-07-04T01:11:22Z", + "intake": false, + "issue": 627, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-06-20T03:07:47Z", + "intake": false, + "issue": 574, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-06-11T05:34:31Z", + "intake": false, + "issue": 510, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-11T05:34:10Z", + "intake": false, + "issue": 504, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-11T05:31:58Z", + "intake": false, + "issue": 467, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-11T05:31:22Z", + "intake": false, + "issue": 457, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-11T05:29:22Z", + "intake": false, + "issue": 423, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-11T05:29:18Z", + "intake": false, + "issue": 422, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-11T05:28:39Z", + "intake": false, + "issue": 411, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-01T02:50:46Z", + "intake": false, + "issue": 376, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-01T02:50:45Z", + "intake": false, + "issue": 375, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-01T02:50:33Z", + "intake": false, + "issue": 374, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-06-01T02:49:30Z", + "intake": false, + "issue": 372, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-05-30T22:59:59Z", + "intake": false, + "issue": 347, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-05-30T22:59:53Z", + "intake": false, + "issue": 342, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-05-30T22:57:21Z", + "intake": false, + "issue": 336, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-05-30T22:57:10Z", + "intake": false, + "issue": 327, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-05-29T04:09:41Z", + "intake": false, + "issue": 325, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-28T22:43:33Z", + "intake": true, + "issue": 321, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-20T20:57:50Z", + "intake": true, + "issue": 316, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-20T20:17:43Z", + "intake": true, + "issue": 315, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-13T21:15:28Z", + "intake": false, + "issue": 312, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-05-08T18:12:12Z", + "intake": true, + "issue": 306, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-07T01:38:44Z", + "intake": false, + "issue": 305, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-07T00:30:15Z", + "intake": false, + "issue": 303, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-06T04:12:00Z", + "intake": false, + "issue": 297, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-06T04:11:59Z", + "intake": false, + "issue": 296, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-06T04:11:57Z", + "intake": false, + "issue": 295, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-06T04:11:56Z", + "intake": false, + "issue": 294, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-06T04:11:47Z", + "intake": false, + "issue": 292, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-05-06T01:05:54Z", + "intake": false, + "issue": 289, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-04-27T19:08:57Z", + "intake": true, + "issue": 283, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-04-21T21:22:54Z", + "intake": true, + "issue": 270, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-04-19T02:43:45Z", + "intake": false, + "issue": 257, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-04-12T01:42:36Z", + "intake": false, + "issue": 212, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-04-08T22:48:25Z", + "intake": false, + "issue": 196, + "reasons": [ + "inverts_deliberate_behaviour" + ], + "signals": { + "inversion_phrase?": true, + "rejected?": false, + "request_shaped?": false + } + }, + { + "chg_loop": false, + "created_at": "2026-03-27T00:13:10Z", + "intake": true, + "issue": 176, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-03-21T21:39:40Z", + "intake": false, + "issue": 150, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + }, + { + "chg_loop": false, + "created_at": "2026-03-21T21:39:40Z", + "intake": false, + "issue": 149, + "reasons": [ + "workflow_change" + ], + "signals": { + "inversion_phrase?": false, + "rejected?": false, + "request_shaped?": true + } + } + ], + "gate": "A", + "intake_is_feature_share": { + "degenerate?": false, + "escalated": 25, + "escalation_reason_kinds": [ + "workflow_change", + "inverts_deliberate_behaviour" + ], + "note": "degenerate? true means every escalation fired on workflow_change alone, so the rate equals the [Feature] share of the corpus and measures the corpus split rather than Gate A's judgement. Weigh the comparison with the design's 54% accordingly.", + "request_shaped": 25, + "tickets": 41 + }, + "meta": { + "corpus": "mkreyman/home_care_billing GitHub issues, all states, fetched 2026-10-19T12:34Z", + "corpus_fingerprint": "744903b1918b", + "generated_at": "2026-10-19T12:34:51.570596Z", + "harness": "mix loopctl.gates.measure_a", + "since": "2026-03-18T17:26:12Z", + "tickets": 536, + "unparseable_records": 0, + "until": "2026-10-19T11:29:45Z" + }, + "question": "How often would Gate A escalate a past ticket to a human?", + "sensitivity": { + "all": { + "by_reason": { + "inverts_deliberate_behaviour": 54 + }, + "by_signal": { + "inversion_phrase": 54, + "rejected": 13, + "request_shaped": 0 + }, + "escalated": 54, + "not_story": 12, + "rate": 0.10074626865671642, + "total": 536 + }, + "chg_loop": { + "by_reason": {}, + "by_signal": { + "inversion_phrase": 0, + "rejected": 0, + "request_shaped": 0 + }, + "escalated": 0, + "not_story": 0, + "rate": null, + "total": 0 + }, + "human": { + "by_reason": { + "inverts_deliberate_behaviour": 54 + }, + "by_signal": { + "inversion_phrase": 54, + "rejected": 13, + "request_shaped": 0 + }, + "escalated": 54, + "not_story": 12, + "rate": 0.10074626865671642, + "total": 536 + }, + "intake": { + "by_reason": { + "inverts_deliberate_behaviour": 1 + }, + "by_signal": { + "inversion_phrase": 1, + "rejected": 0, + "request_shaped": 0 + }, + "escalated": 1, + "not_story": 0, + "rate": 0.024390243902439025, + "total": 41 + }, + "note": "the same corpus with every feature request read as NOT a workflow change. The true rate is between this and the headline, and which end it sits at is a definition question for the design, not a tuning question.", + "run": true, + "suppressed": "workflow_change_not_defect_fix" + }, + "strata": { + "all": { + "by_reason": { + "inverts_deliberate_behaviour": 54, + "workflow_change": 60 + }, + "by_signal": { + "inversion_phrase": 54, + "rejected": 13, + "request_shaped": 60 + }, + "escalated": 111, + "not_story": 12, + "rate": 0.20708955223880596, + "total": 536 + }, + "chg_loop": { + "by_reason": {}, + "by_signal": { + "inversion_phrase": 0, + "rejected": 0, + "request_shaped": 0 + }, + "escalated": 0, + "not_story": 0, + "rate": null, + "total": 0 + }, + "human": { + "by_reason": { + "inverts_deliberate_behaviour": 54, + "workflow_change": 60 + }, + "by_signal": { + "inversion_phrase": 54, + "rejected": 13, + "request_shaped": 60 + }, + "escalated": 111, + "not_story": 12, + "rate": 0.20708955223880596, + "total": 536 + }, + "intake": { + "by_reason": { + "inverts_deliberate_behaviour": 1, + "workflow_change": 25 + }, + "by_signal": { + "inversion_phrase": 1, + "rejected": 0, + "request_shaped": 25 + }, + "escalated": 25, + "not_story": 0, + "rate": 0.6097560975609756, + "total": 41 + } + } +} diff --git a/docs/measurements/gate_a_2026-10-19.md b/docs/measurements/gate_a_2026-10-19.md new file mode 100644 index 000000000..75a679232 --- /dev/null +++ b/docs/measurements/gate_a_2026-10-19.md @@ -0,0 +1,26 @@ +Gate A replay — mkreyman/home_care_billing GitHub issues, all states, fetched 2026-10-19T12:34Z +window: 2026-03-18T17:26:12Z .. 2026-10-19T11:29:45Z (corpus 744903b1918b) + +intake tickets: n=41 escalated=25 (61.0%) not_story=0 by_reason=[inverts_deliberate_behaviour: 1, workflow_change: 25] +all tickets: n=536 escalated=111 (20.7%) not_story=12 by_reason=[inverts_deliberate_behaviour: 54, workflow_change: 60] +chg-loop tickets: n=0 escalated=0 (no measurement) not_story=0 by_reason=[] +human tickets: n=536 escalated=111 (20.7%) not_story=12 by_reason=[inverts_deliberate_behaviour: 54, workflow_change: 60] + +sensitivity — with workflow_change_not_defect_fix suppressed: + intake tickets: n=41 escalated=1 (2.4%) not_story=0 by_reason=[inverts_deliberate_behaviour: 1] + all tickets: n=536 escalated=54 (10.1%) not_story=12 by_reason=[inverts_deliberate_behaviour: 54] + chg-loop tickets: n=0 escalated=0 (no measurement) not_story=0 by_reason=[] + human tickets: n=536 escalated=54 (10.1%) not_story=12 by_reason=[inverts_deliberate_behaviour: 54] + +intake escalation reason kinds: [:inverts_deliberate_behaviour, :workflow_change] (not degenerate) + +design's collapsed-predicate rate, for comparison: 54% + +Read with these: + - Trio DISAGREEMENT, the design's primary signal, is unobservable in a replay: every ticket is presented as three identical outputs. The measured rate is a strict LOWER BOUND. + - The contradicts field is empty by construction, not by measurement. Lower bound again. + - workflow_change_not_defect_fix stands in as the intake chat's [Feature] prefix for an INTAKE ticket and as an enhancement/idea label with no bug label for everything else. `intake` and `all` are NOT two corpora: `all` is the whole corpus and therefore MIXES both classifiers, because it contains the intake tickets too. Only the label half carries hindsight (a label is often applied after the outcome was known); the prefix is stamped at filing time. + - inverts_or_removes_deliberate_behaviour stands in as an inversion phrase in the title or body, so an inversion described in other words is missed. Conservative. + - confidence is sent as 0.0. Gate A requires the field and never reads it; the column is a placeholder, not a measurement. + - Read `intake_is_feature_share` before comparing the intake rate with the design's 54%. Where every intake escalation is a [Feature] ticket firing on workflow_change alone, the rate is ARITHMETICALLY the [Feature] share of the corpus — the harness is measuring the corpus split, not Gate A's judgement, and the comparison is weaker than the bare number reads. + - `chg_loop` and `human` PARTITION `all` on the `chg-loop` label chg stamps at filing time: `human` is every ticket WITHOUT that label, so it holds the intake tickets AND the engineering issues, and is not the `intake` stratum. The label only STRATIFIES: a `chg_loop` ticket is replayed exactly like any other, by the intake prefix if its title carries one and by label otherwise. An empty `chg_loop` stratum means chg filed nothing in the window, not that its tickets never escalate. diff --git a/docs/user_stories/epic_47_change_protocol/us_47.11.json b/docs/user_stories/epic_47_change_protocol/us_47.11.json new file mode 100644 index 000000000..9eb627d19 --- /dev/null +++ b/docs/user_stories/epic_47_change_protocol/us_47.11.json @@ -0,0 +1,99 @@ +{ + "id": "US-47.11", + "title": "Measure Gate A on chg-loop tickets apart from human intake, and pin Gate B's self-modification floor (#986 items 2 and 4)", + "epic": { + "id": "epic_47", + "name": "Change Protocol (loopctl half)" + }, + "priority": "medium", + "phase": "change_protocol", + "story": { + "as_a": "operator", + "i_want_to": "see Gate A's escalation rate on the machine-written chg-loop tickets separately from the human ones, and know which paths of claude-config and chg must stay behind a human", + "so_that": "the claim that evidence-carrying tickets escalate far less than the human baseline is measured rather than assumed, and the loop can never merge, unreviewed, a change to the code that gates it" + }, + "rationale": "chg files issues labelled chg-loop on claude-config, chg and home_care_billing, each carrying a failure fingerprint, its occurrences, a root cause or 'unknown, investigate', and acceptance criteria written as checks. KB a08b94b2 expects them to escalate far less than the 69% human baseline. The Gate A replay report had only the intake and all strata, so that expectation could not be checked. The gate itself is not relaxed: only the report gains the split. For Gate B, a repository absent from DELIVERY_GATES_CONFIG already escalates every change with {:unknown_repo, repo}; claude-config and chg are absent today, which satisfies #986, but nothing said what their entries must contain once they get one.", + "acceptance_criteria": [ + { + "id": "AC-47.11.1", + "description": "Loopctl.DeliveryGates.Measurement.Ticket carries chg_loop?, true exactly when the ticket's labels include chg-loop; the title plays no part." + }, + { + "id": "AC-47.11.2", + "description": "The Gate A report carries two new strata, chg_loop (the chg-loop tickets) and human (every other ticket), in the main strata, in the sensitivity run and in the printed summary. They partition all." + }, + { + "id": "AC-47.11.3", + "description": "The intake and all strata keep their definitions and values, so artifacts from before and after the change compare on them." + }, + { + "id": "AC-47.11.4", + "description": "The committed gate_a_2026-09-14 artifact is not rewritten (docs/measurements/README.md: a new dated pair per run). A new pair, gate_a_2026-10-19, is generated by mix loopctl.gates.measure_a from a fresh gh issue list of mkreyman/home_care_billing and carries the four strata; mix loopctl.gates.check_measurements passes over the directory." + }, + { + "id": "AC-47.11.5", + "description": "docs/agent-delivery-loop.md states that a repository absent from DELIVERY_GATES_CONFIG escalates every change, that claude-config and chg are absent, and states the human_paths floor each must carry once it gets an entry: chg's gate, verified/master, culprit/revert and CI code; claude-config's deny lists and hook wiring, its blocking and granting hooks, its gating plugins, its git hooks and CI. The pasteable globs live in KB article 4a86157b, not in the public doc, because both repositories are private." + }, + { + "id": "AC-47.11.6", + "description": "A Gate B test pins that an unconfigured repository escalates {:unknown_repo, repo} on an otherwise-clear change at merge, and on the triage screen with or without predicted touches." + } + ], + "test_cases": [ + { + "id": "TC-47.11.1", + "verifies": ["AC-47.11.1"], + "name": "the chg_loop flag", + "type": "unit", + "steps": [ + "parse a gh issue record labelled chg-loop, one with no label or a lookalike label, and one whose TITLE says chg-loop" + ], + "expected_results": [ + "only the labelled record has chg_loop? true" + ] + }, + { + "id": "TC-47.11.2", + "verifies": ["AC-47.11.2", "AC-47.11.3"], + "name": "four strata on a mixed corpus", + "type": "unit", + "steps": [ + "build Report.gate_a over two chg-loop tickets (one escalating), an intake [Feature], an intake [Bug], an enhancement and a not-planned issue, with and without a sensitivity run" + ], + "expected_results": [ + "chg_loop is 2 tickets, 1 escalated; human is 4 tickets, 2 escalated, 1 not_story", + "chg_loop + human equals all on total, escalated and not_story", + "intake and all hold the values they held before the change, and on a corpus with no chg-loop ticket human equals all", + "the sensitivity run and the summary carry chg_loop and human", + "breaking the split (splitting on intake? instead of chg_loop?) fails four of these tests" + ] + }, + { + "id": "TC-47.11.3", + "verifies": ["AC-47.11.6"], + "name": "unknown repository escalation pin", + "type": "unit", + "steps": [ + "evaluate Gate B at merge, and run triage_screen/3, for mkreyman/claude-config and mkreyman/chg against a configuration that names neither, on a README-only change that clears for the configured repository" + ], + "expected_results": [ + "outcome :human with exactly {:unknown_repo, repo}", + "triage_screen/3 returns [{:unknown_repo, repo}] with and without predicted touches" + ] + }, + { + "id": "TC-47.11.4", + "verifies": ["AC-47.11.4"], + "name": "committed artifacts stay consistent", + "type": "integration", + "steps": [ + "run ArtifactPins.check over docs/measurements (artifact_pins_test.exs)" + ], + "expected_results": [ + "every artifact carries its pins and the directory passes" + ] + } + ], + "technical_notes": "Strata are computed once by Report.gate_a_strata/1, used by the main report and the sensitivity run so the two cannot diverge. Escalation rows gain a chg_loop flag beside intake. The Gate B path globs were checked against git ls-files of chg at f14c483 and claude-config at 31bd0a8 with Loopctl.DeliveryGates.Glob: every one matches at least one file, so none is a stale trigger. chg's landing lives in chg-server under server/, which does not exist yet; its glob is to be added with it, never before, because a pattern that matches nothing escalates every change as a stale trigger.", + "dependencies": [] +} diff --git a/lib/loopctl/delivery_gates/measurement/gate_a_replay.ex b/lib/loopctl/delivery_gates/measurement/gate_a_replay.ex index 6abb036df..2b9c89a92 100644 --- a/lib/loopctl/delivery_gates/measurement/gate_a_replay.ex +++ b/lib/loopctl/delivery_gates/measurement/gate_a_replay.ex @@ -27,7 +27,7 @@ defmodule Loopctl.DeliveryGates.Measurement.GateAReplay do The prefix is stamped by the in-app chat at FILING time, before anyone knows what the fix turned out to be. A `bug`/`enhancement` LABEL is often applied afterwards, sometimes after the work landed, so a corpus classified by label carries hindsight the triage trio would not - have had. Both strata are reported; the intake one is the headline for exactly this reason. + have had. The report carries four strata (`Report.gate_a_strata/1`); the intake one is the headline for exactly this reason. ## `confidence` is a placeholder, not a measurement diff --git a/lib/loopctl/delivery_gates/measurement/report.ex b/lib/loopctl/delivery_gates/measurement/report.ex index 0911470f0..3761d3158 100644 --- a/lib/loopctl/delivery_gates/measurement/report.ex +++ b/lib/loopctl/delivery_gates/measurement/report.ex @@ -59,10 +59,11 @@ defmodule Loopctl.DeliveryGates.Measurement.Report do @gate_a_biases [ "Trio DISAGREEMENT, the design's primary signal, is unobservable in a replay: every ticket is presented as three identical outputs. The measured rate is a strict LOWER BOUND.", "The contradicts field is empty by construction, not by measurement. Lower bound again.", - "workflow_change_not_defect_fix stands in as the intake chat's [Feature] prefix for an INTAKE ticket and as an enhancement/idea label with no bug label for everything else. The two strata are NOT two corpora: `all` is the whole corpus and therefore MIXES both classifiers, because it contains the intake tickets too. Only the label half carries hindsight (a label is often applied after the outcome was known); the prefix is stamped at filing time.", + "workflow_change_not_defect_fix stands in as the intake chat's [Feature] prefix for an INTAKE ticket and as an enhancement/idea label with no bug label for everything else. `intake` and `all` are NOT two corpora: `all` is the whole corpus and therefore MIXES both classifiers, because it contains the intake tickets too. Only the label half carries hindsight (a label is often applied after the outcome was known); the prefix is stamped at filing time.", "inverts_or_removes_deliberate_behaviour stands in as an inversion phrase in the title or body, so an inversion described in other words is missed. Conservative.", "confidence is sent as 0.0. Gate A requires the field and never reads it; the column is a placeholder, not a measurement.", - "Read `intake_is_feature_share` before comparing the intake rate with the design's 54%. Where every intake escalation is a [Feature] ticket firing on workflow_change alone, the rate is ARITHMETICALLY the [Feature] share of the corpus — the harness is measuring the corpus split, not Gate A's judgement, and the comparison is weaker than the bare number reads." + "Read `intake_is_feature_share` before comparing the intake rate with the design's 54%. Where every intake escalation is a [Feature] ticket firing on workflow_change alone, the rate is ARITHMETICALLY the [Feature] share of the corpus — the harness is measuring the corpus split, not Gate A's judgement, and the comparison is weaker than the bare number reads.", + "`chg_loop` and `human` PARTITION `all` on the `chg-loop` label chg stamps at filing time: `human` is every ticket WITHOUT that label, so it holds the intake tickets AND the engineering issues, and is not the `intake` stratum. The label only STRATIFIES: a `chg_loop` ticket is replayed exactly like any other, by the intake prefix if its title carries one and by label otherwise. An empty `chg_loop` stratum means chg filed nothing in the window, not that its tickets never escalate." ] @doc """ @@ -128,7 +129,7 @@ defmodule Loopctl.DeliveryGates.Measurement.Report do @spec gate_a([GateAReplay.t()], map(), keyword()) :: map() def gate_a(replays, meta, opts \\ []) when is_list(replays) and is_map(meta) do detail = Keyword.get(opts, :detail, :redacted) - intake = Enum.filter(replays, & &1.ticket.intake?) + intake = intake_replays(replays) %{ gate: "A", @@ -140,10 +141,9 @@ defmodule Loopctl.DeliveryGates.Measurement.Report do note: "Design section 5: collapsing Gate A and Gate B into one escalation produced a 54% escalation rate on 80 first-parent commits, and a rate resembling it is a design failure rather than a tuning question." }, - strata: %{ - intake: gate_a_stratum(intake), - all: gate_a_stratum(replays) - }, + # `chg_loop` and `human` sit BESIDE `intake` and `all`, never in place of them: the + # committed artifacts and every comparison quoted against them read those two (US-47.11). + strata: gate_a_strata(replays), intake_is_feature_share: feature_share(intake), sensitivity: sensitivity(Keyword.get(opts, :sensitivity)), escalations: @@ -225,6 +225,8 @@ defmodule Loopctl.DeliveryGates.Measurement.Report do "", gate_a_line("intake tickets", report.strata.intake), gate_a_line("all tickets", report.strata.all), + gate_a_line("chg-loop tickets", report.strata.chg_loop), + gate_a_line("human tickets", report.strata.human), "" ] ++ sensitivity_lines(report.sensitivity) ++ @@ -257,7 +259,9 @@ defmodule Loopctl.DeliveryGates.Measurement.Report do [ "sensitivity — with #{sensitivity.suppressed} suppressed:", " " <> gate_a_line("intake tickets", sensitivity.intake), - " " <> gate_a_line("all tickets", sensitivity.all) + " " <> gate_a_line("all tickets", sensitivity.all), + " " <> gate_a_line("chg-loop tickets", sensitivity.chg_loop), + " " <> gate_a_line("human tickets", sensitivity.human) ] end @@ -362,16 +366,43 @@ defmodule Loopctl.DeliveryGates.Measurement.Report do end defp sensitivity(replays) when is_list(replays) do - %{ + Map.merge(gate_a_strata(replays), %{ suppressed: "workflow_change_not_defect_fix", run: true, note: - "the same corpus with every feature request read as NOT a workflow change. The true rate is between this and the headline, and which end it sits at is a definition question for the design, not a tuning question.", - intake: gate_a_stratum(Enum.filter(replays, & &1.ticket.intake?)), - all: gate_a_stratum(replays) + "the same corpus with every feature request read as NOT a workflow change. The true rate is between this and the headline, and which end it sits at is a definition question for the design, not a tuning question." + }) + end + + @doc """ + The four Gate A strata of one set of replays. + + `intake` and `all` are the original two. `chg_loop` and `human` (US-47.11) PARTITION `all` on + the ticket's `chg-loop` label, so `chg_loop.total + human.total == all.total` and the same + holds for `escalated` and `not_story`. They measure machine-filed tickets apart from + everything else; the gate is unchanged and reads no label. + """ + @spec gate_a_strata([GateAReplay.t()]) :: %{ + intake: map(), + all: map(), + chg_loop: map(), + human: map() + } + def gate_a_strata(replays) when is_list(replays) do + {chg_loop, human} = Enum.split_with(replays, & &1.ticket.chg_loop?) + + %{ + intake: gate_a_stratum(intake_replays(replays)), + all: gate_a_stratum(replays), + chg_loop: gate_a_stratum(chg_loop), + human: gate_a_stratum(human) } end + # The one definition of the intake population: the stratum and `intake_is_feature_share` + # both read it, so they cannot describe different tickets. + defp intake_replays(replays), do: Enum.filter(replays, & &1.ticket.intake?) + @doc """ Whether the intake escalation rate is measuring Gate A at all, or just the corpus split. @@ -551,6 +582,7 @@ defmodule Loopctl.DeliveryGates.Measurement.Report do issue: replay.ticket.number, created_at: replay.ticket.created_at, intake: replay.ticket.intake?, + chg_loop: replay.ticket.chg_loop?, reasons: replay.result.reasons |> Enum.map(&elem(&1, 0)) |> Enum.uniq(), signals: replay.signals } diff --git a/lib/loopctl/delivery_gates/measurement/ticket.ex b/lib/loopctl/delivery_gates/measurement/ticket.ex index e9ef6f648..de660781a 100644 --- a/lib/loopctl/delivery_gates/measurement/ticket.ex +++ b/lib/loopctl/delivery_gates/measurement/ticket.ex @@ -11,11 +11,21 @@ defmodule Loopctl.DeliveryGates.Measurement.Ticket do The delivery loop's Gate A judges tickets Larisa files through the in-app chat, which stamps a `[Bug] : ` or `[Feature] : ` prefix at FILING time. Every other issue in - the repository is engineering work the loop never triages. So the corpus is reported in two - strata and the intake one is the headline: it is the population Gate A is for, and its + the repository is engineering work the loop never triages. So the `intake` stratum is the + headline among the report's four (`Report.gate_a_strata/1`): it is the population Gate A is for, and its bug/feature split was decided by the intake chat rather than by a label somebody applied after the outcome was known. + ## `chg_loop?` — tickets a machine filed (US-47.11, issue #986) + + chg files machine-written, evidence-carrying issues — a failure fingerprint, its occurrences, + a root cause or "unknown, investigate", acceptance criteria written as checks — and labels + each one `chg-loop`. They are expected to escalate far less than the human baseline, so the + report measures them as their own stratum and the rest as `human`. The label is the whole + test, matched exactly: chg applies it at FILING time, so unlike a `bug`/`enhancement` label it + carries no hindsight about how the ticket turned out. The gate itself reads none of this; + only the report splits on it. + ## It refuses rather than guesses A record with no `number` or no `title` is `{:error, reason}` and is counted as @@ -32,7 +42,8 @@ defmodule Loopctl.DeliveryGates.Measurement.Ticket do :state_reason, :created_at, labels: [], - intake?: false + intake?: false, + chg_loop?: false ] @type t :: %__MODULE__{ @@ -43,17 +54,22 @@ defmodule Loopctl.DeliveryGates.Measurement.Ticket do state_reason: String.t() | nil, created_at: String.t() | nil, labels: [String.t()], - intake?: boolean() + intake?: boolean(), + chg_loop?: boolean() } @intake_prefix ~r/\A\[(Bug|Feature)\]\s+.+?:\s/ + @chg_loop_label "chg-loop" + @doc """ Parses one decoded `gh issue list --json` object (string keys). """ @spec parse(term()) :: {:ok, t()} | {:error, term()} def parse(%{"number" => number, "title" => title} = record) when is_integer(number) and number > 0 and is_binary(title) and title != "" do + labels = labels(Map.get(record, "labels")) + {:ok, %__MODULE__{ number: number, @@ -62,8 +78,9 @@ defmodule Loopctl.DeliveryGates.Measurement.Ticket do state: string_or_nil(Map.get(record, "state")), state_reason: string_or_nil(Map.get(record, "stateReason")), created_at: string_or_nil(Map.get(record, "createdAt")), - labels: labels(Map.get(record, "labels")), - intake?: Regex.match?(@intake_prefix, title) + labels: labels, + intake?: Regex.match?(@intake_prefix, title), + chg_loop?: @chg_loop_label in labels }} end diff --git a/test/loopctl/delivery_gates/gate_b_test.exs b/test/loopctl/delivery_gates/gate_b_test.exs index fb8092bd1..52755bb2d 100644 --- a/test/loopctl/delivery_gates/gate_b_test.exs +++ b/test/loopctl/delivery_gates/gate_b_test.exs @@ -352,6 +352,24 @@ defmodule Loopctl.DeliveryGates.GateBTest do assert %Result{outcome: :human, reasons: [{:unknown_repo, nil}]} = GateB.evaluate(:merge, Map.delete(input(), :repo), triggers()) end + + # US-47.11 (#986): docs/agent-delivery-loop.md relies on this to say that claude-config and + # chg, absent from DELIVERY_GATES_CONFIG, send EVERY change to a human. "Every" is the + # claim, so it is pinned on the change the gate would otherwise clear — an unguarded file + # well inside the size bound — and on the triage screen with no predicted touch at all. + test "a repository absent from the configuration escalates every change, at triage and at merge" do + # The control: the same change to a configured repository clears, at merge and at triage. + assert %Result{outcome: :clear} = evaluate(:merge, %{files: ["README.md"]}) + assert GateB.triage_screen(triggers(), input().repo, ["README.md"]) == [] + + for repo <- ["mkreyman/claude-config", "mkreyman/chg"] do + assert %Result{outcome: :human, reasons: [{:unknown_repo, ^repo}]} = + evaluate(:merge, %{repo: repo, files: ["README.md"]}) + + assert GateB.triage_screen(triggers(), repo, []) == [{:unknown_repo, repo}] + assert GateB.triage_screen(triggers(), repo, ["README.md"]) == [{:unknown_repo, repo}] + end + end end describe "stale triggers" do diff --git a/test/loopctl/delivery_gates/measurement/report_test.exs b/test/loopctl/delivery_gates/measurement/report_test.exs index 3f3257f75..57f8314c6 100644 --- a/test/loopctl/delivery_gates/measurement/report_test.exs +++ b/test/loopctl/delivery_gates/measurement/report_test.exs @@ -397,6 +397,94 @@ defmodule Loopctl.DeliveryGates.Measurement.ReportTest do assert report.strata.all.escalated == 2 end + # US-47.11: a mixed corpus — two chg-loop tickets (one escalating on an inversion phrase, + # one not), one intake [Feature], one intake [Bug], one engineering enhancement and one + # not-planned issue. + defp mixed_corpus do + [ + replay(%{ + "number" => 1, + "title" => "Failure 3f9a: claim total off by one", + "body" => "Fingerprint 3f9a. Root cause: unknown, investigate.", + labels: ["chg-loop"] + }), + replay(%{ + "number" => 2, + "title" => "Failure 77c1: the export no longer includes the header", + labels: ["chg-loop", "bug"] + }), + replay(%{"number" => 3, "title" => "[Feature] Acme Homecare: a column"}), + replay(%{"number" => 4}), + replay(%{"number" => 5, "title" => "Refactor the parser", labels: ["enhancement"]}), + replay(%{"number" => 6, "title" => "Old idea", "stateReason" => "NOT_PLANNED"}) + ] + end + + test "chg_loop and human strata split the corpus on the chg-loop label" do + report = Report.gate_a(mixed_corpus(), %{corpus: "acme/repo"}) + + assert %{total: 2, escalated: 1, rate: 0.5, not_story: 0} = report.strata.chg_loop + assert report.strata.chg_loop.by_reason == %{inverts_deliberate_behaviour: 1} + + assert %{total: 4, escalated: 2, rate: 0.5, not_story: 1} = report.strata.human + assert report.strata.human.by_reason == %{workflow_change: 2} + end + + test "chg_loop and human PARTITION all" do + %{all: all, chg_loop: chg_loop, human: human} = + Report.gate_a(mixed_corpus(), %{}).strata + + for key <- [:total, :escalated, :not_story] do + assert Map.fetch!(chg_loop, key) + Map.fetch!(human, key) == Map.fetch!(all, key) + end + end + + test "intake and all are unchanged by the new strata" do + report = Report.gate_a(mixed_corpus(), %{}) + + assert %{total: 2, escalated: 1, rate: 0.5, not_story: 0} = report.strata.intake + assert report.strata.intake.by_reason == %{workflow_change: 1} + + assert %{total: 6, escalated: 3, rate: 0.5, not_story: 1} = report.strata.all + + assert report.strata.all.by_reason == %{ + inverts_deliberate_behaviour: 1, + workflow_change: 2 + } + + # The pre-US-47.11 corpus, with no chg-loop label anywhere: `human` IS `all`. + plain = Report.gate_a(Enum.drop(mixed_corpus(), 2), %{}).strata + assert %{total: 0, escalated: 0, rate: nil, not_story: 0} = plain.chg_loop + assert plain.human == plain.all + end + + test "an empty chg_loop stratum has a nil rate, never a zero one" do + strata = Report.gate_a([replay(%{})], %{}).strata + + assert %{total: 0, escalated: 0, rate: nil} = strata.chg_loop + end + + test "the sensitivity run carries the chg_loop and human strata too" do + replays = mixed_corpus() + + sensitivity = + Enum.map(replays, &GateAReplay.replay(&1.ticket, suppress: [:request_shaped?])) + + report = Report.gate_a(replays, %{}, sensitivity: sensitivity) + + assert %{total: 2, escalated: 1} = report.sensitivity.chg_loop + assert %{total: 4, escalated: 0} = report.sensitivity.human + assert %{total: 2, escalated: 0} = report.sensitivity.intake + assert %{total: 6, escalated: 1} = report.sensitivity.all + end + + test "an escalation row says whether chg filed it" do + rows = Report.gate_a(mixed_corpus(), %{}).escalations + + assert Enum.find(rows, &(&1.issue == 2)).chg_loop + refute Enum.find(rows, &(&1.issue == 3)).chg_loop + end + test "counts not_story apart from escalated" do replays = [replay(%{"stateReason" => "NOT_PLANNED"})] @@ -538,6 +626,25 @@ defmodule Loopctl.DeliveryGates.Measurement.ReportTest do assert summary =~ "NOT evidence that Gate B has none" end + test "the Gate A summary prints the chg-loop and human strata, in the sensitivity run too" do + tickets = [ticket(%{"title" => "Failure 3f9a", labels: ["chg-loop"]}), ticket(%{})] + + summary = + Report.summarize( + Report.gate_a(Enum.map(tickets, &GateAReplay.replay/1), %{corpus: "acme/repo"}, + sensitivity: Enum.map(tickets, &GateAReplay.replay(&1, suppress: [:request_shaped?])) + ) + ) + + assert summary =~ "intake tickets: n=1" + assert summary =~ "all tickets: n=2" + # Each line twice: once unindented in the main strata, once indented in the sensitivity run. + assert summary =~ ~r/^chg-loop tickets: n=1/m + assert summary =~ ~r/^human tickets: n=1/m + assert summary =~ ~r/^ chg-loop tickets: n=1/m + assert summary =~ ~r/^ human tickets: n=1/m + end + test "the Gate A summary states the lower-bound bias and the design's 54%" do summary = Report.summarize(Report.gate_a([replay(%{})], %{corpus: "acme/repo"})) diff --git a/test/loopctl/delivery_gates/measurement/ticket_test.exs b/test/loopctl/delivery_gates/measurement/ticket_test.exs index b2de6f621..bc7f464d3 100644 --- a/test/loopctl/delivery_gates/measurement/ticket_test.exs +++ b/test/loopctl/delivery_gates/measurement/ticket_test.exs @@ -32,6 +32,28 @@ defmodule Loopctl.DeliveryGates.Measurement.TicketTest do Ticket.parse(build(:measurement_ticket, %{"title" => "[Bug] the total is wrong"})) end + test "a chg-loop label marks a ticket chg filed (US-47.11)" do + assert {:ok, %{chg_loop?: true, intake?: false}} = + Ticket.parse( + build(:measurement_ticket, %{ + "title" => "Flaky test: billing_test.exs:42 (fingerprint 3f9a)", + labels: ["bug", "chg-loop"] + }) + ) + end + + test "no chg-loop label, or a label that only resembles it, is not a chg-loop ticket" do + for labels <- [[], ["bug"], ["chg"], ["chg-loop-candidate"], ["loop"]] do + assert {:ok, %{chg_loop?: false}} = + Ticket.parse(build(:measurement_ticket, labels: labels)) + end + end + + test "the label is the whole test: the title does not make a ticket chg-loop" do + assert {:ok, %{chg_loop?: false}} = + Ticket.parse(build(:measurement_ticket, %{"title" => "chg-loop: a failure"})) + end + test "an empty body is nil rather than an empty string" do assert {:ok, %{body: nil}} = Ticket.parse(build(:measurement_ticket, %{"body" => ""})) end