diff --git a/quest/m0/benchmark/guardrails.md b/quest/m0/benchmark/guardrails.md index 3ece6e44..0d6d293e 100644 --- a/quest/m0/benchmark/guardrails.md +++ b/quest/m0/benchmark/guardrails.md @@ -49,6 +49,16 @@ merge on their own, so the starting list holds that class too. review an author without write access needs (rejected: a local `quest review`; no review). +Evidence from the 2026-11-18 [mistake audit](/quest/m0/mistake-audit.md) +run: of 234 merge-pick answers from 2026-11-11 to 11-19, across the +three repositories and the maintainer's hosts, 220 (94%) picked the PR the +agent recommended. Follow-up keep-or-drop answers picked it 79% of the +time, and other questions 73%. That supports widening the list once the +tiers are trusted, not merging every pick on its own. Decided 2026-11-19 +(maintainer, merge pick of +[moq-dev/quest#786](https://github.com/moq-dev/quest/pull/786)): ✅ keep +this evidence here (rejected: drop). + This quest edits the shipped skills and rules, which the maintainer asked for by planning it. diff --git a/quest/m0/mistake-audit.md b/quest/m0/mistake-audit.md index 03ef4e58..9cdb0bcb 100644 --- a/quest/m0/mistake-audit.md +++ b/quest/m0/mistake-audit.md @@ -45,7 +45,17 @@ Agent's choices, for the implementer to revisit. Each run: `--root`), for each PR's review rounds, findings fixed and refuted, hand-backs, and the later PRs naming it under `## Caused by`; the diffs of those culprits and their fixes; and the typed messages from - `quest messages --since --json`. + `quest messages --since --json` on every host the maintainer + types on, since `quest messages` reads only its own host. Where a host + lacks `quest`, copy a `bun run compile` build of this checkout there, + built for that host's OS and CPU (`bun build --compile --target` + when they differ from this one's), and remove it afterwards. Keep one host's counts apart when comparing + with a baseline that read only that host. +- Count a culprit pair once its fix merged since the last run, so a fix + open across a run's start counts in exactly one run. 245 pairs whose fix + opened since 2026-11-11 were read on 2026-11-18, too many for one session: split them by repository + into sub-agents of about 50 pairs, each writing one JSON line per pair + under `.tmp/`, and classify messages in another. - Read each culprit pair's `## Caused by` line before counting it. On 2026-11-11 only 7 of 20 pairs were escapes. `fixedBy` now counts only a merged fix's bullet that opens with the culprit's link, but a culprit @@ -83,7 +93,9 @@ Agent's choices, for the implementer to revisit. Each run: set with `quest eval add`, its defect located at the culprit's head from the fix's diff and confirmed against the culprit's own diff, so `quest eval run` scores reviewers on it. A private repository's set stays in - that repository. + that repository. Another repository's cases go in its run-PR follow-up + for `/quest-plan` there; moq-dev/moq and moq-dev/moq.pro had no set on + 2026-11-18. - Before proposing a fix, find the quest, check, or rule that already owns the class, and report whether its count fell since that landed. A rule whose class still recurs is the case for moving it up a level. @@ -112,69 +124,95 @@ Agent's choices, for the implementer to revisit. Each run: Check first next run: +- The 15 culprit pairs (11 in moq-dev/moq, 4 in moq-dev/moq.pro) whose fix + opened before 2026-11-11 and merged after it: the 2026-11-18 run keyed + pairs by when the fix opened, so neither run read them. Read them with + this run's. +- Contract changes that left a dependent on the old contract, the largest + escape class: about 41 of 204 real escapes, 26 catchable or maybe, all + past a local review. [Review dependents](/quest/m0/review-dependents.md) + takes it, if kept. +- Gating mistakes, after [the gate's cheap commands](/quest/m1/gate-cheap-commands.md) + lands, if kept: 887 of 3,048 gated runs (29%) used under 5 CPU-seconds, + together 0.16% of gated CPU, against 20% to 22% before the 2026-11-15 + rule. They queued 46,000 agent-seconds, a median 5 each. `bun test` in + moq-dev/moq made 347 of the gated runs at a median 0.5 CPU-seconds. + `quest review` was gated 16 times in 2.3 days, against 591 in the 14 + days before the rule. These records began with + [moq-dev/quest#747](https://github.com/moq-dev/quest/pull/747) on + 2026-11-17, so this is 2.3 days, on this host only. +- Ungated shell CPU: unmeasured. No host had a shell-call log on + 2026-11-18, so every `shell` list was empty; the hooks + [moq-dev/quest#747](https://github.com/moq-dev/quest/pull/747) added + were not installed. Planned in kixel-ai/dotfiles, if kept. - Stalls after the stop hook of - [moq-dev/quest#600](https://github.com/moq-dev/quest/pull/600) (landed - 2026-11-10T18:34Z): 3.8 per 100 typed messages after it against 6.5 - before, on only 27 hours and 3 stalls. -- Hand-run `/quest-complete` after `/quest-run` - ([moq-dev/quest#608](https://github.com/moq-dev/quest/pull/608), landed - 2026-11-10T20:04Z): 6.6 a day in its first day against 5.2 before, with - 3 `/quest-run` runs; `/quest-spawn` fell from 4.1 to 0.9 a day. + [moq-dev/quest#600](https://github.com/moq-dev/quest/pull/600): 5.0 per + 100 typed messages on all hosts (4.3 on the audit's host), against 3.8 + before. + Without the 12 asks for new work after a `/quest-run` ended, 2.4. - Output handed back unchecked, after the verification rule of - [moq-dev/quest#625](https://github.com/moq-dev/quest/pull/625) landed - 2026-11-11T18:43Z: no data yet. -- Heads that landed past a finding or decision, after - [moq-dev/quest#652](https://github.com/moq-dev/quest/pull/652) lands. -- Gating mistakes, after `quest context` said on 2026-11-15 to gate by - CPU, not run time: `quest gate quest review` ran 591 times in the 14 - days before, 126 of them on 11-12, at 2.2 CPU-seconds and 0.03 cores - over a 58-second median run; 20% to 22% of gate scopes over 3 days used - under 5 CPU-seconds, together 0.2% of gated CPU, most of them usually - heavy commands that were cheap that time. Ungated CPU was unmeasured - before - [moq-dev/quest#747](https://github.com/moq-dev/quest/pull/747); count - both from the records once it lands, from transcripts for earlier days. + [moq-dev/quest#625](https://github.com/moq-dev/quest/pull/625): 5, 1.1 + per 100, all on the two hosts the earlier baseline did not read; 0 on + the audit's host. +- Heads that landed past a related finding or decision after + [moq-dev/quest#652](https://github.com/moq-dev/quest/pull/652) landed + (2026-11-13T22:12Z): 2 in moq-dev/moq, each a finding the agent answered + by deferring it to a follow-up, which `quest merge` accepts as answered. + A class if it recurs. +- Hand-run `/quest-complete` after `/quest-run` + ([moq-dev/quest#608](https://github.com/moq-dev/quest/pull/608)): 1.7 a + day bare, none after 2026-11-15, against 6.6 in its first day. + `/quest-run` ran 3.6 a day; `/quest-spawn` was never run bare. - Whether moq-dev/moq and moq-dev/moq.pro planned the runtime follow-ups - of moq-dev/quest's 2026-11-11 run PR, and whether their escapes recurred. + of both runs: none of 2026-11-11's had a quest there on 2026-11-18. +- `## Review` records edited by hand: 96 skip records in the window, 3 of + which stopped `quest usage` for a whole repository until this run's PR + reported them per PR instead. -Baseline from the 2026-11-11 run (PRs updated 2026-11-09 to 11-11; typed -messages 2026-10-26 to 11-11, classed by one sub-agent, approximate): +Baseline from the 2026-11-18 run (PRs updated 2026-11-11 to 11-19; culprit +pairs whose fix opened since 2026-11-11; typed messages since 2026-11-11 on +the maintainer's three hosts, of which "the audit's host" below is the +one the 2026-11-11 baseline read alone; each class from sub-agents' hand labels, so +approximate): -- PRs: moq-dev/moq 194 (158 merged), moq-dev/moq.pro 150 (99), and - moq-dev/quest 69 (59). Hand-backs: 4, 20, and 0. 57 of the 316 merged - PRs record implementers that `quest usage` dropped - until its Implementer parsing was fixed. -- Escapes: 7 real of 20 culprit pairs. Catchable 2, maybe 2, runtime 3; - races 2 (one maybe, one runtime). All 7 culprits passed a local Sol - review. The routing audit's sample of 30 earlier escapes split 6, 17, - and 7, with 5 races. -- Landed past an open signal: 3 in moq-dev/moq, 2 catchable and 1 - maybe. A review finding posted after enqueue, a recorded decision the - head lacked, and a red platform run. [moq-dev/quest#652](https://github.com/moq-dev/quest/pull/652) takes the - first two; `quest merge`'s refusal of failed checks - ([moq-dev/quest#601](https://github.com/moq-dev/quest/pull/601)) covers - none of them. -- A scoped gate skipped a test the change reached, runtime: 2, one each in - moq-dev/moq and moq-dev/moq.pro. -- A shell pipeline under `pipefail` whose reader exits early, a race: 1 - this week in moq-dev/moq.pro, its third fix there. -- Named a PR that deferred the work, or kept older behavior, as a - culprit: 4 in moq-dev/moq; `quest context` now requires confirming - the culprit in the blamed PR's diff. -- Typed mid-session messages, per 100, before 2026-11-09 (841) then after - (131): stalls until nudged 6.5 then 4.6; proposals steered simpler 4.8 - then 0; misread intent 1.9 then 3.1; reported instead of fixed 0.7 then - 2.3; routine choices asked 3.4 then 1.5; answers that only picked the - recommendation 7.7 then 3.8 (`/quest-plan` 11, `/quest-complete` 10, - `/quest-spawn` 6, `/quest-iterate` 1, unknown skill 42). The after - window is under 3 days. -- `quest messages` counts more than the 2026-11-09 hand scan: 1,088 - mid-session messages to its 579 over the same days, likely from the - Codex and T3 Code threads and worktree directories it covers. +- PRs: moq-dev/moq 700 (635 merged), moq-dev/moq.pro 492 (431), and + moq-dev/quest 165 (151). `quest usage` attributed sessions from this + host only, as the others had neither `quest` nor `token-max`. +- Escapes: 204 real of 245 pairs. moq-dev/moq 141 of 164 (catchable 24, + maybe 61, runtime 56; races 24, 10 runtime); moq-dev/moq.pro 48 of 64 + (8, 27, 13; races 17, 8 runtime); moq-dev/quest 15 of 17 (4, 10, 1; + races 3). 83 of the 170 distinct culprits merged before 2026-11-04. + Not escapes: 41, mostly kept older behavior (11), deferred (6), or a + design the maintainer later reversed. +- Classes: a contract changed without its dependents, about 41; races, + 44; Linux behavior assumed in portable code or platform-gated code with + no off-platform build, 12 runtime (10 in moq-dev/moq, caught only by + nightly macOS and Windows lanes or by none); unbounded or super-linear + growth, 9. +- Merged past an open signal, any escape: 30 in moq-dev/moq, 3 in + moq-dev/moq.pro, 2 in moq-dev/quest, most of them red checks or findings + unrelated to the defect. +- Eval set: moq-dev/quest's grew from 26 to 32 cases with this run's + escapes. +- Typed messages: 577, 118 set aside as machine-written, 459 left (230 on + the audit's host). Per 100, all hosts then the audit's host: stalls until nudged 5.0 then + 4.3; waited on a blocked dependency instead of removing it 4.1 then 5.7 + (8 behind Pullfrog); routine choices asked 4.1 then 2.6 (7 asked to plan + follow-ups a session had found); resumed after a rate limit or API + error 3.5 then 4.8; reported instead of fixed 2.6 then 3.5; UI and copy + rejected 2.6 then 2.6, all in moq-dev/moq.pro; proposals steered simpler + 2.0 then 0.9; searched only its own host 1.5 then 1.7; followed a plan + past findings that undercut it 1.3 then 1.3; output handed back + unchecked 1.1 then 0; misread intent 0.7 then 0.4. +- Decision prompts: 1,592 answers, 1,517 with a recommendation, 80% of + which picked it. Merge picks 94% (220 of 234), follow-up keep-or-drop + 79%, other questions 73%. By skill: `/quest-merge` 90%, + `/quest-complete` 88%, `/quest-run` 85%, `/quest-afk` 79%, + `/quest-spawn` 75%, `/quest-plan` 68%. ## Scheduled -- 2026-11-18 - one week after the 2026-11-11 run +- 2026-11-25 - one week after the 2026-11-18 run ## Related diff --git a/quest/m0/review-dependents.md b/quest/m0/review-dependents.md new file mode 100644 index 00000000..9e32cb2d --- /dev/null +++ b/quest/m0/review-dependents.md @@ -0,0 +1,54 @@ +# [M] Reviewers trace a changed contract to its dependents + +## Goal + +`quest review`'s reviewers find more of the escapes where a change altered +a contract (a signature, a key's meaning, a default, a format, a guard, a +documented behavior) and left a dependent on the old one: a caller, a +consumer, a sibling implementation or path, a parallel registry, a test, a +bench, or the docs. `quest eval run` shows the gain on a set holding that +class, at no more clean-PR noise. + +## Plan + +Decided 2026-11-19 (maintainer, merge pick of +[moq-dev/quest#786](https://github.com/moq-dev/quest/pull/786)): + +- Keep it? ✅ Keep, in m0 (rejected: m1, drop) + +From the 2026-11-18 [mistake audit](/quest/m0/mistake-audit.md) run: this +was the largest class of real escapes, about 41 of 204 across moq-dev/moq, +moq-dev/moq.pro, and moq-dev/quest. 26 of them were catchable or maybe from +the culprit's diff, and each had passed a local review. Neither the review +prompts in `src/cli/review.ts` nor a `REVIEW.md` asks a reviewer to look +past the diff for what depends on what it changed. A type check already +catches the callers a compiler sees, so what remains is semantic, which +leaves a reviewer instruction as the highest level that holds. + +Recommended: + +- Build the cases first. Add this class's catchable and maybe escapes to + an eval set with `quest eval add`, each defect located at the culprit's + head from the fix's diff. moq-dev/quest's own set holds two + ([moq-dev/quest#186](https://github.com/moq-dev/quest/pull/186) fixed by + [moq-dev/quest#632](https://github.com/moq-dev/quest/pull/632), and + [moq-dev/quest#585](https://github.com/moq-dev/quest/pull/585) fixed by + [moq-dev/quest#685](https://github.com/moq-dev/quest/pull/685)). For + moq-dev/moq, build a set in a clone with `--set` rather than waiting on + a PR there. Its pairs (culprit then fix) are 8541/8580, 8618/8817, + 7943/8920, 5003/8920, 6177/8927, 8956/9013, 7656/9023, 5283/9047, + 8896/9048, 7706/9060, 8596/9065, 2547/9116, 7844/8529, and 8561/8602. + Add clean PRs from the same weeks so noise is measured too. +- Then change the review prompt: ask the reviewer to name each contract + the diff changes and to search the repository for that contract's + dependents, reporting one it left on the old contract. Keep it a short + addition to the existing prompt, not a checklist. +- Score base and head builds with `quest eval run` on both sets, and put + both scores and their cost in the PR's `## Validation`, as this + repository's `AGENTS.md` requires for a review-prompt change. Land it + only if defects found rise and clean findings do not. + +## Related + +- [Routing evidence](/quest/m0/routing-evidence/README.md) - the escape + baseline reviewers are scored against diff --git a/quest/m1/gate-cheap-commands.md b/quest/m1/gate-cheap-commands.md new file mode 100644 index 00000000..b817d0c6 --- /dev/null +++ b/quest/m1/gate-cheap-commands.md @@ -0,0 +1,57 @@ +# [M] The gate starts cheap commands at once + +## Goal + +`quest gate` starts a command without waiting on the load average when its +own recent runs of the same invocation in the same repository show it uses +little CPU, so a cheap command an agent gated anyway costs no wait. It +still waits for free memory, and heavy commands queue as they do now. + +## Plan + +From the 2026-11-18 [mistake audit](/quest/m0/mistake-audit.md) run, on +the gate records [moq-dev/quest#747](https://github.com/moq-dev/quest/pull/747) +added (2026-11-17 to 11-19, this host only): 887 of 3,048 gated runs (29%) +used under 5 CPU-seconds, together 0.16% of gated CPU, against 20% to 22% +before `quest context` said on 2026-11-15 to gate by CPU. They queued a +median 5 seconds, 163 of them over a minute, 46,000 agent-seconds in all. +Most were one class: `bun test` in moq-dev/moq, gated 347 times at a median +0.5 CPU-seconds, 272 of them under 5. The prose rule already covers it and +the class still recurs, so the fix moves up to the gate itself. + +Decided 2026-11-19 (maintainer, merge pick of +[moq-dev/quest#786](https://github.com/moq-dev/quest/pull/786)): + +- Keep it? ✅ Keep, in m1 (rejected: m0, drop) + +Recommended: + +- Key the history by the full invocation the gate is handed (its argv) + and the repository, so `bun test ` and a whole-suite `bun test` + stay apart. Store only a hash of the argv in a local file the gate + owns, never the arguments, which can hold secrets; the shared record's + `command` keeps program names and subcommands alone, too coarse for this. +- Judge by the invocation's typical run, such as its median over recent + runs, so one cheap run of a usually heavy command, such as `just check` + with a warm cache, does not exempt it. +- Derive the cutoff from what a queued slot protects: a command whose + typical run keeps under one core busy for a few seconds cannot overload + the host. State the chosen value and the data behind it in the PR. +- An exempt start skips only the load wait. It still waits for free + memory, since CPU history says nothing of memory, and still enters the + ledger like any admission, charged at its recorded typical cores before + it is sampled. A burst of exempt starts is then bounded: once their + charged cores and the running commands' fill the host's CPUs, the next + one waits as today. +- An exempt run is still measured and recorded. An invocation queues again + once heavy runs are the majority of its recent window, as a median + implies, so a command that turned heavy is exempt for a few runs at most. +- Unmeasured hosts, such as macOS or Linux without a systemd user scope, + keep today's behavior. +- Leave `quest context`'s "gate by CPU" line as it is until the next audit + shows whether misgated cheap runs still cost wait time. + +## Related + +- [Mistake audit](/quest/m0/mistake-audit.md) - counts gating mistakes each + week diff --git a/review-eval.json b/review-eval.json index b45a49f1..e1813d2c 100644 --- a/review-eval.json +++ b/review-eval.json @@ -355,6 +355,33 @@ } ] }, + { + "pull": 553, + "base": "fcca33ed1754b8c29df0840f97a967a40c4ee35c", + "head": "44997f1a0c820fe5e3e7891a432fa0e918cd37c3", + "size": "M", + "tier": "medium", + "reason": "the base has no RISK.md to name what is always high or customer-facing, so every change is medium", + "writer": "claude-opus-5-5", + "defects": [ + { + "fix": 771, + "what": "The gh post checks took `args[1]` as the subcommand, so `gh pr -R OWNER/REPO comment ...` skipped the footer and em dash checks.", + "at": [ + { + "file": "src/cli/rules.ts", + "from": 368, + "to": 368 + }, + { + "file": "src/cli/rules.ts", + "from": 206, + "to": 206 + } + ] + } + ] + }, { "pull": 570, "base": "738bbdecbd0e4482e038a4aac2a0107b815e8985", @@ -435,6 +462,87 @@ } ] }, + { + "pull": 600, + "base": "eaa0831877a271c1bd4b7f90efd013ee8d24581a", + "head": "5fba1128468f5fd6c439aaf45f991562e093957c", + "size": "M", + "tier": "medium", + "reason": "the base has no RISK.md to name what is always high or customer-facing, so every change is medium", + "writer": "claude-opus-5-5", + "defects": [ + { + "fix": 747, + "what": "The `quest` PATH sentence was spliced into the start of the footer-check paragraph, garbling it and duplicating the PATH note below.", + "at": [ + { + "file": "README.md", + "from": 1781, + "to": 1781 + } + ] + } + ] + }, + { + "pull": 621, + "base": "855b46d46bbb82621045402d89720d867419806a", + "head": "9c37c2bca3c88eadeef9eff87599f749bb5f1a78", + "size": "M", + "tier": "medium", + "reason": "the base has no RISK.md to name what is always high or customer-facing, so every change is medium", + "writer": "claude-opus-5-5", + "defects": [ + { + "fix": 722, + "what": "Running the reviewer seats concurrently and aborting the sibling on first failure lets a fake CLI die mid-append, while the tests still parse one shared append-only log line by line.", + "at": [ + { + "file": "src/cli/review.ts", + "from": 1635, + "to": 1641 + }, + { + "file": "test/review.test.ts", + "from": 338, + "to": 343 + } + ] + } + ] + }, + { + "pull": 627, + "base": "f3301efd8c7831c7aa33ae2e6f13995170f3e205", + "head": "5b82f7b4aa0abefffaf1de9a62f34558ccf02387", + "size": "S", + "tier": "medium", + "reason": "the base has no RISK.md to name what is always high or customer-facing, so every change is medium", + "writer": "claude-opus-5-5", + "defects": [ + { + "fix": 763, + "what": "Handbacks quoted only the gate's failure and dropped the fixer's own output, so a fixer that crashed before the agent started named no cause.", + "at": [ + { + "file": "src/cli/land-fixer.ts", + "from": 72, + "to": 73 + }, + { + "file": "src/cli/land-fixer.ts", + "from": 1192, + "to": 1192 + }, + { + "file": "src/cli/land-watch-fix.ts", + "from": 690, + "to": 690 + } + ] + } + ] + }, { "pull": 655, "base": "93c9c2ef65f074e82094bf5272076db44ac19030", @@ -493,6 +601,50 @@ ] } ] + }, + { + "pull": 718, + "base": "85d0a78bd02b4dc7063abdfb8525f4ce9c8e1618", + "head": "4eddf305f23610adb1fe61805f9fd2efbd88eed6", + "size": "L", + "tier": "high", + "reason": "The change introduces a public record format, changes usage records consumers parse, and sends session data with bearer tokens to configured endpoints, so failures can break Kixel Code and scripts or expose data that a revert cannot recover.", + "writer": "claude-opus-5-5", + "defects": [ + { + "fix": 748, + "what": "The sender settled each queued stop after one successful upload, so a session sent before its branch's PR opened was never rebuilt to name that PR.", + "at": [ + { + "file": "src/cli/send.ts", + "from": 457, + "to": 458 + } + ] + } + ] + }, + { + "pull": 751, + "base": "f3bf9ca458a272cee7053e2130a9c77900c3d74b", + "head": "4f903210afdc40108ce76285d86a5fbfa93a6934", + "size": "S", + "tier": "high", + "reason": "it changes .github/workflows/pullfrog.yml, which is always high: workflows run with the repository's secrets and decide what merges and ships", + "writer": "claude-sonnet-5-5", + "defects": [ + { + "fix": 756, + "what": "The Nix installer step left the action's default `trust-runner-user: true`, so the agent's shell could reach root through the Nix daemon.", + "at": [ + { + "file": ".github/workflows/pullfrog.yml", + "from": 35, + "to": 37 + } + ] + } + ] } ] } diff --git a/src/cli/main.ts b/src/cli/main.ts index 98ea3b2c..86bbc3b5 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -1410,7 +1410,7 @@ export function main(argv: string[], out: Output): number | Promise { "this host's transcripts, no prompt or reply text read out. A session found on several hosts counts once, at its " + "costliest copy, and the report says how many copies it folded from which hosts; a session with a request " + "token-max could not price or a line it could not read is listed, its cost a lower bound. A host that fails fails the report. " + - "Each PR also shows its ## Models size, its risk tier from the ## Review record and whether a reviewer raised it, its implementers, and as unread each Implementer line it cannot read, " + + "Each PR also shows its ## Models size, its risk tier from the ## Review record and whether a reviewer raised it, its implementers, and as unread each Implementer line and ## Review record it cannot read, " + "each reviewer the ## Review record names, with its latest round and score and its tally of findings across every round and in its first, marked when it is the shadow, " + "where a merged PR whose last round left only low findings open counts them settled, since the author fixes or refutes those before merge, " + "the later merged PRs whose ## Caused by bullets open with its link, and each online reviewer's record from GitHub: whether it " + diff --git a/src/cli/review-record.ts b/src/cli/review-record.ts index 30e5bf03..23b42d10 100644 --- a/src/cli/review-record.ts +++ b/src/cli/review-record.ts @@ -189,7 +189,7 @@ export function classifierSessions(body: string): string[] { /** * Each reviewer's latest round `body`'s `## Review` section records, or none when `quest review` wrote none - * there, as when the section records a skipped review or only the land fixer's fixes. + * there, as when the section records a skipped review, even under the record's mark, or only the land fixer's fixes. */ export function parseReview(body: string): ReviewSummary[] { const section = reviewSection(body); @@ -202,7 +202,6 @@ export function parseReview(body: string): ReviewSummary[] { .slice(at) .split(/^(?=- Reviewer: )/m) .slice(1); - if (blocks.length === 0) throw new Error("the ## Review record names no reviewer"); return blocks.map((text) => { const match = (pattern: RegExp) => { const found = pattern.exec(text); diff --git a/src/cli/usage.ts b/src/cli/usage.ts index 766c467d..82cb3779 100644 --- a/src/cli/usage.ts +++ b/src/cli/usage.ts @@ -114,6 +114,8 @@ export interface PullRow { review: ReviewSummary[]; /** The risk tier the record names for its head, null when it names none. */ risk: RecordedRisk | null; + /** Why part of its `## Review` record could not be read: its reviewers, its tier, or both, which it then lacks. */ + unreadReview: string | null; /** Who made each mechanical fix the land fixer recorded, or `quest land watch` for a base merge it made alone. */ landFixes: string[]; /** Later PRs that list it under `## Caused by`. */ @@ -532,30 +534,49 @@ function pullRows(pulls: Pull[], repository: Repository): PullRow[] { for (const each of pulls.filter((pull) => pull.state === "merged")) for (const cause of causes(each.body, repository)) if (cause !== each.number) fixedBy.set(cause, [...(fixedBy.get(cause) ?? []), each.number]); - return pulls.map((each) => { - const own = recorded(each.body); - return { - number: each.number, - url: each.url, - title: each.title, - branch: each.branch, - repository: each.repository, - state: each.state, - draft: each.draft, - handedBack: each.draft && handedBack(each.body), - size: own.sizes.map(sized).find((s) => s !== null) ?? null, - implementers: own.implementers.flatMap(({ fields }) => - fields ? [{ ...fields, size: sized(fields.size) }] : [], - ), - unread: own.implementers.filter(({ fields }) => fields === null).map(({ item }) => item), - review: settledOnMerge(parseReview(each.body), each.state === "merged"), - risk: recordedRisk(each.body), - landFixes: fixes(each.body), - fixedBy: (fixedBy.get(each.number) ?? []).sort((a, b) => a - b), - online: [], - sessions: [], - }; - }); + return pulls.map((each) => pullRow(each, fixedBy)); +} + +/** `each`'s record, given the merged PRs that name each culprit as `fixedBy`. */ +function pullRow(each: Pull, fixedBy: Map): PullRow { + const own = recorded(each.body); + // The `## Review` record is its writer's text, at times edited by hand, so one this version cannot read is + // reported on its PR instead of stopping every other PR's. Its reviewers and its tier are read apart, so a + // garbled line in one keeps the other. + const unread: string[] = []; + const read = (parse: () => T, fallback: T): T => { + try { + return parse(); + } catch (error) { + unread.push(error instanceof Error ? error.message : String(error)); + return fallback; + } + }; + const review = read(() => settledOnMerge(parseReview(each.body), each.state === "merged"), []); + const risk = read(() => recordedRisk(each.body), null); + const unreadReview = unread.length ? unread.join("; ") : null; + return { + number: each.number, + url: each.url, + title: each.title, + branch: each.branch, + repository: each.repository, + state: each.state, + draft: each.draft, + handedBack: each.draft && handedBack(each.body), + size: own.sizes.map(sized).find((s) => s !== null) ?? null, + implementers: own.implementers.flatMap(({ fields }) => + fields ? [{ ...fields, size: sized(fields.size) }] : [], + ), + unread: own.implementers.filter(({ fields }) => fields === null).map(({ item }) => item), + review, + risk, + unreadReview, + landFixes: fixes(each.body), + fixedBy: (fixedBy.get(each.number) ?? []).sort((a, b) => a - b), + online: [], + sessions: [], + }; } const add = (to: Tokens, from: Tokens) => { @@ -1389,6 +1410,7 @@ export function usageLines(report: Usage): string[] { lines.push(` implementer: ${i.harness}, ${i.model}, ${i.effort} effort${notes ? ` (${notes})` : ""}`); } for (const item of row.unread) lines.push(` unread: ${item}`); + if (row.unreadReview) lines.push(` unread review: ${row.unreadReview}`); for (const r of row.review) lines.push( ` review: ${r.harness}, ${r.model}, ${r.effort} effort${r.shadow ? ", shadow" : ""}; ${r.round === null ? "not in the latest run" : `round ${r.round}, score ${r.score ?? "-"}/10`}; ${r.tally === null ? "no tally" : `rounds ${r.tally.rounds}, ${findingsText(r.tally)}`}`, diff --git a/test/usage.test.ts b/test/usage.test.ts index 8eec15c3..f75eb6b4 100644 --- a/test/usage.test.ts +++ b/test/usage.test.ts @@ -427,6 +427,12 @@ describe("PR records", () => { test("a skipped review or a fixer-only section has no round", () => { expect(parseReview("## Review\n\nSkipped: too trivial to need one.\n")).toEqual([]); + // A skip written by hand under the mark a round rewrote, as `/quest-review` allows. + expect( + parseReview( + "## Review\n\nRewritten by `quest review` after each round.\n\n- Head: 6c2ce3d381 (skipped review)\n- Skip: one reworded sentence.\n", + ), + ).toEqual([]); expect(parseReview("## Review\n\nmechanical fix by quest land watch: `merged main` (head x)\n")).toEqual([]); expect(fixes("## Review\n\nmechanical fix by quest land watch: `merged main` (head x)\n")).toEqual([ "quest land watch", @@ -1095,6 +1101,40 @@ describe("usage", () => { expect(report.unattributed!.sessions).toBe(6); }); + test("reports the part of a review record it cannot read on its PR and reads the rest", async () => { + const record = (...lines: string[]) => + ["## Review", "", "Rewritten by `quest review` after each round.", "", ...lines, ""].join("\n"); + const head = `- Head: ${"a".repeat(40)}`; + const reviewer = "- Reviewer: Codex, gpt-6.1-sol, high effort"; + // #30's tier survives its garbled Rounds line, and #31's reviewer its garbled Risk line. + const garbledRounds = record(head, RISK, reviewer, "- Rounds: garbled"); + const garbledRisk = record( + head, + "- Risk: garbled", + reviewer, + "- Rounds: 2; findings 1 (medium 1): fixed 1, refuted 0, open 0", + ); + const { root, deps, sessions } = setup({ + pulls: [ + pr(30, "quest/m0/bad", garbledRounds, "2026-11-06T00:00:00Z", { updated_at: "2026-11-07T00:00:00Z" }), + pr(31, "quest/m0/worse", garbledRisk, "2026-11-06T00:00:00Z", { updated_at: "2026-11-07T00:00:00Z" }), + ], + }); + const report = await usage(root, { since: "2026-11-05" }, deps(sessions)); + const rounds = report.pulls.find((p) => p.number === 30)!; + expect(rounds.review).toEqual([]); + expect(rounds.risk?.tier).toBe("high"); + expect(rounds.unreadReview).toBe("the ## Review record's Rounds line is malformed: garbled"); + const risk = report.pulls.find((p) => p.number === 31)!; + expect(risk.review.map((r) => r.model)).toEqual(["gpt-6.1-sol"]); + expect(risk.risk).toBeNull(); + expect(risk.unreadReview).toBe("the ## Review record's risk is malformed: garbled"); + expect(report.pulls.find((p) => p.number === 10)!.unreadReview).toBeNull(); + expect(usageLines(report).join("\n")).toContain( + " unread review: the ## Review record's Rounds line is malformed: garbled", + ); + }); + test("rolls up the writer A/B by size and arm, leaving out a PR a writer ran off its arm", async () => { const models = (...lines: string[]) => [