Repository navigation
quest: run the 2026-11-18 mistake audit, and report review records quest usage cannot read #786
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
e7037d2
chore(quest): claim quest/m0/mistake-audit.md
kixelated 1102c24
fix(usage): report a review record it cannot read on its PR instead o…
kixelated 4c5b694
eval: add six moq-dev/quest escapes from the 2026-11-18 mistake audit
kixelated 5c6522b
quest: second mistake audit run, with its follow-ups
kixelated be71285
fix(usage): read a review record's reviewers and tier apart; name hos…
kixelated c50424b
quest: answer CodeRabbit's findings and record the merge pick's follo…
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # [M] Reviewers trace a changed contract to its dependents | ||
|
|
||
| ## Goal | ||
|
|
||
| `quest review`'s reviewers find more of the escapes where a change altered | ||
| a contract (a signature, a key's meaning, a default, a format, a guard, a | ||
| documented behavior) and left a dependent on the old one: a caller, a | ||
| consumer, a sibling implementation or path, a parallel registry, a test, a | ||
| bench, or the docs. `quest eval run` shows the gain on a set holding that | ||
| class, at no more clean-PR noise. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-19 (maintainer, merge pick of | ||
| [moq-dev/quest#786](https://github.com/moq-dev/quest/pull/786)): | ||
|
|
||
| - Keep it? ✅ Keep, in m0 (rejected: m1, drop) | ||
|
|
||
| From the 2026-11-18 [mistake audit](/quest/m0/mistake-audit.md) run: this | ||
| was the largest class of real escapes, about 41 of 204 across moq-dev/moq, | ||
| moq-dev/moq.pro, and moq-dev/quest. 26 of them were catchable or maybe from | ||
| the culprit's diff, and each had passed a local review. Neither the review | ||
| prompts in `src/cli/review.ts` nor a `REVIEW.md` asks a reviewer to look | ||
| past the diff for what depends on what it changed. A type check already | ||
| catches the callers a compiler sees, so what remains is semantic, which | ||
| leaves a reviewer instruction as the highest level that holds. | ||
|
|
||
| Recommended: | ||
|
|
||
| - Build the cases first. Add this class's catchable and maybe escapes to | ||
| an eval set with `quest eval add`, each defect located at the culprit's | ||
| head from the fix's diff. moq-dev/quest's own set holds two | ||
| ([moq-dev/quest#186](https://github.com/moq-dev/quest/pull/186) fixed by | ||
| [moq-dev/quest#632](https://github.com/moq-dev/quest/pull/632), and | ||
| [moq-dev/quest#585](https://github.com/moq-dev/quest/pull/585) fixed by | ||
| [moq-dev/quest#685](https://github.com/moq-dev/quest/pull/685)). For | ||
| moq-dev/moq, build a set in a clone with `--set` rather than waiting on | ||
| a PR there. Its pairs (culprit then fix) are 8541/8580, 8618/8817, | ||
| 7943/8920, 5003/8920, 6177/8927, 8956/9013, 7656/9023, 5283/9047, | ||
| 8896/9048, 7706/9060, 8596/9065, 2547/9116, 7844/8529, and 8561/8602. | ||
| Add clean PRs from the same weeks so noise is measured too. | ||
| - Then change the review prompt: ask the reviewer to name each contract | ||
| the diff changes and to search the repository for that contract's | ||
| dependents, reporting one it left on the old contract. Keep it a short | ||
| addition to the existing prompt, not a checklist. | ||
| - Score base and head builds with `quest eval run` on both sets, and put | ||
| both scores and their cost in the PR's `## Validation`, as this | ||
| repository's `AGENTS.md` requires for a review-prompt change. Land it | ||
| only if defects found rise and clean findings do not. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Routing evidence](/quest/m0/routing-evidence/README.md) - the escape | ||
| baseline reviewers are scored against |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # [M] The gate starts cheap commands at once | ||
|
|
||
| ## Goal | ||
|
|
||
| `quest gate` starts a command without waiting on the load average when its | ||
| own recent runs of the same invocation in the same repository show it uses | ||
| little CPU, so a cheap command an agent gated anyway costs no wait. It | ||
| still waits for free memory, and heavy commands queue as they do now. | ||
|
|
||
| ## Plan | ||
|
|
||
| From the 2026-11-18 [mistake audit](/quest/m0/mistake-audit.md) run, on | ||
| the gate records [moq-dev/quest#747](https://github.com/moq-dev/quest/pull/747) | ||
| added (2026-11-17 to 11-19, this host only): 887 of 3,048 gated runs (29%) | ||
| used under 5 CPU-seconds, together 0.16% of gated CPU, against 20% to 22% | ||
| before `quest context` said on 2026-11-15 to gate by CPU. They queued a | ||
| median 5 seconds, 163 of them over a minute, 46,000 agent-seconds in all. | ||
| Most were one class: `bun test` in moq-dev/moq, gated 347 times at a median | ||
| 0.5 CPU-seconds, 272 of them under 5. The prose rule already covers it and | ||
| the class still recurs, so the fix moves up to the gate itself. | ||
|
|
||
| Decided 2026-11-19 (maintainer, merge pick of | ||
| [moq-dev/quest#786](https://github.com/moq-dev/quest/pull/786)): | ||
|
|
||
| - Keep it? ✅ Keep, in m1 (rejected: m0, drop) | ||
|
|
||
| Recommended: | ||
|
|
||
| - Key the history by the full invocation the gate is handed (its argv) | ||
| and the repository, so `bun test <file>` and a whole-suite `bun test` | ||
| stay apart. Store only a hash of the argv in a local file the gate | ||
| owns, never the arguments, which can hold secrets; the shared record's | ||
| `command` keeps program names and subcommands alone, too coarse for this. | ||
| - Judge by the invocation's typical run, such as its median over recent | ||
| runs, so one cheap run of a usually heavy command, such as `just check` | ||
| with a warm cache, does not exempt it. | ||
| - Derive the cutoff from what a queued slot protects: a command whose | ||
| typical run keeps under one core busy for a few seconds cannot overload | ||
| the host. State the chosen value and the data behind it in the PR. | ||
| - An exempt start skips only the load wait. It still waits for free | ||
| memory, since CPU history says nothing of memory, and still enters the | ||
| ledger like any admission, charged at its recorded typical cores before | ||
| it is sampled. A burst of exempt starts is then bounded: once their | ||
| charged cores and the running commands' fill the host's CPUs, the next | ||
| one waits as today. | ||
| - An exempt run is still measured and recorded. An invocation queues again | ||
| once heavy runs are the majority of its recent window, as a median | ||
| implies, so a command that turned heavy is exempt for a few runs at most. | ||
| - Unmeasured hosts, such as macOS or Linux without a systemd user scope, | ||
| keep today's behavior. | ||
| - Leave `quest context`'s "gate by CPU" line as it is until the next audit | ||
| shows whether misgated cheap runs still cost wait time. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Mistake audit](/quest/m0/mistake-audit.md) - counts gating mistakes each | ||
| week | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.