Repository navigation
feat(review): give every PR a risk tier from its diff - #716
Conversation
Quest-Claim: c616e45272a39804
quest review sets each head's tier, low, medium, or high, before the reviewers run: a changed path the base's RISK.md, RISK.md itself, or a workflow or action holds high sets high with no model; a base with no RISK.md makes the rest medium; otherwise Claude Code's Haiku, or Codex's Sol at low effort, classifies the diff in one read-only turn. Each reviewer may raise the tier, never lower it. The tier is recorded in the PR body's ## Review section, read back by quest status, and rolled up per tier by quest usage, whose classifier sessions count as classifying. quest check reports a RISK.md it cannot read. Nothing acts on the tier yet; the quest now waits on the maintainer's week of reading it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mas by content A real run showed Claude Code passing its login check, then having its turn rejected through a gateway, which stopped the review although Codex could classify. A classifier that is logged out, has its login rejected, or hit its usage limit now passes the turn to the next; with none left, the run fails on the limit that resets first, or names why each failed. The same run had its reviewer reply without the new risk field: the clone's worktrees share one schema-<bound>.json, and a review from an older quest rewrote it between this run's write and Codex's read. Each schema file is now named by a digest of its content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se adjacent ** A run that leaves a reviewer out, such as a shadow with nothing to settle on the same head, settled the tier from its own rounds alone and dropped that reviewer's raise. Its last round still judged the head, so its raise now holds, and its record block names it; a new head drops it. A pattern with adjacent ** segments, such as **/**/wire.proto, compiled to an extra separator and matched nothing. Adjacent ** now match what one does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sk-tier # Conflicts: # README.md # src/cli/main.ts # src/cli/usage.ts # test/usage.test.ts
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 18 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
There was a problem hiding this comment.
ℹ️ No critical issues. Two small suggestions inline.
Reviewed changes
I reviewed the whole PR at effcd06: risk tiers for quest review, plus how quest status, quest usage, and quest check read them back.
src/core/risk.ts: adds tier ordering, the built-in always-high paths, glob matching where**must be a whole segment,RISK.md## Highparsing, andrecordedRiskreading the record back.- Classification in
review(): a path match makes the changehigh, a base with noRISK.mdmakes itmedium, and otherwise a Haiku turn sets it, falling back to Sol. The result is cached by merge base, head, andRISK.mdtext, and it runs only after the seat and--responseschecks pass. - Reviewer raises: the reply schema now requires
risk.settle()takes the riskiest tier, including the last round of a left-out seat on the same head. - Record and readers: adds the
- Risk:,- Classified:, and- Raised the risk tolines;Status.risk; theclassifyingrole; the tier column; and theBy risk tierrollup. - Content-named schema files: this fixes the race where a different
questrelease could rewrite a sharedschema-<n>.jsonmid-run.
claude-opus-5-5 | 𝕏
…n test fix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ No new issues found. Both findings from the last review are fixed.
Reviewed changes
I reviewed what changed since the review at effcd06: one merge of main and one fix commit.
- Fenced the classified reason:
raising()now wrapsrisk.reasonin<classified-reason>tags and tells the reviewer to treat it as data, the same wayresponsePartsfences<responses>. The test assertions now match the new prompt. - Restored
OnlineRow's doc comment: it sits onOnlineRowagain, andTierRowkeeps its own. - Planned the flaky test fix: the new quest
quest/m0/review-test-partial-log.mdreproduces the partial fake log in the rejected-login test and fixes it at its root.
claude-opus-5-5 | 𝕏

Implements quest/m0/benchmark/risk-tier.md and rewrites the quest to what remains: the maintainer's week of reading the recorded tiers before tier guardrails acts on them.
Summary
quest reviewnow gives each head a risk tier,low,medium, orhigh, with a one-sentence reason, before any reviewer runs. The tier comes from the diff and never from who wrote it. Nothing acts on it yet.RISK.md,.github/workflows/,.github/actions/, or a pattern in the## Highlist of the base'sRISK.mdsetshigh. The file is read from the base's tip, so a change cannot lower its own tier.RISK.md: every other change ismedium, and the reason says so.RISK.mdin one read-only turn. It is Claude Codeclaude-haiku-5-5at low effort, falling back to Codexgpt-6.1-solat low when Claude Code is missing, logged out, has its login rejected, or hit its usage limit. The result is cached per merge base, head, andRISK.md;--newclassifies again.risk; its prompt gives the classified tier. The head's tier is the riskiest.## Reviewsection after the Base line (- Risk:and- Classified:), with- Raised the risk to ...in a raising reviewer's block.quest status(risk <tier> <state> <classified> <reason>), and byquest usage: a tier column per PR, arisk:line, aBy risk tier:rollup of PRs, merges, reviewer raises, and later fixes, and aclassifyingrole for the classifier's session.quest checkreports a rootRISK.mdit cannot read, which would otherwise fail the next review.RISK.md:assets/CONTEXT.md,scripts/installer.sh, andscripts/release-plan.share always high; prose tells the classifier what else others rely on..git/quest/review/underschema-<bound>.json, so a review from anotherquestrelease could rewrite one between a run's write and the CLI's read (it happened in validation, below). They are now named by a digest of their content.Public API impact
quest/coreaddsTIERS,Tier,riskier,RISK_FILE,HighPath,ALWAYS_HIGH,patternError,matches,RiskFile,parseRiskFile,RecordedRisk,recordedRisk,PATHS, andDEFAULT.quest/github'sStatusgainsrisk: { tier, reason, classified, state } | null. Like the review record, a malformed risk line throws.quest reviewprintsrisk,classified,classifier,usage classify, and per reviewerraisedlines, and--jsongainsrisk.quest statusprints ariskline afterreview-record.quest usage's PR header becomes#<n> <size> <tier> <state>: ..., and--jsongains each PR'sriskand atiersrollup.risk, and the review state file gainsclassified. Internally,reviewRecord,recordRounds, androundLinestake the wholeReviewedrun.Choices made without asking
The quest's Plan listed agent's choices to revisit; here is where each landed:
CLASSIFIERSinsrc/cli/models.ts), since the tier never depends on the writer. Haiku is the cheapest, and the 200,000-character diff budget keeps it below its 100k-token surcharge. A larger diff gets only the stat, and the classifier reads the parts it needs.RISK.md. "A tier that rises reopens whatever the lower tier skipped" moved to guardrails' Plan, since nothing is skipped until then.RISK.mdand workflow files always high: kept, built in, plus.github/actions/. A rename out of a held path counts.quest usagealready counts reviewer raises and later fixes (## Caused by). The maintainer override with its reason moved to guardrails' Plan, since nothing reads an override until a tier gates something.medium, with the reason naming the missing file. ARISK.mdthat cannot be parsed does fail, with exit 2, andquest checkcatches it first.## Reviewrecord, not beside Size in## Models. The Goal said "beside its size".## Modelsis written by agents, while the review record is machine-owned and names the head it covers, soquest statuscan say whether the tier is current, stale, or diff-identical.risk -); guardrails will decide what such PRs need.RISK.mdholds only three always-high paths. A path rule makes every change under it high with no model, and guardrails will make high mean a person, so broader surfaces (skills, thequest/coreandquest/githubexports, record formats, the reviewer's sandbox) are described in prose for the classifier instead.Validation
main(whose feat(usage)!: price sessions through token-max, across SSH hosts #704 movedquest usageto token-max, so the merge carries the classifier'sclassifyingrole into its new session model),just checkpasses, andjust testpasses 1251 tests in 40 files. New tests cover:**, andRISK.mdparsing;quest status,quest usage,quest check, and content-named schema files.quest review(Sol xhigh, with the Sol high shadow) found two issues in round 1, both fixed: a left-out reviewer's raise on the same head was dropped, and adjacent**segments matched nothing. Rounds 2 and 3, the latter after the merge, were clean at 10/10.quest status 716printedrisk high current high it changes RISK.md, which is always high: ..., andquest usage --pr 716printed#716 L high open draft: ...with itsrisk:line.a login the API rejects fails as logged out ... (host CLAUDECODE, rejected in turn), where a fake killed mid-append left a partial log line. It passed on 3 reruns; see Follow-ups.RISK.md), with this checkout's binary:lowfor a reworded test comment ("a revert fully undoes it") andmediumfor a rewordedquest checksuccess line ("could break scripts that match its wording"), each in about 16.5k input tokens.quest reviewfailed because its reviewer replied withoutrisk. Another worktree's olderquesthad rewritten the sharedschema-0.jsonmid-run. After the content-named fix, the rerun passed end to end:risk medium,classified medium model, theclassifierandusage classifylines, and a reviewer reply carryingrisk.quest checkon this checkout accepts the newRISK.mdand reports a broken item by its line.Follow-ups
RISK.mdfor moq-dev/moq, moq-dev/moq.pro, and Kixel Code. Recommended: plan one in each repository's own tree. Without one, every PR there recordsmedium, and the maintainer's week of reading covers only repositories that have one.test/review.test.ts's fake log robust to a fake killed mid-append. Recommended: plan it (XS), since the flake can fail the merge queue.Models
🤖 Generated with Claude Code
(Written by Claude Opus 5.5 with xhigh)
Review
Rewritten by
quest reviewafter each round.Head: 48097c6
Base: origin/main at 501c858
Risk: high,
it changes RISK.md, which is always high: it sets the tier of every later changeClassified: high by the always-high paths
Reviewer: Codex, gpt-6.1-sol, xhigh effort
Session:
01a1d76e-1319-76b0-a0a2-dde9354413ef, round 4Score: 10/10,
The incremental changes are ready to land, with no new issues found by code inspection after excluding merged base commits.Rounds: 4; findings 2 (medium 1, low 1): fixed 2, refuted 0, open 0
Findings: none
Reviewer: Codex, gpt-6.1-sol, high effort, shadow
Session:
01a1d76e-1311-7be2-af3b-58119c1f5e68Rounds: 2; findings 2 (medium 1, low 1): fixed 2, refuted 0, open 0