Skip to content

feat(review): give every PR a risk tier from its diff - #716

Merged
kixelated merged 7 commits into
mainfrom
quest/m0/benchmark/risk-tier
Nov 14, 2026
Merged

kixelated merged 7 commits into
mainfrom
quest/m0/benchmark/risk-tier

Conversation

@kixelated

@kixelated kixelated commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Implements quest/m0/benchmark/risk-tier.md and rewrites the quest to what remains: the maintainer's week of reading the recorded tiers before tier guardrails acts on them.

Summary

quest review now gives each head a risk tier, low, medium, or high, with a one-sentence reason, before any reviewer runs. The tier comes from the diff and never from who wrote it. Nothing acts on it yet.

  • Paths held high, no model. A changed path, either side of a rename, matching RISK.md, .github/workflows/, .github/actions/, or a pattern in the ## High list of the base's RISK.md sets high. The file is read from the base's tip, so a change cannot lower its own tier.
  • No RISK.md: every other change is medium, and the reason says so.
  • Otherwise a classifier reads the diff and the base's RISK.md in one read-only turn. It is Claude Code claude-haiku-5-5 at low effort, falling back to Codex gpt-6.1-sol at low when Claude Code is missing, logged out, has its login rejected, or hit its usage limit. The result is cached per merge base, head, and RISK.md; --new classifies again.
  • Reviewers may raise, never lower. Each reviewer's reply schema now requires risk; its prompt gives the classified tier. The head's tier is the riskiest.
  • Recorded in the PR body's ## Review section after the Base line (- Risk: and - Classified:), with - Raised the risk to ... in a raising reviewer's block.
  • Read back by quest status (risk <tier> <state> <classified> <reason>), and by quest usage: a tier column per PR, a risk: line, a By risk tier: rollup of PRs, merges, reviewer raises, and later fixes, and a classifying role for the classifier's session.
  • quest check reports a root RISK.md it cannot read, which would otherwise fail the next review.
  • This repository's own RISK.md: assets/CONTEXT.md, scripts/installer.sh, and scripts/release-plan.sh are always high; prose tells the classifier what else others rely on.
  • Fixed in passing: review schema files lived in the clone's shared .git/quest/review/ under schema-<bound>.json, so a review from another quest release could rewrite one between a run's write and the CLI's read (it happened in validation, below). They are now named by a digest of their content.

Public API impact

  • quest/core adds TIERS, Tier, riskier, RISK_FILE, HighPath, ALWAYS_HIGH, patternError, matches, RiskFile, parseRiskFile, RecordedRisk, recordedRisk, PATHS, and DEFAULT.
  • quest/github's Status gains risk: { tier, reason, classified, state } | null. Like the review record, a malformed risk line throws.
  • CLI output: quest review prints risk, classified, classifier, usage classify, and per reviewer raised lines, and --json gains risk. quest status prints a risk line after review-record. quest usage's PR header becomes #<n> <size> <tier> <state>: ..., and --json gains each PR's risk and a tiers rollup.
  • The reviewer reply schema requires risk, and the review state file gains classified. Internally, reviewRecord, recordRounds, and roundLines take the whole Reviewed run.

Choices made without asking

The quest's Plan listed agent's choices to revisit; here is where each landed:

  1. Classify before the review, path match first: kept. The classifier list is the same for every PR (CLASSIFIERS in src/cli/models.ts), since the tier never depends on the writer. Haiku is the cheapest, and the 200,000-character diff budget keeps it below its 100k-token surcharge. A larger diff gets only the stat, and the classifier reads the parts it needs.
  2. The reviewer raises, never lowers: kept, enforced by taking the maximum, not by trusting the prompt.
  3. A later head is classified again: kept, cached per merge base, head, and RISK.md. "A tier that rises reopens whatever the lower tier skipped" moved to guardrails' Plan, since nothing is skipped until then.
  4. RISK.md and workflow files always high: kept, built in, plus .github/actions/. A rename out of a held path counts.
  5. Accuracy two ways: the per-tier rollup in quest usage already counts reviewer raises and later fixes (## Caused by). The maintainer override with its reason moved to guardrails' Plan, since nothing reads an override until a tier gates something.
  6. "Fail loud" on a repository with no file: read as "say why on every PR", not "fail the review". Every PR there is medium, with the reason naming the missing file. A RISK.md that cannot be parsed does fail, with exit 2, and quest check catches it first.
  7. The tier lives in the ## Review record, not beside Size in ## Models. The Goal said "beside its size". ## Models is written by agents, while the review record is machine-owned and names the head it covers, so quest status can say whether the tier is current, stale, or diff-identical.
  8. A review skipped as trivial records no tier (risk -); guardrails will decide what such PRs need.
  9. This repository's RISK.md holds only three always-high paths. A path rule makes every change under it high with no model, and guardrails will make high mean a person, so broader surfaces (skills, the quest/core and quest/github exports, record formats, the reviewer's sandbox) are described in prose for the classifier instead.

Validation

  • After merging main (whose feat(usage)!: price sessions through token-max, across SSH hosts #704 moved quest usage to token-max, so the merge carries the classifier's classifying role into its new session model), just check passes, and just test passes 1251 tests in 40 files. New tests cover:
    • patterns, including adjacent **, and RISK.md parsing;
    • the path-held tier, including a rename out of a workflow;
    • the Haiku classifier and its reuse, the Codex fallback, and fall-through on a rejected or limited turn;
    • reviewer raises in the record, including a left-out shadow's raise on the same head;
    • quest status, quest usage, quest check, and content-named schema files.
  • quest review (Sol xhigh, with the Sol high shadow) found two issues in round 1, both fixed: a left-out reviewer's raise on the same head was dropped, and adjacent ** segments matched nothing. Rounds 2 and 3, the latter after the merge, were clean at 10/10.
  • With this PR's record in place, this checkout's quest status 716 printed risk high current high it changes RISK.md, which is always high: ..., and quest usage --pr 716 printed #716 L high open draft: ... with its risk: line.
  • One flake seen once and not caused by this change: a login the API rejects fails as logged out ... (host CLAUDECODE, rejected in turn), where a fake killed mid-append left a partial log line. It passed on 3 reruns; see Follow-ups.
  • Real CLIs, on a throwaway local branch based on this branch's first push (so its base had a RISK.md), with this checkout's binary:
    • The classifier set low for a reworded test comment ("a revert fully undoes it") and medium for a reworded quest check success line ("could break scripts that match its wording"), each in about 16.5k input tokens.
    • On this host, Claude Code passes its login check but its turn is rejected through the gateway. That stopped the first run and exposed the fall-through gap, now fixed; Codex classified instead.
    • The first end-to-end quest review failed because its reviewer replied without risk. Another worktree's older quest had rewritten the shared schema-0.json mid-run. After the content-named fix, the rerun passed end to end: risk medium, classified medium model, the classifier and usage classify lines, and a reviewer reply carrying risk.
    • quest check on this checkout accepts the new RISK.md and reports a broken item by its line.
  • Not exercised for real: Claude Code's classifier turn, since no Claude Code login works on this host.

Follow-ups

  • A RISK.md for moq-dev/moq, moq-dev/moq.pro, and Kixel Code. Recommended: plan one in each repository's own tree. Without one, every PR there records medium, and the maintainer's week of reading covers only repositories that have one.
  • Make test/review.test.ts's fake log robust to a fake killed mid-append. Recommended: plan it (XS), since the flake can fail the merge queue.

Models

  • Size: L
  • Implementer: Claude Code, claude-opus-5-5, xhigh effort

🤖 Generated with Claude Code

(Written by Claude Opus 5.5 with xhigh)

Review

Rewritten by quest review after each round.

  • Head: 48097c6

  • Base: origin/main at 501c858

  • Risk: high, it changes RISK.md, which is always high: it sets the tier of every later change

  • Classified: high by the always-high paths

  • Reviewer: Codex, gpt-6.1-sol, xhigh effort

  • Session: 01a1d76e-1319-76b0-a0a2-dde9354413ef, round 4

  • Score: 10/10, The incremental changes are ready to land, with no new issues found by code inspection after excluding merged base commits.

  • Rounds: 4; findings 2 (medium 1, low 1): fixed 2, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, high effort, shadow

  • Session: 01a1d76e-1311-7be2-af3b-58119c1f5e68

  • Rounds: 2; findings 2 (medium 1, low 1): fixed 2, refuted 0, open 0

kixelated and others added 3 commits November 13, 2026 15:46
quest review sets each head's tier, low, medium, or high, before the
reviewers run: a changed path the base's RISK.md, RISK.md itself, or a
workflow or action holds high sets high with no model; a base with no
RISK.md makes the rest medium; otherwise Claude Code's Haiku, or Codex's
Sol at low effort, classifies the diff in one read-only turn. Each
reviewer may raise the tier, never lower it. The tier is recorded in the
PR body's ## Review section, read back by quest status, and rolled up per
tier by quest usage, whose classifier sessions count as classifying.
quest check reports a RISK.md it cannot read. Nothing acts on the tier
yet; the quest now waits on the maintainer's week of reading it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mas by content

A real run showed Claude Code passing its login check, then having its
turn rejected through a gateway, which stopped the review although Codex
could classify. A classifier that is logged out, has its login rejected,
or hit its usage limit now passes the turn to the next; with none left,
the run fails on the limit that resets first, or names why each failed.

The same run had its reviewer reply without the new risk field: the
clone's worktrees share one schema-<bound>.json, and a review from an
older quest rewrote it between this run's write and Codex's read. Each
schema file is now named by a digest of its content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Give every PR a risk tier from its diff feat(review): give every PR a risk tier from its diff Nov 14, 2026
kixelated and others added 2 commits November 13, 2026 16:29
…se adjacent **

A run that leaves a reviewer out, such as a shadow with nothing to settle
on the same head, settled the tier from its own rounds alone and dropped
that reviewer's raise. Its last round still judged the head, so its raise
now holds, and its record block names it; a new head drops it.

A pattern with adjacent ** segments, such as **/**/wire.proto, compiled
to an extra separator and matched nothing. Adjacent ** now match what one
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sk-tier

# Conflicts:
#	README.md
#	src/cli/main.ts
#	src/cli/usage.ts
#	test/usage.test.ts
@kixelated
kixelated marked this pull request as ready for review November 14, 2026 00:41
@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 18 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2ea6b907-0997-4295-8187-79b665e69e69
Commits

Reviewing files that changed from the base of the PR and between 501c858 and 48097c6.

Files selected for processing: 19

  • README.md
  • RISK.md
  • quest/m0/benchmark/guardrails.md
  • quest/m0/benchmark/risk-tier.md
  • quest/m0/review-test-partial-log.md
  • src/cli/main.ts
  • src/cli/models.ts
  • src/cli/review-record.ts
  • src/cli/review.ts
  • src/cli/status.ts
  • src/cli/usage.ts
  • src/core/index.ts
  • src/core/risk.ts
  • src/github/status.ts
  • test/init.test.ts
  • test/review.test.ts
  • test/risk.test.ts
  • test/status.test.ts
  • test/usage.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues. Two small suggestions inline.

Reviewed changes

I reviewed the whole PR at effcd06: risk tiers for quest review, plus how quest status, quest usage, and quest check read them back.

  • src/core/risk.ts: adds tier ordering, the built-in always-high paths, glob matching where ** must be a whole segment, RISK.md ## High parsing, and recordedRisk reading the record back.
  • Classification in review(): a path match makes the change high, a base with no RISK.md makes it medium, and otherwise a Haiku turn sets it, falling back to Sol. The result is cached by merge base, head, and RISK.md text, and it runs only after the seat and --responses checks pass.
  • Reviewer raises: the reply schema now requires risk. settle() takes the riskiest tier, including the last round of a left-out seat on the same head.
  • Record and readers: adds the - Risk:, - Classified:, and - Raised the risk to lines; Status.risk; the classifying role; the tier column; and the By risk tier rollup.
  • Content-named schema files: this fixes the race where a different quest release could rewrite a shared schema-<n>.json mid-run.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread src/cli/review.ts Resolved
Comment thread src/cli/usage.ts Resolved

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. Both findings from the last review are fixed.

Reviewed changes

I reviewed what changed since the review at effcd06: one merge of main and one fix commit.

  • Fenced the classified reason: raising() now wraps risk.reason in <classified-reason> tags and tells the reviewer to treat it as data, the same way responseParts fences <responses>. The test assertions now match the new prompt.
  • Restored OnlineRow's doc comment: it sits on OnlineRow again, and TierRow keeps its own.
  • Planned the flaky test fix: the new quest quest/m0/review-test-partial-log.md reproduces the partial fake log in the rejected-login test and fixes it at its root.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit 4ab5dae Nov 14, 2026
20 checks passed
@kixelated
kixelated deleted the quest/m0/benchmark/risk-tier branch November 14, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant