Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 83 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,9 @@ binary carries, and reports a missing, outdated, or retired one. A job that
checks a PR with a binary built from the base passes `--no-stubs`, since that
binary's skills would reject a PR that changes a stub; the PR's own binary
checks them in its own `quest check`. This repository's merge queue and
`quest merge` do so.
`quest merge` do so. It also reads the repository's [`RISK.md`](#risk-tiers),
if any, and reports each `## High` item it cannot read, which would otherwise
fail the next `quest review` on a branch from it.

A link into a submodule that is not checked out, as in a fresh worktree or
`actions/checkout` by default, resolves at the commit its gitlink pins when a
Expand Down Expand Up @@ -482,6 +484,7 @@ mergeable conflicting
merge-state dirty
review-decision -
review-record diff-identical 5f1c2e9a8b7d6c5e4f3a2b1c0d9e8f7a6b5c4d3e
risk medium diff-identical low It changes when quest gate admits a command.
auto-merge -
merge-queue -
landing-stopped -
Expand All @@ -500,6 +503,7 @@ comes last, and a missing value prints as `-`:
| `url`, `state`, `draft`, `head` | The PR URL; `open`, `closed`, or `merged`; `true` or `false`; the head SHA. |
| `mergeable`, `merge-state`, `review-decision` | GitHub's values, lowercased. |
| `review-record` | Whether the PR body's [`## Review` record](#review-locally) still covers the head, and the head it names last: `current` when it is the head, `diff-identical` when `git diff <base>...<head>` prints the same as for it, `stale` otherwise, or `unknown` when GitHub cannot compare them, such as a diff too large; `-` when the record names no full SHA. Whose record counts is the reader's call. |
| `risk` | The record's [risk tier](#risk-tiers), `low`, `medium`, or `high`; the same state as `review-record` for the head it names; the tier its classifier set before any reviewer raised it; and the reason. `-` when the record names none. |
| `auto-merge` | Method, who enabled it, when, and the head it was enabled on. |
| `merge-queue` | Queue state, position, and when it was enqueued. |
| `landing-stopped` | The reason the latest merge-queue entry, auto-merge, or [`quest land`](#the-watchers-own-queue) queue entry ended (such as `merge_conflict`, `failed_checks`, `manual`, `manually_disabled`, or `merged`), who ended it, and when; `-` once the PR has entered the merge queue or auto-merge again since. |
Expand Down Expand Up @@ -1423,22 +1427,29 @@ root and in each directory holding a changed path, never the checkout's, so a
change cannot instruct its own reviewer. The base's `REVIEW.md` is optional,
given at the start of each session, and chooses the review's stance, such as adversarial.

It prints the size and its source, base, and head, then for each reviewer its
It prints the size and its source, base, and head; the head's `risk` tier and
reason, then `classified` with the classifier's tier, how it was set (`model`,
`paths`, or `default`), and reason, and for a model a `classifier` line with its
harness, model, effort, and session, and its `usage` line; then for each reviewer its
harness, model, and effort, session, round,
an `unchanged` line for a repeated round, a `usage` line per turn (input tokens, the part read from the prompt cache, and output), the
an `unchanged` line for a repeated round, a `usage` line per turn (input tokens, the part read from the prompt cache, and output),
the score, a `raised` line with the tier and reason when it raised the classifier's, the
finding count, then each finding as its number, severity, `path:line`, and title over its
indented body, then `earlier` with the last round's head and its findings, each
as its number, disposition, severity, `path:line`, and title over its reason; last
come a `record` line and a `next` line. `--json` prints one object with
`rounds`, one per reviewer, `tallies`, every reviewer's findings across the
branch's rounds, `record`, and `next`.
branch's rounds, `risk`, the head's tier with its classification, `record`,
and `next`.

After each round, `quest review` rewrites the `## Review` section of the open PR
from the branch into the base, found among the upstream's PRs from the
repository the base's remote pushes to, and prints `record <url>`. The record
names the reviewed head and the base's short SHA, then for each reviewer its
names the reviewed head and the base's short SHA, the head's risk tier and who
classified it, then for each reviewer its
harness, model, and effort, its Reviewer line ending in `, shadow` for the
shadow, the session and round, the score and rationale, its tally, each finding with
shadow, the session and round, the score and rationale, the tier it raised the
risk to and why, its tally, each finding with
its location, and the last round's findings with their dispositions. The
tally, as `- Rounds: 3; findings 4 (high 1, medium 3): fixed 3, refuted 1, open 0`,
counts the reviewer's rounds on the branch and each finding it raised once,
Expand Down Expand Up @@ -1472,6 +1483,59 @@ and Codex, sent to `OPENAI_BASE_URL`, gives up on the gateway's 429 after its
own retries, which also counts as the limit there.
`/quest-review` runs the rounds and marks the PR ready.

### Risk tiers

Each head also gets a risk tier, `low`, `medium`, or `high`: what could break,
for whom, and how hard it is to undo, set from the change and never from who
wrote it. Nothing acts on the tier yet; `quest status` and `quest usage` report
it.

A change is `high`, with no model, when a changed path on either side of a
rename matches a pattern held high: `RISK.md` itself, everything under
`.github/workflows/` and `.github/actions/`, and each pattern in the `## High`
list of the base's `RISK.md`, at the repository's root:

```markdown
# Risk

Customers run the relay and the player; the CLI is for contributors.

## High

- `src/billing/` - charges customers, and refunds are manual
- `**/*.proto` - peers on older releases read the wire format
```

Each item is a backticked path from the root, where `*` and `?` stay within one
segment, a whole `**` segment spans any number, and a matched directory covers
everything under it, then ` - ` and why changing it is high. The rest of the
file is prose for the classifier, such as what is customer-facing. `quest
review` reads the file from the base's tip, so a change cannot lower its own
tier, and fails on one it cannot read, which the repository's own `quest check`
reports first. A base with no `RISK.md` makes every change no path holds high
`medium`.

Otherwise a classifier reads the diff and the base's `RISK.md` in one read-only
turn and replies with a tier and one sentence on why: Claude Code
`claude-haiku-5-5` at low effort, or Codex `gpt-6.1-sol` at low when Claude
Code is missing, logged out, has its login rejected, or hit its usage limit.
When neither can run, the review fails on the limit that resets first if both
hit theirs, and otherwise as a usage error naming why each failed. A diff past 200,000 characters gives it only the
stat, and it reads the parts it needs. The tier is kept with the reviewers'
sessions, so a later round on the same head and base `RISK.md` reuses it, and
`--new` classifies again. Each reviewer judges the tier too, told the
classified one, and can only raise it: the head's tier is the riskiest, with
the reason of the first to judge it so. The record names it after the base:

```markdown
- Risk: high, `It changes the record format quest status parses.`
- Classified: medium by Claude Code, claude-haiku-5-5, low effort, session `5b0e6c1a-27d4-4f8e-9a51-3c2d7e8f9b10`
```

`Classified:` names `the always-high paths` or `default` when no model set the
tier, and a reviewer that raised it adds `- Raised the risk to high: <reason>`
to its block.

## Audit cost and accuracy

`quest usage` reports `origin`'s PRs updated since `--since` (a local
Expand All @@ -1483,14 +1547,16 @@ worked on each PR, in which role, at what token cost, and how the work went.
$ quest usage --pr 569
moq-dev/quest, PRs updated and sessions active since 2026-11-06; costs (~$) estimate API list prices

#569 S merged: quest: finish model routing (quest/m0/model-routing/README)
#569 S medium merged: quest: finish model routing (quest/m0/model-routing/README)
risk: medium, It changes which model reviews each PR size.
implementer: Claude Code, claude-opus-5-5, unknown effort
review: Codex, gpt-6.1-sol, medium effort; round 1, score 10/10; rounds 1, findings 0: fixed 0, refuted 0, open 0
online review: review (OpenAI), not at the final head; 0 findings: 0 accepted, 0 declined, 0 unclassified, 0 unanswered
land fixes: codex, gpt-6.1-sol, medium effort
sessions:
implementing local claude claude-opus-5-5 low in 50, cache 36.2k written 1.1M read, out 6.1k, ~$0.63
reviewing local codex gpt-6.1-sol medium in 14.7k, cache 0 written 39.8k read, out 421, ~$0.04
implementing local claude claude-opus-5-5 low in 50, cache 36.2k written 1.1M read, out 6.1k, ~$0.63
reviewing local codex gpt-6.1-sol medium in 14.7k, cache 0 written 39.8k read, out 421, ~$0.04
classifying local claude claude-haiku-5-5 low in 9.2k, cache 0 written 0 read, out 38, ~$0.00
```

[token-max](https://github.com/kixel-ai/token-max)'s `sessions` reads and
Expand All @@ -1505,10 +1571,12 @@ session must also be this repository's, as the repositories token-max resolved
for it on its own host name, or, once its checkout is deleted, those resolved
there for another session in that exact directory, as a Codex log records
(never a parent's, which a nested clone may not share); otherwise it stays
unattributed. A session is reviewing when the PR's `## Review` record names
it or Codex marks it a review, implementing when its first request since
unattributed. A session is classifying when the PR's `## Review` record names
it as the [risk classifier](#risk-tiers), reviewing when the record names it as
a reviewer or Codex marks it a review, implementing when its first request since
`--since` came before the PR opened, and iterating otherwise, so a session
that opened a PR before `--since` and kept working on it reads as iterating. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer the
that opened a PR before `--since` and kept working on it reads as iterating. Each PR shows its `## Models` size, its risk tier and
whether a reviewer raised it, and its implementers, and as `unread:` each Implementer line it cannot read; each reviewer the
`quest review` record names, with its latest round and score and its tally of
findings across every round, marked `shadow` for the shadow, where a merged PR whose last round left only low
findings open counts them settled, since `/quest-review` fixes or refutes those
Expand Down Expand Up @@ -1540,7 +1608,9 @@ findings measure an effort step rather than the writer. Since a branch's tallies
PR whose record holds its latest tally, by the most rounds and then the newest,
counts them, and a record written before tallies
counts as unreviewed. An `Implementer:` line's own size, as
`(iterate, XS)`, counts over the PR's; and, over the merged PRs, each online
`(iterate, XS)`, counts over the PR's; each risk tier's PRs, how many merged,
how many a reviewer raised, and how many a later PR's `## Caused by` names,
which shows whether the tiers predict what breaks; and, over the merged PRs, each online
reviewer's coverage, final heads reviewed, findings per PR reviewed, and
answers. Sessions on no PR here, from any
repository or agent token-max reads, total on an `Unattributed` line. `--pr` and
Expand Down
35 changes: 35 additions & 0 deletions RISK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Risk

`quest review` reads this file from the base to set each PR's risk tier, as
the README's Risk tiers section describes.

`quest` runs development in moq-dev's and Kixel's repositories. A merge to
`main` that changes what `scripts/release-plan.sh` ships publishes a
release, which contributors and the maintainer's hosts install. A revert
publishes another, but what agents did under the bad one stays done.

What others rely on here:

- The rules and skills under `assets/`, which every agent session in those
repositories loads. A change to when an agent may push, merge, post, or
spend without asking is high.
- The `quest/core` and `quest/github` exports, which Kixel Code imports.
A change to what they export or return is at least medium, and high when
it changes how a record or status is read.
- The records `quest` writes into PR bodies and the output of
`quest status`, `quest usage`, and `quest unblocked`, which skills,
Kixel Code, and scripts parse.
- The reviewer's sandbox and the credentials `quest` passes to the agents
it starts, which keep a change from instructing its own reviewer or
reaching a secret.

Tests, plans under `quest/`, and documentation change nothing others rely
on.

## High

- `assets/CONTEXT.md` - every agent session in every repository using
Quest loads these rules, including when it may write remotely or merge
- `scripts/installer.sh` - users pipe it to a shell, and a release
publishes it as it is
- `scripts/release-plan.sh` - it decides which merges publish a release
18 changes: 11 additions & 7 deletions quest/m0/benchmark/guardrails.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,20 +58,24 @@ Agent's choices, for the implementer to revisit:
- A widening is proposed with its class's numbers: PRs merged, culprits
among them, and what a tap cost in waiting time.
- Tell the maintainer when a tier rises on a later head of a PR they
already picked; the pick ends there, as the standing pick's does.
already picked; the pick ends there, as the standing pick's does. A
tier that rises reopens whatever the lower tier skipped.
- The maintainer can override a tier, recorded with its reason, and the
report counts how often they do as a reading of the tier's accuracy.
[#716](https://github.com/moq-dev/quest/pull/716), which set the tier,
left this here, since nothing reads an override until the tier gates
something.

Open question for the maintainer, asked before the list is written: the
standing pick covers CI changes today, but the post requires a person for
"any CI workflow file", and
[the risk tier](/quest/m0/benchmark/risk-tier.md) makes workflow files
always high. Recommended: keep them high, which takes CI changes out of
"any CI workflow file", and `quest review` holds workflow files always
high. Recommended: keep them high, which takes CI changes out of
automatic merging.

## Required

- [Give every PR a risk tier from its diff](/quest/m0/benchmark/risk-tier.md) -
the tier and the repository's risk file, with a week of tiers the
maintainer has read
- [Read a week of risk tiers before guardrails act on them](/quest/m0/benchmark/risk-tier.md) -
the maintainer's verdict that the recorded tiers can be trusted

## Related

Expand Down
62 changes: 23 additions & 39 deletions quest/m0/benchmark/risk-tier.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,37 @@
# [L] Give every PR a risk tier from its diff
# [XS] Read a week of risk tiers before guardrails act on them

## Goal

Every PR carries a risk tier, low, medium, or high, with a one-line
reason: what could break, for whom, and how hard it is to undo. `quest`
sets it from the diff when the PR is reviewed and again for each later
head, and the tier never depends on who wrote the change. A file checked
into each repository lists the paths that are always high, such as
billing, deploys, workflow files, and a wire protocol, and may describe
what else is customer-facing there. The tier and reason show in the PR
body beside its size, in `quest status`, and in the PR's records.

Until [tier guardrails](/quest/m0/benchmark/guardrails.md) lands, the tier
changes nothing: it is only recorded, so its accuracy can be read before
anything trusts it.
The maintainer has read at least a week of the risk tiers `quest review`
recorded, against what merged and what later broke, and either trusts them
for [tier guardrails](/quest/m0/benchmark/guardrails.md) or has fixed what
set them wrong: a path in a repository's `RISK.md`, its prose on what is
customer-facing, or the classifier's instructions.

## Plan

Decided 2026-11-13 (user, /quest-plan): the tier comes from the diff,
because "effort is for writing the code, and risk is for merging it".
Size keeps routing the writer, so a hard problem still starts on the
strongest model by its size. Rejected: a tier set at planning and
re-judged from the diff; separate scores for customer impact,
reversibility, and breadth.

Agent's choices, for the implementer to revisit:

- Classify before the review starts, since the tier picks the review's
depth: a cheap model call over the diff and the repository's file, with
a path match that needs no model for the always-high list. The reviewer
may raise the tier with a reason and never lower it.
- A later head is classified again, and a tier that rises reopens
whatever the lower tier skipped.
- The repository's file is itself always high, as is any workflow file,
following the post's rule that "some paths always need a person".
- Read accuracy two ways once [breakage](/quest/m0/benchmark/breakage.md)
exists: how often a low-tier PR is later a culprit, and how often the
maintainer overrides a tier. An override is recorded with its reason.
- Fail loud on a repository with no file: every PR there is medium, and
`quest` says why.
`quest review` sets each head's tier since
[#716](https://github.com/moq-dev/quest/pull/716) and records it in the PR
body's `## Review` section. `quest usage --since <date>` shows each PR's
tier and reason, whether a reviewer raised it, and a rollup per tier of
PRs, merges, reviewer raises, and later fixes (`## Caused by`). A tier
reviewers often raise, or a low PR a later one fixed, points at what to
change. A fix lands in this quest's PR; the reading starts over only if
the fix changes most tiers.

A repository with no `RISK.md` records every change medium, so the reading
covers only the repositories that have one.

## Waiting

- Maintainer: a week of recorded tiers read against what merged, before
[tier guardrails](/quest/m0/benchmark/guardrails.md) acts on them; no
agent can judge whether the tiers match their own sense of risk. Check
with the tier column in `quest usage`.
- Maintainer: read a week of recorded tiers once a release with #716 runs
on the hosts, and say whether they match your sense of risk; no agent
can judge that. Check with the tier column and the By risk tier rollup
of `quest usage --since <date>`.

## Related

- [Development as a benchmark](/quest/m0/benchmark/README.md) - the
decisions this builds on
- [Breakage](/quest/m0/benchmark/breakage.md) - records what each tier's
PRs later broke
22 changes: 22 additions & 0 deletions quest/m0/review-test-partial-log.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# [XS] Fix the flaky rejected-login review test at its root

## Goal

`test/review.test.ts`'s "a login the API rejects fails as logged out,
whatever the status check said" passes reliably, including its
`(host CLAUDECODE, rejected in turn)` case, with the root cause fixed
rather than its assertions loosened.

## Plan

[#706](https://github.com/moq-dev/quest/pull/706) loosened the test's
status-check count assertion, yet it failed once more in
[#716](https://github.com/moq-dev/quest/pull/716)'s validation on a cut-off
line in the fake CLI's log, apparently a fake killed mid-append. Reproduce
the failure first, such as by running the case in a loop under load, then
fix the root: for example, make the fake write each log line atomically,
or have the test wait for the fake to exit before reading its log.
Restore any assertion #706 loosened if the root fix makes it hold again.

The fixing PR names #706 under `## Caused by` in its body, once the
implementer confirms #706's fix missed the root cause.
Loading
Loading