Skip to content

test(review): live check that a reviewed head runs nothing - #341

Merged
moq-bot[bot] merged 6 commits into
mainfrom
quest/m0/review-sandbox
Nov 1, 2026
Merged

moq-bot[bot] merged 6 commits into
mainfrom
quest/m0/review-sandbox

Conversation

@kixelated

@kixelated kixelated commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes the live sandbox test quest, planned from the follow-ups of moq-dev/quest#271.

Summary

just review-sandbox runs quest review with the installed Codex and Claude Code CLIs on a head that plants configuration. It fails if any of the plants runs, or if the review writes a trust entry for a new directory into the reviewer's config home.

  • Plants (test/live/review-sandbox.test.ts): a .codex/config.toml MCP server, .codex/hooks.json hooks (SessionStart, UserPromptSubmit), an .mcp.json server, and .claude/settings.json hooks with enableAllProjectMcpServers. Each one touches a marker file named after it, outside the checkout. Every review run asserts that no marker appeared, whichever CLI the plant targets.
  • Control: each CLI first runs one plain turn in the same checkout without quest's flags, and must record all of its own plants. Without this, a plant that a CLI release stops loading (wrong format, renamed feature) would pass as a clean review. It already caught two cases while I built this: Codex 0.165 renamed codex_hooks to hooks, and it skips untrusted hooks.
  • Config homes: each run (control and review) gets a new CODEX_HOME or CLAUDE_CONFIG_DIR. It trusts the reviewed clone ([projects."<clone>"] / hasTrustDialogAccepted) and holds a copy of the user's login. The Codex home also trusts the planted hooks by their current hash. Without that, Codex skips them whatever quest does, and the hooks check would be empty. The hashes come from codex app-server's hooks/list, which needs no login.
  • Trust check: the set of trusted directories after the review must equal the seeded set. Since a home that already trusts the clone can't show a review trusting it, each CLI also reviews from a home that trusts nothing, and that home must still trust nothing afterwards. On Codex, openai/codex#49333 is the reason to watch this.
  • Bounded review: a review past 20 minutes has the reviewer CLI it spawned terminated and awaited before the checks run, so a hung CLI never outlives its test or its config home.
  • Opt-in: vitest.config.ts excludes test/live/** unless QUEST_LIVE=1, which only the recipe sets. just test and CI never run it. Arguments pass through to vitest, such as just review-sandbox -t codex.
  • Docs and quests: README's "Review locally" section says what the recipe checks and when to run it. The finished quest and its m0 link are deleted. Review harness check and Codex keeps a way to skip project config now name just review-sandbox.

No change to src/. The test calls review() with an explicit environment, so it doesn't overlap the parallel review login work in src/cli/review.ts and src/cli/agent.ts.

Validation

Merged origin/main (no rebase). Main flattened MODELS to one tier per harness, so the control now reads MODELS[harness].model.

Run on 2026-11-01 with codex-cli 0.165.0 and claude 2.1.316, using CLAUDE_CONFIG_DIR=~/.claude (this session's own config dir holds no login file):

  • just review-sandbox: 6/6 pass (about 85s). For each CLI, the control recorded all of its plants. The trusted review and the untrusted review recorded nothing and wrote no trust entry. The Claude review leg, which had hit a weekly limit before, now runs through the findings turn. ~/.claude/.credentials.json was unchanged afterwards, since no refresh happened.
  • Bound: with the bound temporarily set to 5s, both review tests failed with the bound's message after about 5s, and no codex exec process remained.
  • just check passes. just test passes (762 passed, 2 skipped; 426 for each build's lifecycle) with CLAUDE_CONFIG_DIR and QUEST_WORKTREE_HOME unset.
  • quest check --base origin/main passes.

Decisions

The maintainer picked the recommended option in a /quest-complete session on 2026-11-01:

  1. How the live test gets each CLI's login. Copying a login means a token refresh during the run happens in the copy, and an OAuth refresh can rotate the refresh token, which would leave the user's own home with a dead one.
    • ✅ A: copy the CLI's own login and hand a refreshed one back. The source is $CODEX_HOME/~/.codex auth.json and $CLAUDE_CONFIG_DIR/~/.claude .credentials.json. After each run, if the copy changed and the source still matches what was copied, the refreshed file replaces the source. The replacement now goes through a rename, so an interrupted write never truncates the user's login. If another session refreshed the same login during the run, it warns instead. Linux only, since macOS keychain logins have no file to copy.
    • B: dedicated logins for the test, in fixed homes the maintainer logs into once.
    • C: symlink the user's credential file into the disposable home.

Choices made without asking

  • The reviewed checkout is the trusted clone itself, not a linked worktree. Probing hooks/list showed that for a linked worktree, nested or sibling, Codex 0.165 reads project hooks from the main checkout's .codex/hooks.json and ignores the worktree's own. The worktree's .codex/config.toml MCP server did still load in the first control run. So in a worktree the head's hooks never load. The clone checked out on the head, trusted directly, exposes all four plants under the strongest trust. feat!: review locally with the other vendor's model (quest review, /quest-review) #271's worktree-specific P1 rests on the same --ignore-user-config protection checked here.
  • Low effort for every turn, with the default tier's models. Only the isolation is under test, and low effort keeps the usage down.
  • Marker files, not a local port. The planted MCP servers and hooks run as harness processes outside the reviewer's sandbox, which is where feat!: review locally with the other vendor's model (quest review, /quest-review) #271's P1 ran. A sandboxed write could miss something, but so could a request to a local port, since the sandbox has no network.
  • One config, switched by QUEST_LIVE=1, rather than a second vitest config. Importing the shared Markdown plugin from a second config triggered Vite's native-loader warning.
  • Bounds: the live suite gets a 30-minute test timeout, and the synchronous control spawn gets 10 minutes, since vitest's timeout can't interrupt it. Both only bound a hung CLI.

Review

  • Reviewer: Codex, gpt-6.1-sol, medium effort (quest review --base origin/main).
  • Round 1 on head 08ced209ea0a7bf555b2484e8d9cb477b46f1839: 2 findings, both fixed.
    • Medium: the trust check couldn't see a review trusting the clone, since every home already trusted it. Fixed with the untrusted-home review.
    • Medium: a hung reviewer outlived vitest's timeout and its config home. Fixed with the 20-minute bound, which terminates and awaits the reviewer CLI.
  • Round 2 on head 2256f58f2243b25c62bfd40ae1ab7a0b0edf75bf: 0 findings.

Suggested follow-ups

  • Cover macOS logins. Both CLIs can keep their login in the keychain, where there is no file to copy. The test could read the keychain item, or the dedicated-login option (B) would avoid copying entirely.
  • Plant more of Claude Code's project settings. .claude/settings.local.json, statusLine, apiKeyHelper, env, and permission rules also run or widen things when settings load. --setting-sources "" should cover all of them, and adding them to the plants is a few lines each.
  • Codex resolves a linked worktree's hooks from the main checkout. In a worktree, a trusted clone's main-checkout hooks, whatever branch it has checked out, apply to sessions in every worktree. That matters for quest agent and land-fix sessions that run with user config in worktrees, not for quest review. Worth checking whether the fixer (src/cli/land-fix.ts, also --ignore-user-config) or headless agents rely on worktree-local hooks.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 2 commits October 31, 2026 23:01
Quest-Claim: ee600bad93cc8ad4
`just review-sandbox` runs `quest review` with the installed Codex and
Claude Code CLIs on a head that plants Codex's project MCP servers and
hooks, an `.mcp.json` server, and Claude Code's settings hooks. It fails
if any of them runs or the review trusts a new directory. Each CLI first
runs once without quest's flags, so a plant that stops firing fails as a
broken control instead of passing as a clean review.

Each run uses a new config home that trusts the clone (and, for Codex,
the planted hooks) with a copy of the user's login, handing a refreshed
login back afterwards. The default vitest run excludes `test/live`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Live test that a reviewed head runs nothing test(review): live check that a reviewed head runs nothing Nov 1, 2026
kixelated and others added 3 commits November 1, 2026 13:20
Main flattened MODELS to one tier per harness, which broke the live test's
control after the merge. Write a refreshed login back through a rename so
an interrupted write never truncates the user's credential file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hung reviewer

A home that already trusts the clone cannot show a review trusting it, so
each CLI also reviews from a home that trusts nothing and must leave it so.
A review past 20 minutes has its reviewer CLI terminated and awaited, so a
hung CLI never outlives its test or its config home.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 1, 2026 21:32
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c7900bea-ea0c-4bf6-b65b-73c76f8051e0
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated

Copy link
Copy Markdown
Collaborator Author

Landing 2256f58.

  • Decision (maintainer, /quest-complete on 2026-11-01): A, copy the CLI's own login and hand a refreshed one back, already implemented. The hand-back now replaces the file by rename.
  • Merged origin/main (no rebase) and adapted the test to main's flattened MODELS.
  • Codex review round 1 found 2 issues, both fixed: an untrusted-home review now checks that a review writes no trust, and a 20-minute bound terminates a hung reviewer. Round 2 on this head found 0.
  • just review-sandbox passes 6/6 on codex-cli 0.165.0 and claude 2.1.316, including the full Claude review leg. just check, just test, CI, and quest check --base origin/main pass.

(Written by Claude Opus 5.5)

@kixelated
kixelated enabled auto-merge (squash) November 1, 2026 21:38
@moq-bot
moq-bot Bot merged commit c2325a3 into main Nov 1, 2026
@moq-bot
moq-bot Bot deleted the quest/m0/review-sandbox branch November 1, 2026 22:28
moq-bot Bot pushed a commit that referenced this pull request Nov 1, 2026
* quest: plan the 2026-11-01 follow-ups (wip)

* quest: plan the 2026-11-01 follow-ups

Fifteen follow-ups the maintainer settled in a /quest-plan interview:
land work (fixer TLS, squash-mismatch comment, landing status), CLI and
agent fixes, runner review checks, and two m1 quests. Also brings
existing quests in line with #349, #276's close, validateFix, #246, and
#341.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* quest: leave review-login to #339 and build the gateway quest on it

#339 finishes and deletes review-login, including reviewing
with credentials from the environment. Narrowing it here would conflict
with that deletion, and the gateway quest would plan that work twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: moq-bot <moq-bot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant