Repository navigation
test(review): live check that a reviewed head runs nothing - #341
Merged
Merged
Conversation
Quest-Claim: ee600bad93cc8ad4
`just review-sandbox` runs `quest review` with the installed Codex and Claude Code CLIs on a head that plants Codex's project MCP servers and hooks, an `.mcp.json` server, and Claude Code's settings hooks. It fails if any of them runs or the review trusts a new directory. Each CLI first runs once without quest's flags, so a plant that stops firing fails as a broken control instead of passing as a clean review. Each run uses a new config home that trusts the clone (and, for Codex, the planted hooks) with a copy of the user's login, handing a refreshed login back afterwards. The default vitest run excludes `test/live`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main flattened MODELS to one tier per harness, which broke the live test's control after the merge. Write a refreshed login back through a rename so an interrupted write never truncates the user's credential file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hung reviewer A home that already trusts the clone cannot show a review trusting it, so each CLI also reviews from a home that trusts nothing and must leave it so. A review past 20 minutes has its reviewer CLI terminated and awaited, so a hung CLI never outlives its test or its config home. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
Collaborator
Author
|
Landing 2256f58.
(Written by Claude Opus 5.5) |
kixelated
enabled auto-merge (squash)
November 1, 2026 21:38
moq-bot Bot
pushed a commit
that referenced
this pull request
Nov 1, 2026
* quest: plan the 2026-11-01 follow-ups (wip) * quest: plan the 2026-11-01 follow-ups Fifteen follow-ups the maintainer settled in a /quest-plan interview: land work (fixer TLS, squash-mismatch comment, landing status), CLI and agent fixes, runner review checks, and two m1 quests. Also brings existing quests in line with #349, #276's close, validateFix, #246, and #341. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * quest: leave review-login to #339 and build the gateway quest on it #339 finishes and deletes review-login, including reviewing with credentials from the environment. Narrowing it here would conflict with that deletion, and the gateway quest would plan that work twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: moq-bot <moq-bot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finishes the live sandbox test quest, planned from the follow-ups of moq-dev/quest#271.
Summary
just review-sandboxrunsquest reviewwith the installed Codex and Claude Code CLIs on a head that plants configuration. It fails if any of the plants runs, or if the review writes a trust entry for a new directory into the reviewer's config home.test/live/review-sandbox.test.ts): a.codex/config.tomlMCP server,.codex/hooks.jsonhooks (SessionStart, UserPromptSubmit), an.mcp.jsonserver, and.claude/settings.jsonhooks withenableAllProjectMcpServers. Each one touches a marker file named after it, outside the checkout. Every review run asserts that no marker appeared, whichever CLI the plant targets.codex_hookstohooks, and it skips untrusted hooks.CODEX_HOMEorCLAUDE_CONFIG_DIR. It trusts the reviewed clone ([projects."<clone>"]/hasTrustDialogAccepted) and holds a copy of the user's login. The Codex home also trusts the planted hooks by their current hash. Without that, Codex skips them whatever quest does, and the hooks check would be empty. The hashes come fromcodex app-server'shooks/list, which needs no login.vitest.config.tsexcludestest/live/**unlessQUEST_LIVE=1, which only the recipe sets.just testand CI never run it. Arguments pass through to vitest, such asjust review-sandbox -t codex.m0link are deleted. Review harness check and Codex keeps a way to skip project config now namejust review-sandbox.No change to
src/. The test callsreview()with an explicit environment, so it doesn't overlap the parallel review login work insrc/cli/review.tsandsrc/cli/agent.ts.Validation
Merged
origin/main(no rebase). Main flattenedMODELSto one tier per harness, so the control now readsMODELS[harness].model.Run on 2026-11-01 with
codex-cli 0.165.0andclaude2.1.316, usingCLAUDE_CONFIG_DIR=~/.claude(this session's own config dir holds no login file):just review-sandbox: 6/6 pass (about 85s). For each CLI, the control recorded all of its plants. The trusted review and the untrusted review recorded nothing and wrote no trust entry. The Claude review leg, which had hit a weekly limit before, now runs through the findings turn.~/.claude/.credentials.jsonwas unchanged afterwards, since no refresh happened.codex execprocess remained.just checkpasses.just testpasses (762 passed, 2 skipped; 426 for each build's lifecycle) withCLAUDE_CONFIG_DIRandQUEST_WORKTREE_HOMEunset.quest check --base origin/mainpasses.Decisions
The maintainer picked the recommended option in a
/quest-completesession on 2026-11-01:$CODEX_HOME/~/.codexauth.jsonand$CLAUDE_CONFIG_DIR/~/.claude.credentials.json. After each run, if the copy changed and the source still matches what was copied, the refreshed file replaces the source. The replacement now goes through a rename, so an interrupted write never truncates the user's login. If another session refreshed the same login during the run, it warns instead. Linux only, since macOS keychain logins have no file to copy.Choices made without asking
hooks/listshowed that for a linked worktree, nested or sibling, Codex 0.165 reads project hooks from the main checkout's.codex/hooks.jsonand ignores the worktree's own. The worktree's.codex/config.tomlMCP server did still load in the first control run. So in a worktree the head's hooks never load. The clone checked out on the head, trusted directly, exposes all four plants under the strongest trust. feat!: review locally with the other vendor's model (quest review, /quest-review) #271's worktree-specific P1 rests on the same--ignore-user-configprotection checked here.QUEST_LIVE=1, rather than a second vitest config. Importing the shared Markdown plugin from a second config triggered Vite's native-loader warning.Review
gpt-6.1-sol, medium effort (quest review --base origin/main).08ced209ea0a7bf555b2484e8d9cb477b46f1839: 2 findings, both fixed.2256f58f2243b25c62bfd40ae1ab7a0b0edf75bf: 0 findings.Suggested follow-ups
.claude/settings.local.json,statusLine,apiKeyHelper,env, and permission rules also run or widen things when settings load.--setting-sources ""should cover all of them, and adding them to the plants is a few lines each.quest agentand land-fix sessions that run with user config in worktrees, not forquest review. Worth checking whether the fixer (src/cli/land-fix.ts, also--ignore-user-config) or headless agents rely on worktree-local hooks.(Written by Claude Opus 5.5)
🤖 Generated with Claude Code