Skip to content

docs: add REVIEW.md for adversarial, approach-first reviews - #7126

Merged
kixelated merged 1 commit into
mainfrom
review-md
Oct 28, 2026
Merged

kixelated merged 1 commit into
mainfrom
review-md

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Problem

Codex cloud is being replaced by local reviews from the other vendor's model (kixelated/quest#267, adopted here by moq-dev/moq#7125). quest review feeds the base's REVIEW.md to the reviewer, and the repo had none.

I audited about two months of review history (6,760 Codex findings, 2,396 CodeRabbit findings) and six weeks of local session transcripts. The bots catch edge-case bugs well, but almost never question the approach. Every "wrong approach" call came from the maintainer, late. Examples: "layering problem" on moq-dev/moq#3663, "seems like a hack" on moq-dev/moq#5030, and "take another stab at the API" on moq-dev/moq#3644. Meanwhile, PRs like moq-dev/moq#3506 went through 94 edge-case findings in 55 commits before closing.

Approach

REVIEW.md is written for whichever model reviews (Codex, Claude, or hosted Claude Code Review), so it stays vendor-neutral and self-contained:

  • Context: read the root AGENTS.md and each nested AGENTS.md on the path to the changed files. Most tools only auto-load the ones above their working directory. The PR text is treated as claims to verify, and decisions recorded as the maintainer's are settled.
  • Stance: be adversarial and review the approach first. The checklist covers the premise, root cause (including whether the area's git log shows a line of fixes), a simpler alternative, layering, long-term API and wire shape, completeness, and evidence. It is distilled from the maintainer's own pushback and the release-audit questions.
  • Not findings: the classes most often refuted in review history, each already decided by AGENTS.md:
    • caps or timeouts proposed as bounds
    • duplicate-ID detection
    • shims for unshipped APIs
    • bare moq-transport "spec violations"
    • cross-node waits for rare races
    • unverified parity demands
    • defensive checks
    • implementation detail asked of quest plans
  • Evidence and later rounds: each finding cites path:line with a concrete failure scenario, and a suggested fix has to follow the rules. Reviews should converge: an author's refutation stands without new evidence, and a run of edge-case rounds means the reviewer should judge whether the approach is overbuilt.

AGENTS.md gets a one-line pointer to REVIEW.md. The opencode review agent now reads the base's REVIEW.md instead of keeping its own copy of the finding rules.

A trial run of Codex with this file on moq-dev/moq#5030 returned one P1 finding and no nits. The finding is a real draft-20+ regression, and its suggested fix is at the ownership level. It still didn't say outright "this is a hack", so this is one data point, not proof.

Impact

  • Public API: none.
  • Wire: none.

Alternatives

  • Move CONTRIBUTING's Reviews section here. I left it alone: it covers the author's side (replying, merge gating), and moq-dev/moq#7125 rewrites it once quest review lands.
  • Pull AGENTS.md in with @ includes. Hosted Code Review only expands those on a branch it counts as protected. main's ruleset (merge queue, required checks) has no update restriction or required approval, so it doesn't count. Telling the reviewer to read the files works everywhere.
  • Cap the number of nits. No: an arbitrary number. The file asks for few, defensible findings instead.

Follow-ups

None. If hosted Claude Code Review is ever enabled, it would ignore any exclusions here until main counts as protected.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 28, 2026 19:45
@coderabbitai

coderabbitai Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 34 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2b23f0c3-4066-4a48-924c-9fd294528fda
Commits

Reviewing files that changed from the base of the PR and between 3903360 and 6690926.

Files selected for processing: 3
  • .opencode/agent/review.md
  • AGENTS.md
  • REVIEW.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T19:46:44.483169Z 6690926 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 6690926

No actionable bugs found. Centralizing the approach-first review guidance in REVIEW.md and having the opencode reviewer consume the base copy is a coherent, simpler direction than maintaining separate finding rules. The changes are limited to review instructions, with no runtime API or wire changes.

Verification: static inspection of all three changed files and the surrounding AGENTS.md, CONTRIBUTING.md, and PROMPTING.md guidance. I did not run a reviewer trial or tests; improved review quality remains an empirical claim, not something this inspection verifies.

(Written by OpenAI)

@kixelated
kixelated added this pull request to the merge queue Oct 28, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Two notes from a trial of the planned quest review flow on this PR's head (6690926b15). Codex ran its built-in codex exec review --base origin/main, then a resumed follow-up turn applied this PR's REVIEW.md, then a second resumed round received an author response.

  1. Finding (P2): a base without REVIEW.md leaves the opencode reviewer with no rules. .opencode/agent/review.md now requires git show origin/<base>:REVIEW.md and drops its inline criteria. release has no REVIEW.md until a release cut carries it over, so a review of that cut finds nothing to follow. Suggest: "If the base has no REVIEW.md, review with the base's AGENTS.md and CONTRIBUTING.md alone; never the PR's copy." This matches how quest review plans to treat a missing file (kixelated/quest#267).

  2. "Later rounds" accepts refutations nobody can check. For round 2 I sent a deliberately unverifiable refutation: "the release cut is reviewed by the maintainer, not the opencode agent". Nothing in the repo shows when the opencode agent runs. The reviewer searched, found no counter-evidence, and dropped the finding, as "a refutation stands unless you bring new evidence" says to. Suggest holding the author to the same bar as a finding: "A refutation stands when it cites evidence (path:line, a command's output, or a recorded maintainer decision) that you cannot rebut; an uncited refutation is a claim to verify, and the finding stays until it is settled."

(Written by Claude Opus 5.5)

Merged via the queue into main with commit 0bd5cfd Oct 28, 2026
10 checks passed
@kixelated
kixelated deleted the review-md branch October 28, 2026 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant