Skip to content

ci: converge codex-review.yml onto the shared guarded_comment_upsert - #20332

Merged
bolichen97 merged 1 commit into
mainfrom
fix/codex-review-guarded-upsert-8462
Oct 14, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/codex-review-guarded-upsert-8462

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The GPT review lane posts its comment through the same guarded upsert every other review lane uses, so one spelling of the verdict-visibility invariant exists in the repository.

Bug / debt: codex-review.yml was the one review lane still carrying its own inline comment lookup, verdict guard and notice strip (the merged #8342 shape). That strip, sed -e '1,8{...}', is position-blind: a bare quoted <!-- codex-stale-notice-begin --> on lines 6-8 of preserved review prose deletes up to three lines of a standing verdict even with no end marker. It was also the one site left with a read-modify-write window on the comment slot, which the shared function's own comments explain is unsafe: a verdict published between the read and the merged write is restored away. Deferred from #8450 as issue #8462 and overdue since 2026-09-18.

Why it matters

Two maintained spellings of one invariant drift apart. Since #8450 the shared function gained head confirmation, a retried lookup and the publication record, none of which the GPT lane's inline copy had; it could still PATCH a verdict for a superseded head over the current one. A maintainer reading a GPT comment also has to know that this one lane behaves differently from the other eleven.

Not a goal

  • Changing the Gate on findings step or what blocks a merge. The gate is byte-identical.
  • Converging claude-review.yml. It has no notice path and the issue scopes it out; it keeps its inline arms and is still pinned by the same tests.
  • Changing the shared guarded_comment_upsert or retry_comment_write functions. Both stay byte-identical across every lane.
  • Changing the override handler, the refused-review classification or the withheld-verdict retention.

What changed (motivation → approach → change)

Motivation: one invariant, one spelling, and the known over-delete gone.

Approach: copy the function in, keep the lane fail-closed outside it. The shared function returns 0 on every arm because the advisory lanes must never invent a BLOCK from an infrastructure fault. The GPT lane is the opposite: its gate reads the captured review text, not the PR, so a verdict that missed the slot would pass the check while the PR shows an older head's verdict. The function records what reached the slot as the publication step output, so the lane reads that back after the call instead of changing the function.

Change, in .github/workflows/codex-review.yml, posting step only:

  • The inline lookup, the [GPT-REVIEWED] body guard, the preserve-and-prepend branch with its sed strip and the four exit arms are deleted.
  • guarded_comment_upsert() is defined byte-identical to the other eleven lanes and called once: with [GPT-OVERRIDE] when kind is override, else with [GPT-REVIEWED]. The head_needle variable is gone; the stamp is named at the call site, after kind is assigned.
  • One fail-closed tail: publication is read back from $GITHUB_OUTPUT; when the body carries a head-scoped stamp for $HEAD and publication is neither published nor superseded, the step annotates ::error:: and exits 1. A body with no stamp for this head (incomplete or refused) does not exit here: the gate already fails it closed on its own evidence.

Behaviour that changes for a reader of the comment:

  • An incomplete or refused run with a standing verdict in the slot now leaves that comment untouched and emits a ::warning::, like every other lane. It no longer prepends a dated staleness notice to the comment. The gate's red check and its message carry the incomplete/refused state, as before.
  • A completed verdict is now published only after the PR's head is confirmed still current; a superseded head is reported superseded and stays green, as status 4 did before.
  • A write that stopped at an accepted human override (status 5 inside the retry) now exits 1 instead of staying green. The shared function records it as not_published, the same record the fork GPT finalize fails on, and the override handler cancels and re-runs the in-flight lane with the human decision, so the green comes from that re-run.

test/test_ai_review_workflows.py: codex-review.yml joins _GUARDED_LANES (so the identity test and every behavioural case now run against its real step bash) with a per-lane missed_slot_rc that the completed-but-missed cases assert through _missed_slot_rc(lane). TestGptVerdictVisibility now pins the two call sites, their order after kind="override", the absence of the inline shape, and the untouched gate; test_only_the_fail_closed_lanes_exit_on_a_verdict_that_missed_the_slot pins the one tail arm. docs/ci/ci-and-reviews.md describes the converged shape.

Backwards compatibility

Compatible: no input, output name, marker, stamp or check name changes; the posting step's exit stays non-zero on every outcome it was non-zero on before, and additionally on a withheld-unreadable-head and an override-stopped repeat.

Reader: .github/workflows/codex-review.yml:Gate on findings -- release -- reads codex-review-output.md, untouched; test_the_gate_is_untouched
Reader: .github/workflows/codex-review.yml:Retain this run's unpublished verdict -- release -- reads steps.post.outputs.retained_verdict_artifact, still written by the shared retain_unpublished_verdict; test_withheld_verdict_retention.py
Reader: src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-prepare-pr/scripts/pr_status.py -- chat -- matches [GPT-REVIEWED] <head> in the comment, which the body still carries; test_prepare_pr_status.py
Reader: .github/workflows/ai-review-human-override.yml -- release -- re-runs this lane and reads nothing from the deleted notice markers; test_ai_review_workflows.py::TestHumanOverrideHandler
Reader: docs/ci/ci-and-reviews.md -- release -- the section naming codex-review.yml as the excluded lane is rewritten in this PR

Tests

  • TestReviewLaneVerdictVisibility[codex] and TestTheGuardedUpsertRecordsWhatReachedTheSlot[codex]: every shared contract case now runs against the GPT lane's real step bash with the stubbed gh: incomplete never overwrites, failed lookup posts nothing, superseded leaves the slot alone, unreadable head withholds, completed replaces, lost write retried and never claimed, duplicate ruled out before create, override repeat stopped. The fail-closed cases assert exit 1 for this lane and 0 for the advisory lanes.
  • test_guard_function_is_byte_identical_across_all_lanes: now eleven lanes.
  • TestForkLanePublishesTheOverrideNote[codex]: the override note replaces a standing block for this head through the guarded slot.
  • TestGptVerdictVisibility: structural pin of both call sites, their order, the removed inline shape and the untouched gate.
  • test_only_the_fail_closed_lanes_exit_on_a_verdict_that_missed_the_slot: Opus keeps four inline arms; GPT has exactly one tail arm gated on a head-scoped stamp, and the shared function carries no exit in any lane.

Determinism proof (testing-conventions, Proving a determinism fix), on the 38 new or changed node ids ([codex] cases, TestGptVerdictVisibility, the identity, call-site and fail-closed structural tests, the refusal pin):

  • Repeats: for i in $(seq 20); do python -m pytest -n0 -p no:cacheprovider -q @ids.txt; done -- 38 passed, 20/20 green. Then 10 runs at -n 4 --dist loadgroup --max-worker-restart=2 with test/test_withheld_verdict_retention.py test/test_review_slot_read_error.py as co-tenants -- 10/10 green.
  • Shuffled order: seed=9272, python -m pytest -n0 -p no:cacheprovider -q @shuffled.txt -- 38 passed.
  • Whole files: test/test_ai_review_workflows.py test/test_withheld_verdict_retention.py test/test_review_slot_read_error.py test/test_ai_review_workflows_fork_verdict.py at -n 4 --dist worksteal, 3 runs -- 1643 passed each.
  • scripts/local-gate.py backend related set: 2037 passed. The frontend related set could not run on this host (npm registry 401); CI runs it. No frontend file changes.

Not a flaky-test fix, so there is no forced condition or negative control to show.

Manual verification

bash -n on the extracted posting step passes. N/A beyond that: the step is driven end to end by the stubbed-gh harness above; the live path is a review comment on this PR itself, which the GPT lane posts on this head.

Related Issues

Closes #8462

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 14, 2026 14:16
@bolichen97
bolichen97 requested a review from mbriones98 October 14, 2026 14:16
@bolichen97
bolichen97 enabled auto-merge (squash) October 14, 2026 14:16
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision e9721550238e8d15a866ee85ba7d0c7cf8787bd5 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of e9721550238e8d15a866ee85ba7d0c7cf8787bd5 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check: when two runs race on one head, the second gets rc 3 ("already in this lane's slot"), lands as not_published and fails red while the slot is correct.

[DESIGN-REVIEWED] e972155

Description read

[DESCRIPTION-READ] ff9803824a6fce06a1a1964d7141c415b2973e099636e2406edc37cbf4866bff

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of e9721550238e8d15a866ee85ba7d0c7cf8787bd5 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

One thing to check before merge: a repeat stopped at an accepted override now turns the step red. It only goes green again if the override handler cancels and re-runs that same-repo GPT run.

What this change ships

Inventory (7 items) — 7 justified

Intent: one guarded publication path for the GPT review comment, which also removes the known over-delete in its notice strip (#8462). FIX.

  1. The GPT lane's comment is now posted through the shared guarded_comment_upsert. The inline lookup, guard and sed strip are deleted. — justified
  2. An incomplete or refused run no longer adds a dated staleness notice above the standing verdict. It leaves the comment untouched and emits a warning. — justified
  3. A completed verdict is published only after the PR's head is confirmed current. — justified
  4. An unreadable PR head now holds back a stamped verdict and fails the step. — justified
  5. A repeat that stops at an accepted override now fails red instead of passing green. — justified
  6. ci-and-reviews.md no longer lists codex-review.yml as an excluded lane. — justified
  7. codex joins the shared guarded-lane tests, and the inline-shape tests are deleted. — justified

[FIRST-PRINCIPLES-REVIEWED] e972155

Description read

[DESCRIPTION-READ] fc4a981b38254008fb88208b07278e8252d29f852a188990fa9a5b28d25442df

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e9721550238e8d15a866ee85ba7d0c7cf8787bd5 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e972155

Verdict parsed from the review's SHA-scoped output markers for commit e9721550238e8d15a866ee85ba7d0c7cf8787bd5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e9721550238e8d15a866ee85ba7d0c7cf8787bd5: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of e9721550238e8d15a866ee85ba7d0c7cf8787bd5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e972155

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e9721550238e8d15a866ee85ba7d0c7cf8787bd5: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the fix/codex-review-guarded-upsert-8462 branch from 43651de to eb2e664 Compare October 14, 2026 14:31
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 14, 2026
The same-repo GPT lane's posting step carried its own inline comment
lookup, verdict guard and a position-blind `sed -e '1,8{...}'` notice
strip, under which a bare quoted begin marker on lines 6-8 of preserved
review prose deleted up to three lines of a verdict. Every other review
lane publishes through one byte-identical `guarded_comment_upsert()`
function. The GPT lane now defines the same function and calls it with
`[GPT-REVIEWED]`, or `[GPT-OVERRIDE]` for an accepted human override;
an incomplete or refused run leaves a standing verdict untouched instead
of merging a staleness notice back over it. The lane stays fail-closed:
after the function returns, the step reads its `publication` output back
and exits non-zero when a body stamped for this head reached neither
`published` nor `superseded`. The `Gate on findings` step is untouched.

The lane joins the parametrized guarded-lane suite (identity test
included) with a per-lane `missed_slot_rc`; the tests that pinned the
inline shape are replaced by a structural pin of the call sites and the
untouched gate. docs/ci/ci-and-reviews.md describes the new shape.

Closes #8462
@bolichen97
bolichen97 force-pushed the fix/codex-review-guarded-upsert-8462 branch from eb2e664 to e972155 Compare October 14, 2026 14:41
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Two runs racing on one head: the second gets rc 3, lands as not_published and fails red while the slot is correct — rebutted (this is the GPT lane's standing rule, not a new loss)

The GPT lane is fail-closed and already exits 1 on rc 3 on main (codex-review.yml, create arm: "this lane's slot already holds one for $HEAD from an earlier run ... Re-run this lane"); test_only_the_fail_closed_lanes_exit_on_a_verdict_that_missed_the_slot pins that arm on main. The slot holding the EARLIER sample is correct for the stamp readers but not for the re-run that asked for a fresh sample: the earlier verdict keeps whatever it said, blocking line included, so a green here would let a stale block gate the PR silently.
The advisory lanes stay green on rc 3 because readiness attributes their failure as a judged BLOCK; the GPT lane's failure is attributed plainly and a writer clears a wrong one with a head-scoped override. The same race on main's inline shape had the same red outcome; nothing changes for it here. Pinned for this head by test_an_existing_current_head_comment_is_never_duplicated[codex] (rc 1) and test_an_earlier_runs_sample_for_this_head_is_not_this_runs[codex].

[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • A repeat stopped at an accepted override now turns the step red; it only goes green again if the override handler cancels and re-runs the same-repo GPT run — rebutted (confirmed by reading ai-review-human-override.yml)

Confirmed: ai-review-human-override.yml step "Re-run line reviewers with the human decision" runs for target=gpt|all; rerun_reviewer "codex-review.yml" ... (line 432) resolves the same-repo run for the current head and PR, and rerun_run POSTs actions/runs/<id>/cancel on an in-progress run before actions/runs/<id>/rerun. The re-run's human_override step reads the bot-authored marker for that head, so its post step publishes the override note through the guard with [GPT-OVERRIDE] <head> and the gate reads HUMAN_OVERRIDE=true.
The red is confined to the race the finding names: a verdict for this head whose first PATCH failed and whose retry found the override landed in between. That run is the one the handler cancels; a run that completed BEFORE the override was posted never reaches this arm. This mirrors the fork GPT lane, whose finalize already fails not_published on this same arm and passes an override for this head only through its own override input (fork-gpt-review.yml, publication record).

[operator: bolichen97]

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 14, 2026

@mbriones98 mbriones98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks green and PR Readiness passed at e972155; non-feature change.

@bolichen97
bolichen97 merged commit eea7f0d into main Oct 14, 2026
102 checks passed
@bolichen97
bolichen97 deleted the fix/codex-review-guarded-upsert-8462 branch October 14, 2026 16:41
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: converge codex-review.yml onto the shared guarded_comment_upsert shape

2 participants