Repository navigation
fix(prepare-pr): a third question, a retrospective instead of a stall, and rounds read from the PR - #8840
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound memory-from-the-thread design, but it quietly rewrites security-finding policy — a maintainer ruling the author themselves says is still open. Watch
Suggestions
[DESIGN-REVIEWED] 773ce8f |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsThe candidate's own (a) — a concrete input occurring in practice — is unconfirmed and structurally so: No new grounded findings surfaced in the diff; the changes are additive and covered by their tests. No findings. [OPUS-REVIEWED] 773ce8f Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/pr_status.py:1065 -- Standard False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All server-side facts the skill's new guidance leans on are verified real: the ledger jq filter at First-Principles-Verdict: CONCERNS The local "never applies to Critical/High" limits are deleted while the description itself calls that an open maintainer policy question. What this change shipsIntent: stop the prepare-pr loop from patching its own round-grown code for twenty rounds — a FIX (of a measured, four-PR defect) delivered as process machinery.
Watch
[FIRST-PRINCIPLES-REVIEWED] 773ce8f |
124a285 to
ee48375
Compare
ee48375 to
b80b598
Compare
b80b598 to
28c10e6
Compare
|
rebutted — span=67f250492cae
Override, if the fence still holds on the new head: |
|
fixed — span=677a9f6493d3
|
|
fixed — span=677a9f6493d3
|
|
rebutted — span=682fb57e2fa4
The adjudicator reached the same conclusion and FLAGged it; its own rationale is the override: |
|
/ai-review override gpt 28c10e6: Requires two concurrent prepare-pr loops on the same PR branch (contradicting the single-owner-per-PR design) colliding in a once-per-round sub-millisecond read-modify-write window, and the only casualty is one entry in a deliberately non-gating advisory log (line 25), never code, the PR, or user data. |
Human judgment recorded@iamwhatever marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
d2c7ded to
3f3d3be
Compare
|
needs-a-decision — Watch: security-class findings can now be ruled disproportional with a pre-drafted override
|
|
fixed — Suggestion: make the retrospective trigger an exit code, not prose
|
|
rebutted — Watch:
|
|
rebutted — Subtraction: drop
|
|
fixed — span=682fb57e2fa4
|
|
fixed — span=682fb57e2fa4
|
|
fixed — span=682fb57e2fa4
|
|
fixed — span=677a9f6493d3
|
|
fixed — span=677a9f6493d3
|
GPT 5.6 Review —
|
…, and rounds read from the PR The loop had no memory between rounds, so a finding in code the PR added two rounds ago read as a fresh bug and got a fresh patch. Four PRs grew 2-6x under review that way. - pr_findings.py --rounds: the loop's cross-round memory, rebuilt from the writer dispositions already on the PR (one round per judged head, spans, self-added count, declared mechanisms, recurrence, growth). Exit 30 when a retrospective is due. Nothing is stored locally; the PR thread is the record. - SKILL.md: third question per finding (repair vs remove, both written down); security hard limits removed; every-3rd-round retrospective replaces "escalate at 3 stalled rounds"; whole-design lanes (Design, First Principles, UX) triaged before line-level lanes; dispositions written for the reviewer's ledger (`> ` lines, class-level); max_cycles=80 is a cap the loop does not raise. - pr_status.py binds the First Principles lane and prints each whole-design lane's verdict. - test_prepare_pr_ledger_contract.py pins the server-side facts the disposition guidance relies on.
3f3d3be to
773ce8f
Compare
|
Intent: Give the prepare-pr loop memory across rounds, so a finding in code the loop itself added is answered by weighing repair against removal, a retrospective every 3rd round subtracts what the intent does not need, and dispositions are written in the shape the remote reviewer's ledger actually reads. |
|
fixed — span=682fb57e2fa4
|
bolichen97
left a comment
There was a problem hiding this comment.
Approved. Both prior blockers are resolved on this head: the ~1k-line decision-gate trigger is restored (and clarified to measure the whole-PR diff), and Design Review plus First Principles Review now each carry per-finding disposition comments. PR Readiness is green.
…, and rounds read from the PR (kirodotdev#8840) The loop had no memory between rounds, so a finding in code the PR added two rounds ago read as a fresh bug and got a fresh patch. Four PRs grew 2-6x under review that way. - pr_findings.py --rounds: the loop's cross-round memory, rebuilt from the writer dispositions already on the PR (one round per judged head, spans, self-added count, declared mechanisms, recurrence, growth). Exit 30 when a retrospective is due. Nothing is stored locally; the PR thread is the record. - SKILL.md: third question per finding (repair vs remove, both written down); security hard limits removed; every-3rd-round retrospective replaces "escalate at 3 stalled rounds"; whole-design lanes (Design, First Principles, UX) triaged before line-level lanes; dispositions written for the reviewer's ledger (`> ` lines, class-level); max_cycles=80 is a cap the loop does not raise. - pr_status.py binds the First Principles lane and prints each whole-design lane's verdict. - test_prepare_pr_ledger_contract.py pins the server-side facts the disposition guidance relies on. Co-authored-by: Joe Guo <zejiangg@amazon.com>
Problem / Motivation
The prepare-pr loop has no memory between rounds. Each round sees this round's findings and fixes them. It never sees that the code the finding sits in was itself added two rounds ago, to fix an earlier finding.
Four PRs show what that does. #8501 ran 15 pushes and grew from 2776 to 5182 added lines. #8530 ran 11 pushes and grew from 1094 to 2894. #7518 ran 22 pushes and grew from 387 to 2635. #7362 ran 24 pushes and grew from 3328 to 10143, then deleted 2802 of them. In every one, the author's own disposition comments say the same thing: "this was my own regression", "introduced by the previous round's fix". One span on #7362 drew a finding in eight rounds. It closed when the guard was deleted, not patched a ninth time.
Two smaller defects sit under that one. The skill says the "disproportional" answer never applies to a security-class finding, so the agent fixes every one of them, even one whose trigger needs 2001 concurrent artifacts. And the dispositions the agent writes never reach the remote reviewer: the GPT lane's ledger keeps only the marker line,
>lines, and- **title**bullets from each comment, and the dispositions on #8530 had zero>lines.Why it matters
A PR that should take two rounds takes twenty. The maintainer babysits it for days. The code that lands is 2x to 6x the size the change needed, and most of the extra is machinery the loop grew to answer its own earlier fixes. The rebuttal rate on those four PRs was between 4.5% and 18%.
What changed (motivation → approach → change)
The loop needs to look back. The record it needs is already on the PR: every writer-authored disposition names the head it judged, the finding's
span=, the ruling, and its reasoning. Sopr_findings.py --roundsrebuilds the rounds from those comments — one round per judged head, the spans each round disposed, how many were in self-added code, the mechanisms each round declared, span recurrence, and growth — and exits 30 when the loop's own rules call for a retrospective (a span at ×3, or the 3rd/6th/9th round). Nothing is stored locally. The PR thread is the record, and it stays after merge for anyone to study later. Two optional plain lines in a disposition feed the view:self-added: yes|noandmechanism: <one line>, placed outside the>block so the reviewer's ledger never sees them. The intent is a comment too, posted once.An earlier revision of this PR kept that memory in a file on disk (
round_notes.py, 360 lines, its own directory). Two retrospectives on this PR found first that two of its mechanisms served no part of the intent, then that the file itself did not: everything it held was already in the thread. It is gone.With that memory,
SKILL.mdchanges in these places:max_cycles=80on themonitor_startloop, and the loop does not raise it. When spent, the agent stops and hands the user the rounds view and the open findings; the user decides on another eighty.>lines, written at class level so convergence rule 1 covers the next variant, and a pre-drafted/ai-review overridesentence when a security-class finding is honestly disproportional.pr_status.pybinds the First Principles lane and prints each whole-design lane'sverdict=on its marker line, so an unanswered CONCERNS is visible in the loop's own output.--roundsand branches on its exit code; Phase 3 dispositions carry the two optional lines. Four new Common Mistakes entries.Tests
test/test_prepare_pr_rounds_view.py(4 tests): rounds are grouped by judged head in first-disposed order across lanes, with per-round spans, self-added counts, declared mechanisms, and growth; aspan=inside a>line is evidence, not a claim; exit 30 fires on the 3rd round and on a span disposed in three rounds (twice in one round counts once), and not on a 5th round with no recurrence; the view fails closed when comments or writer verdicts cannot be read;SKILL.mdnames the view, the intent comment, and the two lines.test/test_prepare_pr_ledger_contract.py(3 tests) pins the server-side facts the disposition guidance leans on: the ledger's jq filter selects exactly marker />/ title-bullet lines with the per-record cap SKILL.md restates; convergence rule 1's class coverage and rule 5's no-deferral-for-security are present; SKILL.md states each dependency.test/test_fix_loop_discipline.pynow asserts the skill states how the server treats a deferred security-class finding, instead of the removed "never deferrable" sentence.All existing
test_prepare_pr_*tests pass unchanged.Manual verification
This PR was driven by the skill it changes, from the worktree. Local round: GPT raised three blockers; one was answered by removing an option, two by one-line fixes. Server round 1: one security-fenced blocker rebutted at class level in
>lines; it did not recur. Server round 2: one new fenced blocker, FLAGged by the adjudicator with a complete override rationale. Server round 3: Design and First Principles CONCERNS answered per item; the size trigger and the hard cap they asked for came back. Server round 4: the memory file had drawn GPT findings four rounds running, so a retrospective ran and removed two mechanisms the intent never required. A second retrospective then asked whether the file itself was needed and found it was not.pr_findings.py --roundson this PR reconstructs rounds 0–2 from the comments already posted, with no local state. The PR is +518 lines against +887 one push earlier. The one item left open is a policy question for the maintainer: whether security-class findings may be ruled disproportional with a pre-drafted override.Related Issues
no linked issue: the evidence is the four PRs named above, not a filed issue.
Pattern harvest
Rule candidate: review-prompt
Pattern: "a finding on code the previous review round added is answered by a patch to that code, never by asking whether the code should exist" — the third question in SKILL.md is the local form; the GPT lane's prompt could ask the same of itself before reporting.
Checklist