Skip to content

feat(sage): safely parallelize run-scoped reviews - #8143

Closed
Premshay wants to merge 1 commit into
kirodotdev:mainfrom
Premshay:fix/upstream-sage-parallel-review-v3
Closed

Premshay wants to merge 1 commit into
kirodotdev:mainfrom
Premshay:fix/upstream-sage-parallel-review-v3

Conversation

@Premshay

@Premshay Premshay commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Code Review Sage previously serialized independent run-scoped reviews because workers wrote through a shared result path. That protected result attribution, but it also prevented unrelated claimed reviews from using the existing bounded review pool concurrently.

Why it matters

Independent reviews unnecessarily queued behind each other, increasing turnaround time. Restoring concurrency must not let a worker's result be adopted by a different review dispatch.

What changed (motivation → approach → change)

Each worker now hands its record back inside its own dispatch response, tagged with <code-review-sage-result>, and the driver reads it out of the response to the dispatch it made. That channel is point-to-point — no sibling can write it — so independent claimed reviews can use the bounded pool without cross-run attribution.

The run lock now protects only claim acquisition. Run-scoped reviews delegate concurrency to the pool, while calls without a run scope are clamped to serial — they still write through the legacy shared result directory, where any worker can write any change's record. The standalone CLI is always in that mode, so it no longer offers a --concurrency flag: the value reached the clamp and was discarded, while the help text claimed it applied. The rebased route keeps failure-reason reporting intact in the lock-free control flow.

Tests

  • Route coverage proves independent run bodies overlap and run-scoped reviews delegate concurrency to the bounded pool.
  • Record-adoption coverage rejects a record planted in another change's shared result slot, and proves an invalid record fails its own change without aborting the batch.
  • Prompt-contract coverage requires the <code-review-sage-result> tag in both initial and follow-up reviewer tasks, and that the prompt names no result file on disk.
  • Focused Sage suite: 302 passed.

Manual verification

N/A — the concurrency and adoption boundary are exercised by deterministic backend and driver tests.

Related Issues

N/A.

Relationship to #9077 (read before merging either)

#9077 and this PR solve the SAME problem — the driver cannot prove a result record came from the worker it dispatched — with two different wire formats, and both rewrite the same sage-review/SKILL.md "result" section and the same driver handback path. They conflict textually, and shipping both would give one contract two formats.

This PR should land first and own the transport. The binding is a prerequisite for the parallelization here (a shared result path is exactly what makes concurrent reviews unattributable), and the response-bound <code-review-sage-result> tag is sufficient on its own: the driver reads the record out of the response to the specific dispatch it made, so the channel is already point-to-point and a driver-issued capability token authenticates nothing further.

#9077 is then rescoped to the part this PR does not cover: enforcing a coverage follow-up against the first pass IN THE DRIVER. build_review_followup_task here asks the worker for a complete replacement record and preserves phase1 by instruction only, so a follow-up can silently rewrite or drop first-pass findings. #9077's base_digest binding, preserved first-pass metadata, append-only findings and recomputed counts close that, and layer cleanly on this transport.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat: safely parallelize run-scoped reviews)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (not applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

The template's exact CLA wording has not yet been supplied by OSPO.

@Premshay
Premshay requested review from a team and buluoray September 3, 2026 09:24
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 3, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

3 similar comments
@iamwhatever

Copy link
Copy Markdown
Contributor

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

@iamwhatever

Copy link
Copy Markdown
Contributor

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

@dwu96

dwu96 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • PR #5274 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #5274: REBASE. Complementary goals (per-run model pinning vs cross-run concurrency safety) that meet in one function and in the pool's reuse guard. Whoever lands second must reconcile the hunk and decide what an overlapping mismatched-model review does: queue, or surface the busy error. Files: src/kiro_crew/apps/builtins/code_review_sage/backend/routes.py.
  • PR #6802 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #6802: CONTINUE_DEVELOPMENT. Same file and same lock region, different goals, verified textual conflict. Both fixes are wanted; whichever lands second must re-place the _CANCELLED recheck against the other's lock scope rather than reapply its own hunk. Files: src/kiro_crew/apps/builtins/code_review_sage/backend/routes.py.
  • This PR is OVERLAPPING with PR #8186. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #8143: REBASE. Unrelated goal (failure-cause token vs parallelization) but a direct textual and semantic collision in the rewritten block; the rebase must re-add PR #8186's run["reason"] handling in the new, lock-free control flow. Files: src/kiro_crew/apps/builtins/code_review_sage/backend/routes.py.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

@dwu96

dwu96 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

1 similar comment
@iamwhatever

Copy link
Copy Markdown
Contributor

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 4, 2026
@Premshay
Premshay force-pushed the fix/upstream-sage-parallel-review-v3 branch from db7f1a7 to c9246a6 Compare September 4, 2026 21:17
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 4, 2026
@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — 🔴 changes requested (blocking)

Reviewed 3e133450e00786bea58b69d650374def595b67e7 via the fork AI-review pipeline; updated in place on each push.

Adjudication produced no [GPT-ADJUDICATED] marker for 3e13345: the adjudication stage FAILED TO RUN (its raw output is in the Adjudication details block -- typically an API or setup error), so no proportionality review took place and GPT's blocking verdict stands unreviewed.

BLOCKING -- src/kiro_crew/apps/builtins/code_review_sage/sage_lib/results.py:295 -- response handoff bypasses the result-size bound
record = json.loads(raw)
Oversized tagged response -> unbounded JSON is parsed and persisted -> memory and disk exhaustion.
Anchor: a-bound-bounds-every-field-it-retains
Fix: Reject raw above _RECORD_MAX_BYTES before parsing.

BLOCKING -- src/kiro_crew/apps/builtins/code_review_sage/sage_lib/review_driver.py:779 -- posting discards the only durable review before delivery
results.result_path(change_id, root, run_id).unlink() / post = _post_pending(change_id, link, rev_rec if handoff_rec is not None else None)
Explicit or automatic post -> result exists only in memory during dispatch -> gateway termination permanently loses findings and retry data.
Anchor: residual/crash-data-loss-corruption
Fix: Keep a durable copy until posting completes, then atomically update it.

[BLOCK-MERGE] 3e13345
[GPT-REVIEWED] 3e13345

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of ea3d7aa3fa0930e7583d89c649ca0db629be7e3c via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Sound root-cause fix — the shared write channel is removed, not re-matched — but the record must now fit one LLM response, which caps the biggest reviews.

Watch

  • The record moves from a tool-written file (4 MB cap) to the worker's final message ("Return the complete record in your final response… Do not write the result record to disk"). A large PR with many findings/snippets can exceed the model's output-token ceiling; the truncated handoff fails result_from_handoff and 90 minutes of review collapses into no_review_recorded — indistinguishable from a worker that produced nothing, so the operator can't tell truncation from failure. The follow-up and poster prompts also inline the full record/payload, so prompt size scales the same way.
    Clears when: a distinct failure reason for a present-but-unparseable handoff plus evidence (measured max record size or a size guard) that records fit under the worker's output cap.
  • In post_recorded, the run-scoped retry path unlinks the on-disk record before dispatching the poster ("could not remove the record before posting") and rewrites it afterward. A gateway crash mid-post destroys the only durable copy on exactly the path whose purpose is retrying delivery — the review must be redone to re-post. The in-memory cur is already authoritative and the poster no longer reads the file, so the unlink removes no hazard the final write-back doesn't already handle.
    Clears when: the record stays on disk through the post (final atomic write_result remains the authority), or a stated hazard justifies the delete.

Suggestions

  • Replace the unlink/rewrite dance with "ignore the filesystem during the post, atomically write back at the end" — same attribution guarantee, no crash window.

[DESIGN-REVIEWED] ea3d7aa

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of ea3d7aa3fa0930e7583d89c649ca0db629be7e3c via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All evidence is in. The change replaces Code Review Sage's shared-file result handback with a response-bound tagged handoff and restores pooled concurrency; the base tree confirms the deleted serialization comments themselves named "per-dispatch staging whose path a sibling worker cannot guess" as the condition for restoring parallelism, which this PR meets — so the rewritten pins carry their own evidence. Two things don't earn their place as shipped: the unscoped-serial clamp protects a legacy file fallback whose worker-side producers this same PR removes (both prompt builders and SKILL.md now forbid disk writes), and 11 baselined files are wholesale black-reformatted, burying the transport change.

First-Principles-Verdict: CONCERNS

The serial clamp guards a legacy file fallback no prompt feeds anymore, and whole-file reformatting of 11 baselined files buries the actual change.

Not justified as shipped

  • Item 7 — symptom-level: the description says unscoped calls "still write through the legacy shared result directory," but this diff's own prompt says "Do not write the result record to disk" unconditionally and test_prompt_exposes_no_result_file pins data/results out of the prompt — I count 0 remaining worker-side producers of that directory, so the clamp serializes to protect a fallback only test fakes feed.
  • Item 8 — rides along: the --concurrency flag removal exists only because item 7's clamp made it dead.
  • Item 10 — rides along, undeclared: all 11 touched .py files sit in .github/black-baseline.txt (grepped code_review_sage there: lines 38–74) and are reformatted wholesale — review_pool.py's hunks are almost entirely format-only — yet the baseline file itself is untouched.

What this change ships

Inventory (10 items) — 7 justified

Intent: let independent claimed reviews run concurrently again by having each worker return its result inside its own dispatch response instead of through a shared file — an ADDITION.

  1. Reviews within a run now run in parallel, bounded by the pool — justified
  2. Two independent runs no longer queue whole-run behind one lock — justified
  3. Reviewer hands its record back in <code-review-sage-result> response tags; disk writes forbidden — justified
  4. Poster reports delivery in <code-review-sage-posted> tags and gets the payload in its prompt — justified
  5. Follow-up reviewer receives the prior record inline; phase1 preserved by instruction only, deferred to fix(sage): bind review results to worker responses #9077 — justified
  6. The whole shared-staging bridge (stake/adopt/publish/clear_staged) and its symlink guards are deleted — justified
  7. Standalone/unscoped reviews are clamped to serial (previously defaulted to 5) — symptom-level, protects a fallback with no remaining producer
  8. The CLI loses its --concurrency flag — rides along on 7
  9. Run-scoped records are persisted only after every worker turn ends — justified
  10. Wholesale black reformatting of 11 baselined files, plus gutting _first_change_failure's docstring — rides along, undeclared

Watch

  • The clamp/fallback pair keeps two result transports: result_from_handoff is tried first for every caller, so the unscoped file read (if rev_rec is None and not run_id) only fires when the tag is absent — and nothing in the shipped prompts or skill produces that file. Clears when: the fallback and the if not run_id: concurrency = 1 clamp are deleted, or a real out-of-prompt producer of data/results/<id>.json is named.
  • If the reformatted files are now black-clean, check_black_formatting.py fails on graduated entries the diff never prunes. Clears when: the black gate passes on this head, or the format-only hunks are dropped.

Subtractions

  • Delete the unscoped file fallback in review_driver.py and the concurrency = 1 clamp with it — one transport for all callers; the flag debate then evaporates.
  • Drop the format-only hunks in the 11 baselined files, including the _first_change_failure docstring deletion and write_result's duplicated path = result_path(...) line in both branches.

[FIRST-PRINCIPLES-REVIEWED] ea3d7aa

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed ea3d7aa3fa0930e7583d89c649ca0db629be7e3c via the fork AI-review pipeline; updated in place on each push.

Review details

I have enough to complete validation. Let me verify my reconstruction of the PR-head post_recorded retry path and confirm the base kept the record on disk (already confirmed in the on-disk base file at lines 807/820/844). The diff replaces that with an up-front unlink() + in-memory hold.

Now I'll assess the two candidates against the falsification bar.

Candidate 1 (retry-post unlinks the durable record before the poster dispatch): grounded. On the dashboard explicit-retry path (_handle_run_post → _post_comments_bg → post_recorded(record=None, run_id set)), the PR-head code does results.result_path(...).unlink() up front, holds cur only in memory, dispatches the poster (bounded by DEFAULT_TASK_TIMEOUT), and only rewrites via the terminal write_result(after, ...). Base kept the run-scoped record on disk throughout (write_result(cur) + publish_to_shared copy, never removing the original). So a process restart during the posting window loses the only durable copy → retry hits read_result → None → "no recorded review for this change". Real regression. But the trigger is an external crash in a (usually short) window and the loss is recoverable by re-review — not a deterministic/irreversible data-loss defect. Advisory, not blocking. The in-diff fix (don't unlink; let the final write overwrite) keeps it out of the override.

Candidate 2 (records held in memory until the pool drains): this is the deliberate, load-bearing design of the PR — the deferred write exists precisely "so a sibling session never gets a filesystem window onto another worker's mutable handoff." A mid-run crash loses in-progress (not-yet-complete) reviews that would be re-run anyway; the fix would require the exact per-change filesystem persistence the PR intentionally removed for a security reason. By-design, recoverable, low harm — dropped.

No new grounded 80+ findings surfaced in Step 2.

No blocking issues; one advisory: the dashboard retry-post path deletes the only durable result record before a long poster dispatch.

FINDING — sage_lib/review_driver.py (post_recorded, the results.result_path(change_id, root, run_id).unlink() before spawn = dispatch(post_prompt, timeout)) — on the explicit dashboard retry-post (record=None, run_id set) the run-scoped record is unlinked up front and held only in cur, so a gateway restart during the poster dispatch loses the sole durable copy and the retry reports "no recorded review for this change"; base kept it on disk via publish_to_shared throughout → Fix: don't unlink() first — leave the record in place and let the terminal write_result(after, ...) overwrite it (or write a temp and rename over it only on success).

[OPUS-REVIEWED] ea3d7aa

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 4, 2026
@Premshay
Premshay force-pushed the fix/upstream-sage-parallel-review-v3 branch from c9246a6 to b910950 Compare September 5, 2026 09:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 5, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 10, 2026
@Premshay
Premshay force-pushed the fix/upstream-sage-parallel-review-v3 branch from 9a0e01e to e69bada Compare September 10, 2026 18:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 10, 2026
@chenmingwei23 chenmingwei23 added the needs-pr-triage PR scanner: awaiting automated triage label Sep 13, 2026
@NicholasRBowers NicholasRBowers added needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 13, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

When you've addressed these, the pipeline will re-assess on its next cycle.

@Premshay
Premshay force-pushed the fix/upstream-sage-parallel-review-v3 branch 3 times, most recently from e69bada to 51ceb11 Compare September 17, 2026 21:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@Premshay

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (c5842a635) — mergeable again (51ceb11ca).

Nine conflict hunks across five files, all one architectural disagreement:

  • main pinned concurrency=1 and made parallelism safe through shared-filesystem adoption — stake_shared / adopt_from_shared / publish_to_shared, with symlink refusal, schema validation and atomic rename, covered by TestAdoptionRefusesAPlantedLink, TestAdoptionValidatesBeforeItWrites and TestPublishRefusesAPlantedDestinationLink.
  • This branch routes the reviewer's record back through the dispatcher response (result_from_handoff) and restores concurrency=0.

The resolution takes this branch's side throughout, which deletes that adoption machinery and its tests: with no shared result path left, there is nothing for those guards to protect. main's own note — "restoring parallelism needs per-dispatch staging whose path a sibling worker cannot guess" — is satisfied more directly by the response handoff. Please weigh that deletion explicitly; it is a security-relevant trade, not a mechanical rebase artifact.

Also fixed: the branch was red at its own head. e69bada9b removed the result_capability mechanism from the source but left test_malformed_unicode_capability_fails_one_change_not_the_batch asserting it (result_capability existed only in that test). Removed, folded into the same refactor commit.

779 tests in code_review_sage/tests pass. Ready for review.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
Bind reviewer and poster results to their dispatcher responses instead of a
shared filesystem staging path: each worker returns its record inside its own
dispatch response, which no sibling session can write. Validate the outer
handoff envelope and keep GitHub read-back as delivery confirmation.

The prose, help text and concurrency surface describe only mechanisms the code
actually provides. A result capability is never checked, produced, or named
outside a rejection guard, so no docstring, comment, or flag references one.
The --concurrency flag is gone with it: it reached no execution path, since
standalone runs have no run id and are clamped to 1 immediately after the value
resolves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Premshay
Premshay force-pushed the fix/upstream-sage-parallel-review-v3 branch from 51ceb11 to 3e13345 Compare September 19, 2026 20:32
@Premshay

Copy link
Copy Markdown
Contributor Author

Adopting the split. #8143 shrinks to the run-lock narrowing plus concurrency=0 pool delegation. The response handoff rides in #9077 and the interrupted-posting reconciliation in #9087.

The narrowing is only safe once records return inside each worker's own dispatch response, so I'll re-push #8143 as that narrowed slice right after #9077 lands, rebased onto whatever main carries then. The current head still carries the full scope — including the shared-adoption deletion, which the narrowed push will not re-apply — so the two blocking GPT findings, which live in the handoff and poster paths this PR is giving up, go away with it.

Until #9077 lands, that PR is the one worth review; nothing here needs another look until the narrowed push.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 24, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Closing in favour of #9077, which implements the same change.

Evidence. 9 shared non-infrastructure files on one mechanism: sage_lib/results.py, sage_lib/review_driver.py, the sage-review SKILL.md and the activity-progress tests. Overlap 0.75 and unchanged after excluding common files, so it is real.

Why this one and not #9077. #9077 is the focused correctness fix and is mergeable; this branch is dirty and carries a much wider refactor around the same code. The parallelisation here is worth re-cutting on top of #9077 once it lands.

This is not a judgement on the work -- the two changes were written independently and this one is not the weaker of the pair, it is the one that does not need to land twice. If you disagree, or if #9077 turns out to miss something this branch handles, say so and I will reopen this.

From a full relationship audit of every open PR against main (baseline a992d5cda5). Overlap was measured with git diff against each branch's own merge base, and re-measured after excluding files that 20+ open PRs touch, so shared locale and lockfile edits could not inflate it.

@bolichen97 bolichen97 closed this Sep 25, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 25, 2026
@Premshay
Premshay deleted the fix/upstream-sage-parallel-review-v3 branch November 13, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants