Skip to content

feat(code-review-sage): run reviews on claude and other client backends - #31145

Merged
bolichen97 merged 1 commit into
mainfrom
fix/22196-review-backend-client
Nov 9, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/22196-review-backend-client

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A host whose configured agent backend is Claude (or another backend the shared review runtime cannot drive) runs its Code Review Sage reviews on that backend instead of on kiro-cli.

Code Review Sage drives every review through one shared AcpRuntime, and that runtime only serves kiro, KAS and Codex. On a host set to claude, opencode, goose, pi or deepseek, reviews quietly run on kiro-cli with one warning. Review quality, sign-in, quotas and models then come from a provider the operator did not pick, and a host with only claude-agent-acp installed cannot review at all. An LTS build refuses those reviews outright.

Why it matters

An operator who chose a backend expects reviews on it. Today a review failure on such a host is traced to the wrong provider, and a host without kiro-cli has no working Code Review Sage.

Not a goal

What changed (motivation → approach → change)

Reviews now follow the configured backend on every backend a per-session client can run.

Configured backend Before After
kiro, KAS, Codex one shared runtime per batch unchanged
claude, opencode, goose, pi, deepseek, copilot, strands kiro-cli with a warning; refused in an LTS build each review task runs on a provider of its own on that backend
a backend served neither way kiro-cli with a warning; refused in an LTS build unchanged

Approach. review_backend() asks the same capability the knowledge pool asks, acp_client_spawnable. For such a backend the batch spawns no shared runtime. Each task starts one AcpProvider through build_acp_provider with the shared runtime's own settings (sandbox_mode="auto", the app root as cwd, the reviewer agent) plus the reviewer model and the configured review.effort. The effort is keyed on that model, or carried unkeyed when the reviewer runs the unpinned default, so the dashboard's effort is the one the review runs at. It runs through the same loop as a runtime session: the security-floor refusal, auto-approval, the per-tool SEL audit and the host-deny notice. Then it is shut down.

Three things keep it as contained as a destroyed runtime session:

  • memory_mode="temporary": Claude's auto-memory is off, so nothing one review writes reaches the next, and its transcript is removed at shutdown.
  • Its PID is shielded from the orphan sweep while the task runs, because the process sits outside the session map.
  • ReviewPool.shutdown() stops any such provider still running. Pool occupancy counts these tasks too.

The config provider factory was the other option. It was not used because it takes the operator's agent.sandbox, and strict hides ~/.config/gh, which the reviewer's gh CLI needs. It also takes the Claude permission mode, while a review must run under the same fixed settings as the shared runtime.

The acp-client, governance, resource-protection and sel specs and the Windows install guide (windows-install.md) describe the per-task client.

The app's own repository carries the same change: commit 0429f42f2528a8e8dfa9bbf2370872276f2a90b2 of KiroCrewCaptain/kirocrew-app-code-review-sage (its pull request 1), merged to its main. There the four host names are looked up by name, and a host without any of them keeps the kiro-cli fallback.

Backwards compatibility

Compatible: kiro, KAS and Codex hosts run exactly as before. A host on another client backend that ran reviews on kiro-cli now runs them on its configured backend. An LTS build that refused such a review now runs it on the backend its policy allows.

Removes nothing: no key, record, route or public function is removed. The kiro-cli fallback and its warning stay for a backend no per-task client can run.

Tests

  • test_claude_runs_each_review_on_a_claude_client: with agent.acp_backend=claude, no kiro-cli runtime spawns. The review runs on a claude provider built with sandbox_mode="auto", memory_mode="temporary" and the app root. Its PID is shielded during the turn, and the provider is shut down and unshielded after it.
  • test_a_claude_client_takes_the_configured_review_effort: a pinned model gets review.effort keyed on it, the unpinned default carries it unkeyed, and an empty effort passes nothing.
  • test_a_client_that_cannot_start_is_shut_down_and_never_shielded: a sandbox refusal at start becomes ReviewRuntimeUnavailable. The provider is shut down, and nothing is shielded or counted.
  • test_shutdown_stops_a_client_still_running: ReviewPool.shutdown() stops a provider mid-turn, exactly once.
  • test_a_backend_served_neither_way_falls_back_to_kiro_and_warns_once: the claude fallback test, narrowed to a backend no client can run, keeps the single warning.
  • LTS floor (test_review_pool_lts_floor.py): test_a_client_backend_the_policy_allows_runs_on_it. The three refusal tests now model a backend served neither way.
  • test_followup.py's pool stub returns the runtime with its backend, which is what the follow-up decision reads.
  • test_pid_reader_ratchet.py counts the per-task client's PID read as an owner-marked site, since the pool spawns that process and ends it.
  • Related backend set (run_scoped_tests.py --surface backend, 4 workers): 3380 passed.
  • Determinism, for test_review_pool.py::TestReviewBackend and test_review_pool_lts_floor.py: 20/20 green at -n0; 10/10 green at -n 4 --dist loadgroup --max-worker-restart=2 beside test_followup.py and test_review_driver.py; their 22 ids shuffled with seed 1625, all passed.

Manual verification

Not run here, and required before relying on it: one real review on a host with claude-agent-acp signed in and gh authenticated. This host has no claude-agent-acp. Unit coverage pins the pool's wiring, approval and teardown, not the reviewer's output quality on another harness.

Related Issues

Part of #22196

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 6, 2026 06:37
@bolichen97
bolichen97 requested a review from Zedmor November 6, 2026 06:37
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 6, 2026 06:43
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 746c9478c30c5145db6da7580d54936993efb7fe touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 746c9478c30c5145db6da7580d54936993efb7fe — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Reviews now run on the backend the operator configured, using the same capability check the knowledge pool already relies on. The one remaining concern is that the PR hand-writes per-process lifecycle code an existing engine already owns.

Not checked: whether claude-agent-acp runs a real review end to end. The author says this was not run.

Notes

  • Item 3 — duplicate of src/kiro_crew/acp/worker_pool.py: _start_review_client hand-writes register_protected_pid/unregister_protected_pid, a self._clients teardown set and shutdown sweeping. WorkerPool already owns all of these, and its own comment says so: "The engine owns this lifecycle so every pool built on it is protected by construction — no per-worker wiring, and it can't be forgotten by a new consumer." 1 existing owner, 1 new hand-written copy. Subtraction: replace the _clients set and the inline shield calls with a WorkerPool whose workers are not reused (one fresh worker per task), and delete the hand-written copy.

What this change ships

Inventory (6 items) — 5 justified

Intent: an operator who set agent.acp_backend to Claude (or a similar client backend) gets Code Review Sage reviews on that backend instead of a silent kiro-cli fallback. ADDITION.

  1. On a claude/opencode/goose/pi/deepseek/copilot/strands host, reviews run on the configured backend instead of kiro-cli plus a warning — justified
  2. An enterprise build that refused those reviews now runs them on the backend its policy allows — justified
  3. Each review task on such a backend starts its own process, keeps its PID safe from the orphan sweep, and stops it afterwards — duplicate of acp/worker_pool.py lifecycle
  4. The review's memory is temporary, so one review's notes never reach the next — justified
  5. Pool occupancy counts these per-task processes, and pool shutdown stops any still running — justified
  6. No follow-up chat is offered for reviews on these backends — justified

[FIRST-PRINCIPLES-REVIEWED] 746c947

Description read

[DESCRIPTION-READ] dbf23e8f0b2e74615d27e69e31cb57b8f8451c1692b78e8c7b9d0d02e527be83

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

Design-level review of 746c9478c30c5145db6da7580d54936993efb7fe — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

The new per-task client path changes where every review runs on claude/opencode/goose/pi/deepseek/copilot/strands hosts, and the description says it has never been run against a real harness.

Blockers

Manual verification skipped on a branch that only runs where CI can't reach
Trigger: "Manual verification — Not run here, and required before relying on it: one real review on a host with claude-agent-acp signed in … This host has no claude-agent-acp."
Outcome: review_backend() now returns claude (and the other client backends) through capabilities_for(resolved).acp_client_spawnable. Every review on those hosts then goes to _start_review_client → build_acp_provider(..., sandbox_mode="auto", memory_mode="temporary") → provider.stream. Before, a host with kiro-cli installed got working reviews on kiro-cli. Now it gets reviews on a harness this path has never driven. The tests use stubs, so they cannot show any of these working:

  • the reviewer agent spec, which "Not a goal" says gets no projection change;
  • gh running under that harness's sandbox;
  • the permission-event shapes the auto-approve loop expects;
  • the PID shielding.

If any one fails, every review on those backends fails, and no setting brings back the kiro-cli path.

Fix: run one real review end to end on a claude host (findings posted, provider torn down, no orphan process) and record the result in the description.

Clears when: the description's Manual verification section records a real claude review completing on this revision, with the observed outcome (review posted, stats() back to 0, process gone).

Notes

[DESIGN-REVIEWED] 746c947

Description read

[DESCRIPTION-READ] c3e6b75d48003f7e6956cf43bb4d620df4c72316d81e21257b8a58a55119519c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 746c9478c30c5145db6da7580d54936993efb7fe — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 746c947

Verdict parsed from the review's SHA-scoped output markers for commit 746c9478c30c5145db6da7580d54936993efb7fe.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 746c9478c30c5145db6da7580d54936993efb7fe false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 746c9478c30c5145db6da7580d54936993efb7fe and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 746c947

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 746c9478c30c5145db6da7580d54936993efb7fe false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/22196-review-backend-client branch from 69a4217 to dc1a308 Compare November 7, 2026 02:59
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/architecture/resource-protection.md, docs/system-specs/modules/acp-client.md, docs/system-specs/modules/governance.md, docs/system-specs/modules/sel.md since this head went green (1e990f8 -> d0583bc); CI re-running.

The same push records the review pool's PID read for its per-task client as an owner-marked site in test/test_pid_reader_ratchet.py, which clears the Static Ratchet Gate red. It also drops a contradictory clause from the Windows install guide row.
[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Pool shutdown misses providers still starting span=e22a8ffbc364

Rebutted: this is a short window during teardown, and the process does not leak. In the gateway, shutdown_pool() runs only from on_cleanup, and then asyncio.run cancels every task still pending.
A cancel inside provider.start() lands in its except BaseException, which shuts the provider down. A cancel after start lands in _send's finally, where close() shuts it down. Either way the process ends when the loop exits.
Tracking pending starts and draining them in shutdown() would add a new mechanism for a teardown window a few seconds long. The lane's own adjudication downgraded this finding as disproportionate.
[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/22196-review-backend-client branch from dc1a308 to bf22d0e Compare November 7, 2026 05:52
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/architecture/resource-protection.md, docs/guides/windows-install.md, docs/system-specs/modules/acp-client.md, docs/system-specs/modules/governance.md since this head went green (d0583bc -> 480d162); CI re-running.

The rebase is clean and the patch is unchanged. The previous head's red lanes all failed on the CI app's installation rate limit, not on a finding, so this run re-judges them.
[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026
@bolichen97
bolichen97 force-pushed the fix/22196-review-backend-client branch from bf22d0e to 42ea4c6 Compare November 7, 2026 08:24
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/acp-client.md, docs/system-specs/modules/governance.md, docs/system-specs/modules/sel.md since this head went green (480d162 -> 0795ebc); CI re-running.

The rebase is clean and the patch is unchanged. The previous head's two red test shards failed on tests this diff does not touch: test_the_scan_is_linear_not_quadratic (the wall-clock flake tracked in #28212) and test_connect_runs_through_the_mint_row_to_granted; Coverage Gate failed only because those shards did.
[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 7, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 7, 2026
@bolichen97
bolichen97 force-pushed the fix/22196-review-backend-client branch from 42ea4c6 to 05d2e1f Compare November 7, 2026 10:09
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The two red Backend Tests jobs (shard 2, Linux and Windows) fail only test/test_security_gate_liveness.py::test_the_package_stays_within_its_line_budget. That test measures the kiro_crew.security package, which this PR does not touch: at this head's base (e30915e) the package was already 34,855 lines against a 34,845 budget. Main raised the budget in #32252. Coverage Gate and PR Readiness fail only because those shards did.

A rerun reuses the old merge commit, so only a re-sync onto current main clears it, and this PR has spent its re-sync budget (3 of 3). Main has since moved in three files this PR edits (docs/architecture/resource-protection.md, docs/system-specs/modules/acp-client.md, docs/system-specs/modules/governance.md), and the merge is clean. A project member's Re-sync budget reset comment lets the loop re-sync it once. It also still needs a maintainer approval, the Sensitive Change sign-off, and the real claude-agent-acp review run the Design Review asks for.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Re-sync budget reset

The base moved under this PR more times than the re-sync budget allows; a maintainer asked for one more round of re-syncs so it can be rebased and its conflicts resolved.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/22196-review-backend-client branch from 05d2e1f to df402d2 Compare November 8, 2026 20:31
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/architecture/resource-protection.md, docs/guides/windows-install.md, docs/system-specs/modules/acp-client.md, docs/system-specs/modules/governance.md since this head went green (e30915e -> a398062); CI re-running.

The rebase was clean. The old head's only red was main's own: Backend Tests shard 2 (Linux and Windows) failed test/test_security_gate_liveness.py::test_the_package_stays_within_its_line_budget at the old base, a package this PR does not touch. The new base carries #32252, which raised that budget, and later main commits.

[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Subtraction in this push, after a retrospective over every mechanism this PR adds.

removed in df402d2: _BatchRuntimeHolder._backend copied a fact the runtime already carries. A live runtime's backend is read from rt.acp_backend again, as main's follow-up check reads it. The holder builds a runtime only for a backend in ACP_BACKENDS_ACP_RUNTIME, and passes that backend in, so the value is the one the runtime was spawned under.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Subtraction in this push, after a retrospective over every mechanism this PR adds.

reverted in df402d2: the kiro-cli fallback warning and its reason are back to main's wording ("is not served by the shared review runtime"). That text is still true for the only case that reaches it: a backend neither the shared runtime nor a per-task client can run.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The changed path has no manual verification, and CI cannot exercise it span=52807898341a

Acknowledged: only one real review on a host with claude-agent-acp signed in and gh authenticated clears this, and no code change can. The host this PR is driven from has no claude-agent-acp, which the description states under Manual verification.
Unit tests pin the per-task client's wiring, approval, PID shield and teardown, not a reviewer's output on another harness.
Question for a maintainer: can someone with a claude backend host run one review on this branch and post the result here, or rule that the unit coverage is enough for this PR?

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 8, 2026
@bolichen97
bolichen97 force-pushed the fix/22196-review-backend-client branch from df402d2 to 1465be4 Compare November 8, 2026 23:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 8, 2026
A host whose agent.acp_backend is a backend the shared review runtime
cannot drive (claude, opencode, goose, pi, deepseek, ...) ran every
review on kiro-cli instead, so a host with only claude-agent-acp could
not review at all. Such a backend now runs each review on a provider
of its own, with the runtime's sandbox mode, cwd, approval and audit,
the configured review effort, and its PID shielded from the orphan
sweep while the review runs.

Part of #22196
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] On the First Principles note (item 3, a duplicate of acp/worker_pool.py's lifecycle), kept as is, with reasons:

Rebutted: WorkerPool cannot host this path without hiding the part that must be shared. Its worker contract is send_message(prompt) -> str, so the per-event loop (the security-floor refusal, auto-approval, the per-tool SEL audit, the host-deny notice, activity and stall handling) would have to move inside a worker, beside a second copy for the shared-runtime path. Here both paths run through one loop in _send.
Its semantics also differ from what a review needs: idle workers are reused after a reset(), while each review must get a fresh provider with temporary memory; and it carries its own task and start semaphores, which would sit under ReviewPool's own semaphore. One fresh worker per task would turn off most of the engine.
The copy is small: a two-call PID shield and a teardown set that stats() and shutdown() read.

The same change is ported to the app's own repository through its _host lookups, so the design note about new core imports is answered there: a host without them keeps the kiro-cli fallback.

KiroCrewCaptain added a commit to KiroCrewCaptain/kirocrew-app-code-review-sage that referenced this pull request Nov 9, 2026
A host whose agent.acp_backend is a backend the shared review runtime
cannot drive (claude, opencode, goose, pi, deepseek, copilot, strands)
ran every review on kiro-cli with one warning, and an enterprise build
refused it. Review quality, sign-in, quotas and models then came from a
provider the operator did not pick.

For such a backend the batch now spawns no shared runtime. Each task
starts one provider through the host's build_acp_provider with the
runtime's own settings (sandbox_mode auto, the app root as cwd, the
reviewer agent) plus the reviewer model and review.effort, runs through
the same approval, audit and host-deny loop, and is shut down after.
It runs with temporary memory, its PID is kept out of the orphan sweep
while it runs, shutdown() stops one still running, and the pool's
occupancy counts it. No follow-up is offered for such a review.

The four host names are looked up through _host; a host missing any of
them keeps the backend on the kiro-cli fallback, as before.

Ported from kirodotdev/KiroCrew#31145 (part of kirodotdev/KiroCrew#22196).

Co-authored-by: Bolin Chen <bolichen@amazon.com>
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto main (d0583bc-era base -> 41ba859): main had renamed the Enterprise Edition to the LTS Edition (#32970), which conflicted in docs/system-specs/modules/governance.md and in the floor test, now test_review_pool_lts_floor.py. The resolution keeps this PR's wording under the LTS name, and the new floor test calls _lts(). The description's naming follows.

The old head's red Backend Tests shard 6 (Linux and Windows) failed only test/test_dashboard_server_composition_contract.py::test_the_lazy_imports_stay_inside_the_functions_that_need_them (140 == 139), a dashboard file this PR does not touch; it passes on this base. Coverage Gate failed only because the shard did.

The same push drops the em dashes this PR's added and rewrapped lines carried (review_pool.py, acp-client.md, windows-install.md); no code changed. Local gate: black, encoding, isort, flake8, mypy, comment history, brand, line cap and docs lint clean; the related backend set (74 files) 3380 passed.

The app's own repository has the same change: commit 0429f42f2528a8e8dfa9bbf2370872276f2a90b2 of KiroCrewCaptain/kirocrew-app-code-review-sage.
[operator: bolichen97]

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 9, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on head 746c947: PR Readiness is green, every AI review lane on this head passes with no blocking findings, and no change request stands.

@bolichen97
bolichen97 merged commit 4a0cb55 into main Nov 9, 2026
109 checks passed
@bolichen97
bolichen97 deleted the fix/22196-review-backend-client branch November 9, 2026 00:54
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants