Repository navigation
fix(decisions): run the wake judge on the shared background runtime - #22448
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check what happens when ticks fire faster than the shared runtime clears its queue: each abandoned 60 s call keeps running on the runtime that titles and STT endpointing also use. [DESIGN-REVIEWED] 40b9361 Description read[DESCRIPTION-READ] 9226b926237500e00e4b2c4375de6a9b6dc6e671748491482584adc8f0a2b3b5 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The real cause stays open: What this change shipsInventory (6 items) — 6 justifiedIntent: make the wake judge answer inside its 8 s budget, so a gated PR loop stops paying for a turn on ticks a working judge would answer QUIET. FIX.
[FIRST-PRINCIPLES-REVIEWED] 40b9361 Description read[DESCRIPTION-READ] dd6fcd5e450f1653093d31db7f36001fb3603127fa2f7cd7f06996c825b3fde1 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
3da73e7 to
2771221
Compare
|
self-added: yes
|
|
Design punchline (PASS) on 3da73e7: a pinned The judge then runs on the session's default model, the same as before this PR. The old per-call session withheld an unservable pin at spawn ("not available to this account; staying on the backend default", |
|
|
|
|
Red not from this PR: |
Every nudge.wake call on this gateway timed out at the LLM lane's 8 s floor, and a timeout fires the turn. The judge opened a session of its own per call: a fresh kiro-cli/KAS process (about 2 s to spawn), then the MCP drain's 6 s no-report ceiling, which an agent with no MCP server always pays in full. The prompt went out about 9 s in, so no call could answer in time, whatever the host load. On a backend with a shared background runtime (kiro-cli, KAS, codex) the runner now makes one run_bg_oneliner call, the path titles, summaries and STT endpointing take: a fresh session/new on the one warm kirocrew-lite runtime, declared MCP-free. It asks for the background role model, so inherit keeps the model the judge ran on. The call is awaited shielded: the gate's budget still ends the wait at 8 s with a FALLBACK, but the call itself finishes, so a timeout can never kill a respawn of the shared runtime or leak a session/new already sent. Any other backend keeps the session of its own: its background session is one persistent conversation that cannot set a model, and a judge turn there would leave third-party evidence in the history titles read next. Fixes #22290
|
Rebased onto main with the #22433 fix; CI re-running. |
2771221 to
40b9361
Compare
|
Design punchline (PASS) on 40b9361: abandoned judge calls on the shared runtime when ticks come fast.
|
|
First Principles punchline (PASS) on 40b9361: agreed, the drain ceiling for MCP-free agents is the cause this PR leaves open. The four callers that still pay it are named under "Left open" in the description, as the cause-level follow-up; this PR changes the drain for no one. |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Every review lane on the head is clean (only runner notices and PR Scope size warnings); held by the triage cron on a category, path or P0-area rule, not a finding. Approved on the watch holder's request.
Problem / Motivation
Goal: The wake judge answers inside its 8 s budget, so a gated pull-request loop stops paying a chat turn for a changed tick that asks nothing of its owner.
On this gateway the judge never answered. All 294
nudge.wakedecision rows from 2026-10-05 to 2026-10-19 say"error": "timeout", at 8.3 to 10.2 s. A timeout is a FALLBACK verdict, and FALLBACK fires the turn.Why it matters
A judge that never answers costs more than no judge: the loop waits for it, then pays the turn anyway. The report counts 433 worker turns (2,137 credits) in 18.8 days on ticks that a working judge exists to screen.
Not a goal
decisions.provider.timeout_ms, clamped to 8..60 s) or to its fallback. No new default or setting."outcome": "needs_action"instead of the{"choice", "probabilities"}object. Those calls fall back, on both runners at a similar rate (4 of 24 old, 1 of 24 new, side by side). That needs its own issue.What changed (motivation → approach → change)
Cause. The judge opened a session of its own for every call, through
sessions.get_or_create("judge-<uuid>", agent="kirocrew-lite"). That spawns a new kiro-cli and KAS process. The session then waits out the MCP drain's no-report ceiling,_MCP_DRAIN_NO_REPORT_CEILING = 6.0s:kirocrew-litemounts no MCP server, so no report ever comes and the wait always runs to the end. On this host the prompt went out about 9 s after the call began. So no call could answer in 8 s, at any load. Prompt size, the model and the start queue are not the cause.Fix. Background one-liners already have a warm path.
run_bg_onelineropens asession/newon the one sharedkirocrew-literuntime, which is declared MCP-free. Titles, summaries and STT endpointing use it.build_session_runnerindecisions/impl_llm.pynow makes one such call wherever the backend has that runtime: kiro-cli, KAS and codex, by the same ruleget_bg_sessionuses.session/newon the warm shared runtimeclaude-haiku-4.5here)max_output_byteswake_judgebg:wake_judgeusage rowThe runner awaits the call shielded. Cancelling
run_bg_onelinerwhile it gets its session kills a respawn of the shared runtime, or leaks asession/newit already sent. The rejected-model retry is off, because its fallback is the first model the account lists, which can be the most expensive one.Any other backend (Claude, OpenCode, Pi, goose) keeps the session of its own, as before. There the background session is one persistent conversation: it cannot set a model, so a pinned
llm_modelwould be lost, and a judge turn would leave third-party evidence in the history the next title reads. A session manager that cannot answer the question takes that path too. The decisions spec (§14) says the same.What a timeout does now. For the loop, nothing new: a call the budget cuts off writes a
timeoutrow, the gate returns FALLBACK at 8 s, and the tick fires. On the shared runtime the model call then runs to its end (its turn capped at 60 s) and its answer is dropped; on a session of its own it is cancelled, as before.Left open. The root cause is the drain ceiling itself:
_MCP_DRAIN_NO_REPORT_CEILINGholds every new session on an MCP-free agent for the full 6 s (acp/session_handle_mcp.pycalls that agent "the one case that pays the full ceiling"). This PR moves the judge off that path and changes the drain for no one. Four other one-shot callers still open akirocrew-litesession withget_or_createand pay it:apps/builtins/meetings/backend/domain/translate.py:114(live translation, a new session per call),apps/builtins/issue_radar/backend/http_routes/ai.py:231,dashboard/handlers/optimizer.py:361andworkflows/service.py:908.Backwards compatibility
Compatible: the judge's questions, prompt, parser, budget, fallback and model are unchanged; only the session the call runs on moves.
Removes nothing: the per-call
judge-<uuid>session was private to the runner, and nothing onorigin/mainreads those keys.Tests
test/test_decisions_judge_llm.py,TestRunnerSeam:test_a_call_runs_on_the_shared_background_runtime: one call is asession/newon the shared runtime and never a session of its own; the handle is destroyed.test_the_gate_budget_ends_the_wait_but_never_the_acquisition: a budget that expires while the session is still being acquired raises the caller's timeout at once; the call still acquires, answers, destroys its handle, and leaves nothing held.test_inherit_asks_for_the_background_role_model,test_session_runner_drops_a_model_id_that_is_not_one,test_session_runner_passes_a_valid_model_id: the model eachllm_modelvalue asks for, thewake_judgesource, the output bound, no rejected-model retry.test_the_response_ceiling_refuses_mid_stream: through the realrun_bg_oneliner, a runaway reply stops before the tenth 16 KiB chunk.test_the_runner_bound_never_cuts_a_call_the_gate_waits_for: the runner's 60 s cap is not below the gate's LLM ceiling.test_a_backend_with_no_shared_runtime_keeps_a_session_of_its_own,test_the_own_session_ceiling_refuses_mid_stream,test_an_unreadable_backend_keeps_a_session_of_its_own: on any other backend the call opens its own tool-less session on the pinned model, underREJECT_ALL, bounded per chunk, then torn down; it never touches_bg.test_the_real_session_manager_answers_the_probe: the realSessionManageranswers the probe the runner reads by name (KAS and codex yes, Claude no), so a rename fails here instead of quietly sending every call back to a per-call session.Proof:
prove.pywith the fix reverted: PROVEN. Ten one-line mutations (no shield, no task hold, no output bound, retry on, inherit sends no model, wrong audit source, no backend split, an unreadable backend read as shared, a renamed probe, own session drops the pinned model) each turn a named test red. Determinism: 20 runs at-n0, 10 at-n 2besidetest_decisions_gate.pyandtest_decisions_impl_jev.py, and the whole file in a shuffled order (seed 17839): all green. Everytest/test_decisions_*.pypasses.Manual verification
Measured on this gateway's host: Linux aarch64, KAS backend, judge model
claude-haiku-4.5. A probe process with its ownKIROCREW_HOME(a copy of this config) wires aSessionManagerthe wayDashboardStatedoes and callsnudge_wake.judge_tickwith a pull-request tick (3,766-char state, 6,200-char prompt). Every call writes its ordinary decision row.Where the old runner's time went (16 calls, stages from the gateway's own
acp_cold_start/acp_startup_stagelines):session/newset_mode→ session ready (the 6 s drain)Before and after, through the real gate:
On a progress-only change (only the two pinned rows, four more lanes passed), every answer the parser accepted was QUIET (
needs_owner quiet 0.85,outcome progress_only 0.70-0.75).Related Issues
Fixes #22290
Pattern harvest
Rule candidate: review-prompt
Pattern: a one-shot model call that opens its own session with
sessions.get_or_create(..., agent="kirocrew-lite")pays a process spawn plus the 6 s MCP no-report drain; a one-shot call goes throughrun_bg_oneliner.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)