Skip to content

fix(decisions): run the wake judge on the shared background runtime - #22448

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/wake-judge-timeout
Oct 20, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/wake-judge-timeout

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The wake judge answers inside its 8 s budget, so a gated pull-request loop stops paying a chat turn for a changed tick that asks nothing of its owner.

On this gateway the judge never answered. All 294 nudge.wake decision rows from 2026-10-05 to 2026-10-19 say "error": "timeout", at 8.3 to 10.2 s. A timeout is a FALLBACK verdict, and FALLBACK fires the turn.

Why it matters

A judge that never answers costs more than no judge: the loop waits for it, then pays the turn anyway. The report counts 433 worker turns (2,137 credits) in 18.8 days on ticks that a working judge exists to screen.

Not a goal

  • No change to the judge's budget (decisions.provider.timeout_ms, clamped to 8..60 s) or to its fallback. No new default or setting.
  • No change to the pinned board rows. Since fix(autonudge): ask no judge about an unchanged pull-request board #22299 an unchanged, whole board asks no judge, so a board that reaches the judge has changed. On a progress-only change the judge itself answers QUIET (measured below).
  • No change to answers the strict parser refuses. Haiku 4.5 sometimes writes "outcome": "needs_action" instead of the {"choice", "probabilities"} object. Those calls fall back, on both runners at a similar rate (4 of 24 old, 1 of 24 new, side by side). That needs its own issue.
  • No change to the MCP drain for other sessions.
  • No speed-up on a backend without a shared runtime (Claude, OpenCode, Pi, goose). The judge keeps its per-call session there, and that start was not measured here.

What changed (motivation → approach → change)

Cause. The judge opened a session of its own for every call, through sessions.get_or_create("judge-<uuid>", agent="kirocrew-lite"). That spawns a new kiro-cli and KAS process. The session then waits out the MCP drain's no-report ceiling, _MCP_DRAIN_NO_REPORT_CEILING = 6.0 s: kirocrew-lite mounts no MCP server, so no report ever comes and the wait always runs to the end. On this host the prompt went out about 9 s after the call began. So no call could answer in 8 s, at any load. Prompt size, the model and the start queue are not the cause.

Fix. Background one-liners already have a warm path. run_bg_oneliner opens a session/new on the one shared kirocrew-lite runtime, which is declared MCP-free. Titles, summaries and STT endpointing use it. build_session_runner in decisions/impl_llm.py now makes one such call wherever the backend has that runtime: kiro-cli, KAS and codex, by the same rule get_bg_session uses.

Part Before After, on a shared-runtime backend
Session a new process per call session/new on the warm shared runtime
Model on inherit the template's (claude-haiku-4.5 here) the background role model: the same id
Output bound 64 KiB, counted per chunk the same, as max_output_bytes
Tool requests rejected rejected, audited as wake_judge
Spend no usage row a bg:wake_judge usage row
Budget expires the call is cancelled the wait ends; the call finishes

The runner awaits the call shielded. Cancelling run_bg_oneliner while it gets its session kills a respawn of the shared runtime, or leaks a session/new it already sent. The rejected-model retry is off, because its fallback is the first model the account lists, which can be the most expensive one.

Any other backend (Claude, OpenCode, Pi, goose) keeps the session of its own, as before. There the background session is one persistent conversation: it cannot set a model, so a pinned llm_model would be lost, and a judge turn would leave third-party evidence in the history the next title reads. A session manager that cannot answer the question takes that path too. The decisions spec (§14) says the same.

What a timeout does now. For the loop, nothing new: a call the budget cuts off writes a timeout row, the gate returns FALLBACK at 8 s, and the tick fires. On the shared runtime the model call then runs to its end (its turn capped at 60 s) and its answer is dropped; on a session of its own it is cancelled, as before.

Left open. The root cause is the drain ceiling itself: _MCP_DRAIN_NO_REPORT_CEILING holds every new session on an MCP-free agent for the full 6 s (acp/session_handle_mcp.py calls that agent "the one case that pays the full ceiling"). This PR moves the judge off that path and changes the drain for no one. Four other one-shot callers still open a kirocrew-lite session with get_or_create and pay it: apps/builtins/meetings/backend/domain/translate.py:114 (live translation, a new session per call), apps/builtins/issue_radar/backend/http_routes/ai.py:231, dashboard/handlers/optimizer.py:361 and workflows/service.py:908.

Backwards compatibility

Compatible: the judge's questions, prompt, parser, budget, fallback and model are unchanged; only the session the call runs on moves.

Removes nothing: the per-call judge-<uuid> session was private to the runner, and nothing on origin/main reads those keys.

Tests

test/test_decisions_judge_llm.py, TestRunnerSeam:

  • test_a_call_runs_on_the_shared_background_runtime: one call is a session/new on the shared runtime and never a session of its own; the handle is destroyed.
  • test_the_gate_budget_ends_the_wait_but_never_the_acquisition: a budget that expires while the session is still being acquired raises the caller's timeout at once; the call still acquires, answers, destroys its handle, and leaves nothing held.
  • test_inherit_asks_for_the_background_role_model, test_session_runner_drops_a_model_id_that_is_not_one, test_session_runner_passes_a_valid_model_id: the model each llm_model value asks for, the wake_judge source, the output bound, no rejected-model retry.
  • test_the_response_ceiling_refuses_mid_stream: through the real run_bg_oneliner, a runaway reply stops before the tenth 16 KiB chunk.
  • test_the_runner_bound_never_cuts_a_call_the_gate_waits_for: the runner's 60 s cap is not below the gate's LLM ceiling.
  • test_a_backend_with_no_shared_runtime_keeps_a_session_of_its_own, test_the_own_session_ceiling_refuses_mid_stream, test_an_unreadable_backend_keeps_a_session_of_its_own: on any other backend the call opens its own tool-less session on the pinned model, under REJECT_ALL, bounded per chunk, then torn down; it never touches _bg.
  • test_the_real_session_manager_answers_the_probe: the real SessionManager answers the probe the runner reads by name (KAS and codex yes, Claude no), so a rename fails here instead of quietly sending every call back to a per-call session.

Proof: prove.py with the fix reverted: PROVEN. Ten one-line mutations (no shield, no task hold, no output bound, retry on, inherit sends no model, wrong audit source, no backend split, an unreadable backend read as shared, a renamed probe, own session drops the pinned model) each turn a named test red. Determinism: 20 runs at -n0, 10 at -n 2 beside test_decisions_gate.py and test_decisions_impl_jev.py, and the whole file in a shuffled order (seed 17839): all green. Every test/test_decisions_*.py passes.

Manual verification

Measured on this gateway's host: Linux aarch64, KAS backend, judge model claude-haiku-4.5. A probe process with its own KIROCREW_HOME (a copy of this config) wires a SessionManager the way DashboardState does and calls nudge_wake.judge_tick with a pull-request tick (3,766-char state, 6,200-char prompt). Every call writes its ordinary decision row.

Where the old runner's time went (16 calls, stages from the gateway's own acp_cold_start / acp_startup_stage lines):

Stage Time
spawn kiro-cli + KAS (initialize 1.4-1.6 s inside it) 1.8-2.3 s
session/new 0.36-0.44 s
set_mode → session ready (the 6 s drain) 6.3-6.6 s
prompt → whole answer 1.4-4.3 s

Before and after, through the real gate:

Run Load (1 min) Calls Timeouts Latency
main, 8 s budget 8.0-10.2 8 8 8.8-8.9 s
this PR, 8 s budget 9.2-10.2 20 0 2.7-6.9 s, median 3.9 s
main, budget lifted to 30 s, side by side 9.0-14.7 24 24 over 8 s 11.1-16.5 s, median 12.2 s
this PR, same windows 9.0-14.7 24 1 over 8 s 3.9-8.7 s, median 4.2 s

On a progress-only change (only the two pinned rows, four more lanes passed), every answer the parser accepted was QUIET (needs_owner quiet 0.85, outcome progress_only 0.70-0.75).

Related Issues

Fixes #22290

Pattern harvest

Rule candidate: review-prompt
Pattern: a one-shot model call that opens its own session with sessions.get_or_create(..., agent="kirocrew-lite") pays a process spawn plus the 6 s MCP no-report drain; a one-shot call goes through run_bg_oneliner.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 40b93618ccd8f51e5da8241d8dcc4dcb0115cb4e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 40b93618ccd8f51e5da8241d8dcc4dcb0115cb4e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check what happens when ticks fire faster than the shared runtime clears its queue: each abandoned 60 s call keeps running on the runtime that titles and STT endpointing also use.

[DESIGN-REVIEWED] 40b9361

Description read

[DESCRIPTION-READ] 9226b926237500e00e4b2c4375de6a9b6dc6e671748491482584adc8f0a2b3b5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 40b93618ccd8f51e5da8241d8dcc4dcb0115cb4e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The real cause stays open: _MCP_DRAIN_NO_REPORT_CEILING still adds 6 s to four other kirocrew-lite callers (counted). The description defers them by name.

What this change ships

Inventory (6 items) — 6 justified

Intent: make the wake judge answer inside its 8 s budget, so a gated PR loop stops paying for a turn on ticks a working judge would answer QUIET. FIX.

  1. On kiro-cli, KAS and codex, a judge call now runs on the warm shared kirocrew-lite runtime instead of starting its own process — justified
  2. With inherit (auto), the judge now asks for the background role model by id instead of sending no model — justified
  3. When the budget runs out on the shared runtime, the model call keeps running (up to 60 s) and its answer is thrown away, instead of being cancelled — justified
  4. Judge calls now write a bg:wake_judge usage row and an audit source for refused tool requests — justified
  5. When a model id is rejected, the judge no longer retries on the account's first listed model — justified
  6. Claude, OpenCode, Pi, goose and backends the probe can't read keep a session of their own per call — justified

[FIRST-PRINCIPLES-REVIEWED] 40b9361

Description read

[DESCRIPTION-READ] dd6fcd5e450f1653093d31db7f36001fb3603127fa2f7cd7f06996c825b3fde1

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 40b93618ccd8f51e5da8241d8dcc4dcb0115cb4e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 40b9361

Verdict parsed from the review's SHA-scoped output markers for commit 40b93618ccd8f51e5da8241d8dcc4dcb0115cb4e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 40b93618ccd8f51e5da8241d8dcc4dcb0115cb4e: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 40b93618ccd8f51e5da8241d8dcc4dcb0115cb4e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 40b9361

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 40b93618ccd8f51e5da8241d8dcc4dcb0115cb4e: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the fix/wake-judge-timeout branch from 3da73e7 to 2771221 Compare October 20, 2026 08:09
@iamwhatever

iamwhatever commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the runner reads the session manager's runtime probe by name

  • span=431232e66e45: fixed in 2771221

Fixed by the second option the suggestion names: test_the_real_session_manager_answers_the_probe asks the real SessionManager through impl_llm._has_shared_runtime (KAS and codex answer yes, Claude no).
A rename of _bg_backend_supports_runtime now fails that test instead of sending every judge call back to a per-call session; a mutation that renames the read turns it red.
The probe keeps the private name because it is the rule get_bg_session and the task runner (session_allocation.py) dispatch on, so no new public surface is added.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Design punchline (PASS) on 3da73e7: a pinned llm_model that set_model cannot apply.

The judge then runs on the session's default model, the same as before this PR. The old per-call session withheld an unservable pin at spawn ("not available to this account; staying on the backend default", acp/client_model_selection.py), and resolve_usable_model does the same on the shared runtime ("concrete + not served under either spelling -> inherit the served default"). A pin the account serves does apply: measured on this host, the shared-runtime session reports model: claude-haiku-4.5 after set_model. strict_model=True would turn an unservable pin into a FALLBACK on every tick instead, which is the failure this PR removes, so strict_model stays off.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Inherit names the same model on every backend

Measured: on this build agent.acp_backend = "" (kiro-cli) loads as kas. On it the per-call session and the shared-runtime session both report model: claude-haiku-4.5.
Codex cannot spawn inside this host's sandbox (AcpToolGateUnroutable), so by code: the old path's id was the lite template's, and that template is written from agent.resolve_model("background") (agent_materialization/service_agents.py), the same call the runner now makes. The old path also logged model=claude-haiku-4.5 for a codex judge session.
Where a backend does not serve that id, both paths end on the backend default: the old one through the spawn-time withhold in acp/client_model_selection.py, the new one because resolve_usable_model returns "" and set_model keeps the session default.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 20, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=bd2734253eab: the description now names the root cause it leaves open

Done as the item's clear condition asks: "What changed" has a "Left open" paragraph naming _MCP_DRAIN_NO_REPORT_CEILING for MCP-free agents as the root cause this PR does not touch.
It lists the four callers that still pay it: apps/builtins/meetings/backend/domain/translate.py:114, apps/builtins/issue_radar/backend/http_routes/ai.py:231, dashboard/handlers/optimizer.py:361, workflows/service.py:908.
The cause-level change (the drain for an agent that mounts no MCP server) changes every session start on those backends, so it stays out of this fix, as "Not a goal" declares.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=6551bf9f6fd9: strict_model for a pinned judge model, rebutted

Parity with main: the old per-call session withheld a pin the account does not serve at spawn and stayed on the backend default (acp/client_model_selection.py, "staying on the backend default"); the shared runtime does the same (resolve_usable_model: "concrete + not served -> inherit the served default").
A pin the account serves does apply: measured on this host, the shared-runtime session reports model: claude-haiku-4.5 after set_model, the id the per-call session served.
strict_model=True compares served_model to the requested spelling verbatim, so an unservable pin, or one the backend serves under another spelling, would FALLBACK on every tick: the always-firing judge this PR removes. Tightening the pin is a contract change for its own issue.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Red not from this PR: Backend Tests (Windows) (3) fails on test/test_chat_follow_default_slot_kinds.py::TestSlotCreatorRegistry::test_every_creator_is_classified and ::test_no_stale_entry. The registry still names slack/interactions.py for _import_thread_to_slot, which #22079 moved to slack/interactions_options.py. main fails the same shards (run 39384066208 on cf37920: Backend Tests (3.12, 3) and Backend Tests (Windows) (3)), and the same two tests fail locally on this branch's base. Tracked in #22432, fixed by #22433. This PR waits for that merge, then rebases onto it.

Every nudge.wake call on this gateway timed out at the LLM lane's 8 s
floor, and a timeout fires the turn. The judge opened a session of its
own per call: a fresh kiro-cli/KAS process (about 2 s to spawn), then the
MCP drain's 6 s no-report ceiling, which an agent with no MCP server
always pays in full. The prompt went out about 9 s in, so no call could
answer in time, whatever the host load.

On a backend with a shared background runtime (kiro-cli, KAS, codex) the
runner now makes one run_bg_oneliner call, the path titles, summaries and
STT endpointing take: a fresh session/new on the one warm kirocrew-lite
runtime, declared MCP-free. It asks for the background role model, so
inherit keeps the model the judge ran on. The call is awaited shielded:
the gate's budget still ends the wait at 8 s with a FALLBACK, but the
call itself finishes, so a timeout can never kill a respawn of the shared
runtime or leak a session/new already sent.

Any other backend keeps the session of its own: its background session
is one persistent conversation that cannot set a model, and a judge turn
there would leave third-party evidence in the history titles read next.

Fixes #22290
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main with the #22433 fix; CI re-running.

@iamwhatever
iamwhatever force-pushed the fix/wake-judge-timeout branch from 2771221 to 40b9361 Compare October 20, 2026 08:45
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Design punchline (PASS) on 40b9361: abandoned judge calls on the shared runtime when ticks come fast.

  • How often a call is abandoned. Only when it runs past the 8 s budget. Measured on this host at load 9-14.7: 1 of 44 calls on the shared runtime went over 8 s (p90 6.9 s).
  • What one costs while it runs. It holds a session, not a start permit: the one-permit _bg start gate covers session/new only (0.3-0.9 s here), and the runtime runs turns side by side, so titles and STT endpointing still get their own session/new and turn. run_bg_oneliner's own timeout ends the turn at 60 s at most.
  • How fast ticks come. Since fix(autonudge): ask no judge about an unchanged pull-request board #22299 a gated pull-request loop asks the judge only when its reading changed, once per loop interval (300 s by default), so one loop rarely has a second judge call open while its first still runs.
  • Against main. There, every judge call spawned a whole kiro-cli + KAS process (about 2 s of start-up, plus the 6 s drain) and every call timed out. An abandoned call here is one more session on a process that is already running.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

First Principles punchline (PASS) on 40b9361: agreed, the drain ceiling for MCP-free agents is the cause this PR leaves open. The four callers that still pay it are named under "Left open" in the description, as the cause-level follow-up; this PR changes the drain for no one.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 20, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every review lane on the head is clean (only runner notices and PR Scope size warnings); held by the triage cron on a category, path or P0-area rule, not a finding. Approved on the watch holder's request.

@KiroCrewCaptain
KiroCrewCaptain merged commit c35ba5e into main Oct 20, 2026
104 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/wake-judge-timeout branch October 20, 2026 13:46
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wake judge times out on every call at the 8 s LLM floor, so gated PR loops wake their chat anyway

2 participants