Skip to content

fix(chat): route every queue drain through one queue-held gate - #11029

Closed
rnoack1 wants to merge 1 commit into
kirodotdev:mainfrom
rnoack1:split/drain-gate-queue-held
Closed

rnoack1 wants to merge 1 commit into
kirodotdev:mainfrom
rnoack1:split/drain-gate-queue-held

Conversation

@rnoack1

@rnoack1 rnoack1 commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Carved out of #7161, which is 8.4x over the 1500 added-line cap and is being split so it can
actually be reviewed. This unit is self-contained and does not depend on any other piece.

Typed fix, not refactor. Unifying the two readers is where this started, but the unit also
changes runtime behaviour, so every such change is enumerated under "What changed" below rather
than left to the code comments to explain.

Problem / Motivation

chat_orchestrator and chat_runner each decided separately whether a slot's queue still held
work, so the two could disagree about when a drain was finished.

That hand-derivation is also the shape of a worse defect: a queue held back for one reason is
drained anyway by a path that was never taught to consult the hold, which burns the queued
prompts into the very failure they were held back from.

Why it matters

The two readers are what a drain waits on. When they disagree, the wait either ends while the
queue still holds work or keeps waiting after it is empty, and which one happens depends on the
order the two are consulted rather than on the queue.

The leak is the sharper half. A signed-out CLI holds the queue at the tail drain, and the
automatic synthesis turn then dequeues the same prompts behind it, reaches the same auth failure,
and loses them. At the merge base will_synthesize does not consult the hold at all; this PR adds
and not slot._queue_held to it.

What changed

Every drain site now reads one published flag, slot._queue_held, instead of deciding
independently. recompute_queue_hold in chat_utils.py derives that flag from all live causes at
the release sites, so a cause that is still live leaves the hold SET. A release that only ever
clears is what turns a still-pending teardown into a queue nothing drains.

Behaviour changes this unit ships, each deliberate:

  • A held queue now also blocks the automatic synthesis dispatch, which previously drained it.
    This changes a pre-existing default. The auth cause is deliberately not carved out of the new
    gate: a carve-out would knowingly keep the leak for the one cause that predates this change.
  • Two new hold causes. A project reset or a conversation discard left deferred at the end of a
    turn now holds the queue. Previously only a signed-out CLI did.
  • A new idle-send gate in api_chat. An idle send to a slot that still owes a deferred
    teardown now lands that teardown under slot._lock and queues behind the hold, instead of
    dispatching ahead of it and answering a conversation that is awaiting discard.
  • The background reset retry now drains the queue it releases once the teardown lands, and its
    liveness was generalised from the reset key to either pending cause. Previously a declined
    discard armed no retry at all, so an accepted entry parked until the user happened to send
    again.
  • Two new refusals on that gate, both reusing codes that already exist. A send whose slot was
    replaced under its key during the teardown await answers 409 session_rebound; a relayed send
    that would otherwise be queued answers 409 remote_turn_busy, because the 200 queue receipt
    precedes the relay_mode mirror and would silently lose the peer's frames. Neither code is new
    public surface: at the merge base session_rebound appears 17x and remote_turn_busy 1x in
    chat_handlers.py, and this PR adds one further emit site for each.
  • A new slot field _queue_held_auth records the sign-out cause, which is the one cause no
    release site can re-derive from slot state once the turn has ended.

Deliberately unchanged: subagent and cron injections still deliver past a held prompt, and
remote-bound sends do not enter the new gate at all.

The cross-cutting coverage census in test/test_chat_runner_coverage.py asserts on this unit's
symbol and on several others from the parent change in a single block, so it is deliberately left
with the parent rather than split down the middle. This unit carries its own targeted tests instead.

Tests

Cut off the current main tip, not off the parent branch. Every file except one was modified
upstream since the parent's merge base, so the change was applied hunk-wise with git apply -3
rather than by checking files out at the older revision, which would have silently reverted
upstream work.

  • the synthesis hold is pinned by test_a_signed_out_cli_also_holds_the_queue_against_synthesis,
    which drives the real AcpAuthRequired path, against
    test_synthesis_still_dispatches_when_nothing_is_held as the positive control that an unheld
    cycle still synthesises
  • each new guard carries a mutation control: the guard is reverted, the named test is confirmed
    RED, and the source file is restored byte-identically afterwards
  • 1379 tests pass across the seven suites this unit touches
  • the repo's baselined black gate passes, and the full local run matches the inherited failure
    baseline exactly, with no failing file in common with this diff

1123 added lines and 38 removed, across 12 files.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 15, 2026
@dwu96

dwu96 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@rnoack1
rnoack1 marked this pull request as ready for review September 15, 2026 12:00
@rnoack1
rnoack1 requested review from a team and smeyffret September 15, 2026 12:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 15, 2026
@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 72aba27e77d07153d4c0add8f86d28d439df4412 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the idle-send gate's await _consume_pending_reset under slot._lock is bounded — a stalled ACP reset/discard would wedge every send to that slot.

Suggestions

  • recompute_queue_hold's own docstring proves _queue_held is fully derivable from _queue_held_auth + the two pending keys; making it a read-only property would delete the helper, the finally publish, and the entire missed-release-site hazard class the comments defend against — worth a follow-up.

[DESIGN-REVIEWED] 72aba27

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 72aba27e77d07153d4c0add8f86d28d439df4412 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/dashboard/chat_orchestrator.py:417 -- "not slot._queue_held" bypasses the selective drain, leaving queued cron/subagent injections undelivered until another send -> Fix: remove the outer hold checks and let _start_next_queued_turn filter user prompts.

[GPT-REVIEWED] 72aba27

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 72aba27e77d07153d4c0add8f86d28d439df4412 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the author's self-reported mutation controls: the new hold tests must be red on base — they are the two new hold causes' only checkable provenance.

What this change ships

Inventory (8 items) — 8 justified

Intent: FIX — stop queued prompts being burned when a drain path never consults why the queue was held; every drain now reads one published flag. (Verified on base: will_synthesize at chat_runner.py:7824 has no hold check; all 8 drain callers route through the now-gated _start_next_queued_turn, so no unfixed siblings.)

  1. A signed-out CLI's hold now also stops the automatic synthesis turn, which previously drained the held prompts into the same failure — justified
  2. A project reset left deferred at end of turn now holds the queue — justified
  3. A conversation discard left deferred now holds the queue — justified
  4. An idle send to a slot owing a teardown now lands it and queues behind the hold instead of dispatching ahead — justified
  5. A declined discard now arms the background retry, which drains the queue it releases — justified
  6. A send whose slot was replaced during teardown gets 409 session_rebound instead of a stranded 200 — justified
  7. A relayed send that would queue gets 409 remote_turn_busy; verified _peer_turn_chunks raises a resend prompt on non-2xx — justified
  8. Hold flag renamed to _queue_held, derived in one helper; _queue_held_auth records the one non-re-derivable cause — justified

Items 2–7 are the same root cause's sibling instances and release machinery, all declared in the description — the general fix, not riders. Counts checked: _last_turn_auth_required 6 refs/3 files on base; remote_turn_busy 1 occurrence in chat_handlers.py; session_rebound 20 occurrences (author says 17 emits — comments account for the gap).

[FIRST-PRINCIPLES-REVIEWED] 72aba27

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 72aba27e77d07153d4c0add8f86d28d439df4412 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

The sole candidate (chat_runner.py except Exception in _consume_pending_reset not arming a retry) does not survive falsification. It requires discard_conversation to raise — but the normal deferral outcome (turn in flight / sub-agents attached) returns False, and those two branches already arm the retry in this diff. A raise is an error-only path (e.g. provider.shutdown() failing), which the session pops the session before, so where it can occur the conversation has already been discarded and the armed flag is merely stale — not the "prompt runs against an undiscarded conversation" harm claimed. The held-queue state is also self-healing: any subsequent send (dashboard api_chat gate) or channel message (a fresh turn whose end-of-turn consume re-runs discard_conversation, now returning True on the already-gone session) clears the flag and drains. Requirement (a) is "if teardown errors," not a concrete input that occurs in practice, and (c) is a transient, self-healing hold rather than an observable permanent loss. The candidate's own confidence is "low," and nothing I opened raised it to the 80+ bar.

[OPUS-REVIEWED] 72aba27

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Sep 15, 2026
@rnoack1
rnoack1 force-pushed the split/drain-gate-queue-held branch from 83a3449 to ddc92e6 Compare September 15, 2026 23:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 15, 2026
@rnoack1
rnoack1 force-pushed the split/drain-gate-queue-held branch from ddc92e6 to 8e67345 Compare September 16, 2026 01:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 16, 2026
@rnoack1
rnoack1 force-pushed the split/drain-gate-queue-held branch from 8e67345 to 869d1d6 Compare September 16, 2026 02:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 16, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 16, 2026
@rnoack1
rnoack1 force-pushed the split/drain-gate-queue-held branch from d65eb82 to cf7dbb2 Compare September 16, 2026 11:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 16, 2026
@rnoack1
rnoack1 force-pushed the split/drain-gate-queue-held branch from cf7dbb2 to 9fc4357 Compare September 16, 2026 12:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 16, 2026
@rnoack1
rnoack1 force-pushed the split/drain-gate-queue-held branch from 9fc4357 to 12fc6ce Compare September 16, 2026 14:07
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 16, 2026
@rnoack1 rnoack1 changed the title refactor(chat): route the drain wait through one queue-held gate fix(chat): route every queue drain through one queue-held gate Sep 19, 2026
@dwu96 dwu96 added needs-pr-triage PR scanner: awaiting automated triage and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 22, 2026
@dwu96

dwu96 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: dwu96#8c08bee0]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: No review lane blocks this PR — Design and First Principles are advisory CONCERNS, and GPT 5.6 and Opus 4.8 both report no blocking findings. The binding blocker is the merge conflict against current main, which needs a rebase. The only failing test, test/metrics/test_process_gauges.py::test_peak_rss_at_least_current_rss, is environmental: it failed by ~389 KB (2003644416 vs 2004033536) on an assertion its own docstring describes as comparing gauges that "use different approximate per-CPU accounting paths", and this PR touches no metrics or process-gauge code — so the rebase's fresh CI run should clear it. Coverage Gate and PR Readiness are cascading from that one shard. Separately, all three advisory lanes converge on the same honest point: the description sells a reader-consolidation refactor while the diff also ships two new public 409 codes (session_rebound, remote_turn_busy), teardown queue-holds, and a deliberately changed synthesis default. I'll address that by declaring those behavior changes in the description — I will not split the PR or alter its approach; if splitting turns out to be required, I'll stop and hand it back to you.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

The orchestrator and the runner each decided separately whether a slot's queue still held
work, so the two could disagree about when a drain was finished.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: the new idle-send gate can leave a chat queuing every send forever after a project change

If a chat's project changes while no session is registered under its key, every later send is answered 200 {"queued": true} and never runs.

How it happens (at f146acd):

  • api_chat_slot_project sets slot._pending_reset_history_key = session_key on every project change, whether or not a session exists (src/kiro_crew/dashboard/chat_handlers.py:10008).
  • SessionLifecycle.reset ends with return session is not None (src/kiro_crew/session_lifecycle.py:978). For a key with no session it returns False on every call.
  • _consume_pending_reset then leaves the flag set and starts the 5 s retry (chat_runner.py:5448-5460). _eager_spawn does nothing while the flag is set (chat_runner.py:5930-5939).
  • The new gate (chat_handlers.py:1022-1070) finds the teardown still pending and queues the send. recompute_queue_hold(slot) returns True, so nothing drains:
    _would_queue = _teardown_pending or slot._queue or _busy
    ...
    if not _busy and not recompute_queue_hold(slot):
        await _start_next_queued_turn(state, slot)
    return web.json_response({"ok": True, "queued": True, "queue_id": qid})
  • _release_and_drain only runs once _teardown_owed() is False, which never happens here. hold_users also includes slot._queue_held (chat_runner.py:7454-7459).

At the merge base and on main, the same send starts _run_chat. Its get_or_create creates the session in the new project, and the end-of-turn consume clears the flag. The comment "Leaving it armed is always safe: the next consume lands it" (chat_runner.py:5416-5417) depends on that turn running. The new gate stops it from running.

Who hits it:

  • Anyone who starts a new chat after picking a project on the welcome screen. ChatPage.tsx:2694-2698 creates the slot, calls chatSlotProject, then sends. The first message never runs and no error is shown.
  • Anyone who changes the project of a chat whose session was idle-reaped (default timeout 3600 s), or after a gateway restart, or with session.eager_spawn: false.

The only way out is deleting the chat or restarting the gateway. From reading the code merged with main, the queued card's "Run now" (PR #12833) also hits hold_users and returns 409 queue_item_unavailable.

Independent checkers reproduced this with the real SessionManager.reset on an empty registry:

  • On this PR's head, and on it merged onto main, two sends were both queued and neither ran.
  • On the merge base and on main, both were dispatched.

This PR's tests only cover a decline that later succeeds (side_effect=[False, True]) or a busy session.

Fix direction: when the only thing owed is the reset and state.sessions.has_session(key) is False, let the gate fall through to dispatch. Leave the flag set so the turn's end-of-turn consume clears it, as it does on main. PR #11831 and PR #11983 add a has_session(pending_key) clear, but this PR says it depends on no other piece. Please add a test where reset keeps returning False for a key with no session.

From a full open-PR audit against origin/main c66f95b (2026-09-25); findings were checked by independent reviewers reading the code. Reply here if this is wrong.

Copy link
Copy Markdown
Contributor

Triage: rebase needed, and coordinate with #11831 / #11983.

Checked against origin/main a92dde1 (2026-10-05). git merge-tree reports conflicts in chat_handlers.py, chat_runner.py, chat_utils.py, state.py, test/test_dashboard_chat.py, plus a modify/delete conflict:

  • src/kiro_crew/dashboard/chat_orchestrator.py was deleted on main by chore: remove Autopilot plan/stage loop #15362 (d2db310, "remove Autopilot plan/stage loop"). Both of this PR's _exit_cancelled_plan / _stage_loop gate edits now have no target.
  • fix(chat): hand off the queue on every _run_chat exit #16279 (98f2b96) rewrote the _run_chat exit handoff. The tail drain is now drain=memory_admitted and not slot._last_turn_auth_required and not setup_failed (chat_runner.py:7792), and chat_done_payload takes a queue_held parameter (chat_utils.py:1350). So _queue_held has to be merged into these existing seams, not added next to them.

Main still does not cover the core fix. _finish_queue_cycle on main still computes will_synthesize without checking any hold (chat_runner.py:7420-7430). The discard-decline branches still don't arm _arm_pending_reset_retry. This PR is still needed after the rebase. It is not superseded.

Overlap: #11831 and #11983 (both open) edit the same _consume_pending_reset reset-decline branch and add the has_session(pending_key) clear. That clear is the fix direction @bolichen97's CHANGES_REQUESTED review gives for the queue-forever P0 on this PR. Please agree with those authors on which PR owns that clear before you rebase, or carry it here with the missing test where reset keeps returning False for a key with no session.

From an automated open-PR audit; reply if this is wrong.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you for this work. Main has moved a lot since this was opened: src/kiro_crew/dashboard/chat_orchestrator.py was removed in #15362, and the queue-cycle logic now lives in dashboard/chat_turn/queue_cycle.py with its own hold rules. The parent PR #7161 was also closed as superseded, and there has been no activity here since 2026-09-22. I'm closing this one. If the unified gate is still needed, a new PR built on the current queue_cycle.py would be the way forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants