Skip to content

fix(dashboard): abort an SSE chat stream whose client stops reading - #18993

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/sse-stalled-reader-is-aborted
Oct 12, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/sse-stalled-reader-is-aborted

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: End an SSE chat stream whose client has stopped reading, so the stream stops holding its slot's pending queue and broadcast.

api_chat's SSE loop (dashboard/chat_handlers.py) and the OpenAI-compatible stream (openai_compat._stream_response) send every frame with await resp.write(...), which has no deadline. Once a client stops reading without closing its socket, aiohttp parks the write in drain and the loop never ends. While it waits, the loop keeps holding the slot:

  • Every row the turn streams stays in slot._pending. The turn-end release of chunk rows (release_pending_chunks) is refused, because the parked loop still holds pending_consumer(), the guard fix(dashboard): release streamed chunk rows the window purge left behind #4704 added so that a live reader never loses rows.
  • For api_chat, the reader also holds _has_reader, which suppresses the slot's broadcast to other tabs. A row added after the turn (a cron or sub-agent notice) is never pushed. The handler's own disconnect path discards it later.
  • Each later turn on the slot is held in full the same way. The pre-dispatch drain discards the previous turn's rows, and the new turn's release is refused again.

Measured on main with aiohttp's test server and the real api_chat handler, against a raw socket that sends its request and never reads. With a 4,000-chunk turn (about 4 MB), 2,372,102 B stay in _pending and _has_reader stays True. A row added after the turn is not broadcast within 10 s. A second turn on the slot, sent from a dashboard tab, is held whole (4,120,012 B). A reader that keeps reading holds 0 B.

Who meets it: the dashboard's own tabs send with ?ws=1 and use the WebSocket path, so they never take these streams. This affects SSE callers of /api/chat (the app SDK's ChatEmbed, API clients) and of /v1/chat/completions.

Why it matters

One stalled client keeps a whole turn's streamed output in gateway memory for as long as its socket stays open. Measured: 14.7 MB after a 16,000-chunk turn, and 16.5 MB after the next one. For /api/chat it also stops every other tab on that slot from receiving live rows, and the rows added during the hold are never pushed. Nothing times the hold out. A peer that stays connected and never reads keeps it until the gateway restarts.

Not a goal

  • No cap on slot._pending while a consumer holds it. A reader that is only slow would lose rows from the middle of its answer, where the deadline ends the stream visibly.
  • The other dashboard SSE endpoints are left as they are. Each was checked under the same stalled reader: /api/stream and /api/logs use bounded per-client queues that drop on overflow, and /api/file-watch and the sub-agent output stream hold only their own request.
  • The WebSocket push channel (websocket_hub.py) has its own slow-client findings, handled separately.
  • A client cut off by the deadline loses the rest of that stream, exactly as it does when it disconnects. The turn still runs to completion, and its transcript keeps every row.

What changed (motivation → approach → change)

  • Motivation: a write with no deadline turns "the client stopped reading" into "this stream holds the slot forever".
  • Approach: treat a write that cannot finish within a deadline as a disconnect, and reuse the disconnect handling both loops already have.
  • Change:
    • chat_handlers.py: a new write_sse_or_abort(request, resp, data) awaits resp.write under _SSE_WRITE_DEADLINE_SECS = 120.0. Past the deadline it calls request.transport.abort() and raises ConnectionResetError. Every write in api_chat's SSE loop goes through it: frames, [DONE] and the keepalive. The loop's existing except/finally then drains _pending, clears _has_reader (so the broadcast resumes) and leaves pending_consumer(), which retries the deferred release. Aborting the transport also makes aiohttp's write_eof after the handler fail fast instead of waiting on the full socket.
    • openai_compat.py: every write in _stream_response and _stream_server_error goes through the same helper (_stream_server_error now takes the request). Its existing except ConnectionResetError / finally and the caller's pending_consumer() scope release the slot.
    • Why 120 s: aiohttp parks a write once its buffer is full and resumes it when the client has read about 48 KiB of it. A reader that is merely slow finishes every write inside 120 s down to roughly 400 bytes a second, far below any working connection. A write still parked after 120 s means the client is not reading.
  • Tests:
    • test/test_dashboard_chat.py::TestApiChatStalledReader::test_a_stalled_reader_is_cut_off_and_releases_the_slot: a raw socket with SO_RCVBUF=4096 posts and never reads, through an 8,000-chunk turn, with the deadline at 0.5 s. It asserts that the reader is released, no chunk rows are held, a later row is broadcast, and the next (ws=1) turn's chunks are released.
    • test_a_reading_client_gets_the_whole_turn (control): a client that reads gets the whole turn ending in [DONE], under the default deadline.
    • test/test_openai_compat.py::test_a_stream_write_the_client_never_drains_aborts_the_connection: a stream write that never completes. It asserts that the request returns and the transport is aborted once.

Backwards compatibility

Compatible: a reader that keeps reading sees the same frames. Only a write parked for 120 s ends the stream, which closes the connection the same way a client disconnect does.
Removes nothing: no route, field, config key or test is removed. _stream_server_error is module-private and gains a request parameter; its two callers are updated.

Tests

Red: the branch's two test files copied onto main 843abe3, so the new tests run against main's chat_handlers.py and openai_compat.py (-n 0, three runs, the same result each time):

E       AssertionError: a stalled SSE reader still holds its slot 10 s after the turn: pending_consumers=1, _has_reader=True, 6303 chunk rows (6492090 bytes) held, a later row broadcast to other tabs=False; after the next turn 8000 chunk rows (8240000 bytes) stay held
E       AssertionError: a stream write the client never drains keeps the request open after 10 s; the connection is never aborted
=========== 2 failed, 1 passed, 943 deselected, 7 warnings in 27.05s ===========

The first failure pins the /api/chat hold: the stalled reader keeps pending_consumer() and _has_reader, so the turn's rows stay in _pending and a later row is not broadcast. The second pins the OpenAI-compatible stream: its request never returns and its connection is never aborted. The control passes on both sides. Green on this branch, both files whole (-n 0, three runs):

946 passed

Neighbour files, in one serial -n 0 run: the four that cover the two handlers (test/test_dashboard_chat.py, test/test_openai_compat.py, test/test_chat_handlers_composition_contract.py, test/test_dashboard_chat_handlers_coverage.py), plus every test file that imports chat_handlers and names api_chat, or imports openai_compat and names _stream_response or api_completions (test_chat_mode_security.py, test_crew_recency.py, test_members_dm_thread.py, test_pruned_crewmate_records.py, test_queue_drain_preserve_attachments.py, test_relay_archive.py, test_slot_chunk_retention.py, test_slot_create_default_agent.py, test_slot_ownership_checkpoint.py, test_unattended_slot_guardrails.py). The run exits 0 across all 14, with 1,906 passed and none failed. The largest are test_dashboard_chat.py (905), test_chat_mode_security.py (226), test_members_dm_thread.py (191), test_slot_ownership_checkpoint.py (130), test_dashboard_chat_handlers_coverage.py (123) and test_chat_handlers_composition_contract.py (121).

Static gates on the commit: mypy, flake8, isort, black, scripts/check_black_formatting.py, COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py, ./scripts/docs-lint.sh, scripts/check_memory_store_seam.py and scripts/check_harness_parity.py all exit 0.

Manual verification

  • A probe beside the reproducer measured what a stalled reader holds across turns, on main. Turn 1 (4,000 chunks) left 2,372,102 B. Turn 2 (3,000 chunks) left 3,090,012 B, turn 3 (1,000) left 1,030,012 B, and 0 B remained after the client's RST. pending_consumers=1 and release_deferred=True held after every turn. So each turn is held whole and replaces the previous one.
  • Overlap: 72 open PRs touch the two files, and six have hunks in the lines this change edits. For each of the six, git merge-tree of the PR head was run against this commit and against its base, and the conflict blocks in the two files were counted. Related Issues lists where this commit adds one. No open PR changes api_chat's SSE loop.

Related Issues

Related: #4704 added the pending_consumer() guard that defers the chunk release while a reader drains the queue. This change keeps that guard and ends the reader that no longer drains. No open issue or PR reports a stalled SSE reader (searched pending_consumer, _has_reader, slot._pending, release_pending_chunks, "SSE slow reader", "event-stream backpressure", "stalled SSE").

Overlapping open PRs (conflict blocks from git merge-tree, each PR head against this commit compared with against its base):

Pattern harvest

Rule candidate: review-prompt
Pattern: a streaming handler awaits a network write with no deadline while it holds shared state (a queue, a flag, a consumer count) that other code waits on.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): not applicable
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 00:49
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 75c2975627d03ad75008cf37381a6fb3ba7412b8 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 75c2975627d03ad75008cf37381a6fb3ba7412b8 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the 120 s write deadline never cuts off a slow but live ChatEmbed or API client behind a buffering proxy.

Suggestions

  • Put write_sse_or_abort in its own small dashboard SSE module rather than at the bottom of the 11k-line chat_handlers.py. That way openai_compat doesn't have to import the whole chat handler module for a single transport helper.

[DESIGN-REVIEWED] 75c2975

Description read

[DESCRIPTION-READ] 1c69bab8f45a9632c45736b5d92ec2955c85e7cb24dafad25416853921b48331

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 75c2975627d03ad75008cf37381a6fb3ba7412b8 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check whether a ChatEmbed or API client that pauses reading for more than 120 s (for example a suspended mobile tab) should lose the rest of its answer.

What this change ships

Inventory (6 items) — 6 justified

Intent: stop one client that has stopped reading from holding a chat slot's queued output and its broadcast to other tabs indefinitely. This is a FIX, reproduced by tests that fail on base.

  1. An /api/chat SSE stream whose client stops reading is cut off after 120 s, which frees the slot's pending rows and turns its broadcast back on — justified
  2. A /v1/chat/completions stream whose client stops reading is cut off the same way — justified
  3. A 120 s write deadline, applied to every SSE frame, [DONE] and keepalive on both paths — justified
  4. A new helper, write_sse_or_abort, used by both handlers (2 consumers; the existing transport aborts in apps/routes.py:3575 and :4616 end relays, so it is not a second spelling) — justified
  5. _stream_server_error now takes the request, and both of its callers are updated — justified
  6. An info log line when a stalled client is aborted — justified

[FIRST-PRINCIPLES-REVIEWED] 75c2975

Description read

[DESCRIPTION-READ] ccd16ee64f87726aaf82408b36b7c216b28915aa5438f5db26399b8f8c497616

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 75c2975627d03ad75008cf37381a6fb3ba7412b8 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 75c2975

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 75c2975627d03ad75008cf37381a6fb3ba7412b8 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 75c2975

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@javenciu
javenciu force-pushed the fix/sse-stalled-reader-is-aborted branch 2 times, most recently from d364087 to 21cf468 Compare October 12, 2026 11:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
api_chat's SSE loop and the OpenAI-compatible stream await resp.write with
no deadline. Once a client stops reading without closing its socket, the
write parks in aiohttp's drain and the loop never ends. While it waits, the
loop still holds the slot: every row the turn streams stays in
slot._pending, the turn-end release of chunk rows is refused because the
reader holds pending_consumer(), and for api_chat the slot's live rows are
not broadcast to other tabs because the reader holds _has_reader. Each
later turn on that slot is held in full the same way.

Every write in both streams now goes through write_sse_or_abort, which
gives the write 120 seconds. Past that it aborts the transport and raises
ConnectionResetError, so the existing disconnect handling releases the
slot: the pending queue is drained, the broadcast resumes, and the
deferred release runs.
@javenciu
javenciu force-pushed the fix/sse-stalled-reader-is-aborted branch from 21cf468 to 75c2975 Compare October 12, 2026 14:50
@javenciu

Copy link
Copy Markdown
Contributor Author

Folded into the existing commit: the cleanup asyncio.wait_for ceiling at test/test_openai_compat.py:1044 is 30 s instead of 10 s, and the 200-turn asyncio.sleep(0.05) poll for slot._pending_consumers == 0 at test/test_dashboard_chat.py:26306 now waits through kiro_crew.testing.wait.async_wait_until (its failure text no longer claims 10 s).
test/test_wait_discipline_ratchet.py counted both files as gaining short fixed waits (22 to 23 and 0 to 1); it passes on the head merged with main (c4c51aa), and both files pass 3 of 3 runs (946 tests per run).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bounds every SSE write in api_chat's stream loop and in openai_compat._stream_response / _stream_server_error with write_sse_or_abort (a 120 s deadline). Past the deadline it aborts the transport and raises ConnectionResetError, so the loops' existing disconnect handling releases _pending, _has_reader and pending_consumer(). That fixes the root cause: an un-deadlined resp.write let a client that stops reading hold its slot forever. I checked the side effects. On Python 3.12 (requires-python >=3.12), wait_for adds no task per write. A cancelled write followed by transport.abort() makes aiohttp's later write_eof fail fast. There is no import cycle (chat_handlers does not import openai_compat). A reader that keeps reading sees identical frames, which the control test pins. One edge to be aware of, not a blocker: one very large frame needs the client to drain close to that frame's size within 120 s, not just 48 KiB. That is still far below a working connection. The raw-socket regression test shows _pending, the broadcast and the next turn's release all recover. The OpenAI test proves the abort happens. Checks are green and both AI lanes pass. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 12, 2026 17:19
@iamwhatever
iamwhatever merged commit 006c69c into kirodotdev:main Oct 12, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants