Repository navigation
fix(dashboard): abort an SSE chat stream whose client stops reading - #18993
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the 120 s write deadline never cuts off a slow but live Suggestions
[DESIGN-REVIEWED] 75c2975 Description read[DESCRIPTION-READ] 1c69bab8f45a9632c45736b5d92ec2955c85e7cb24dafad25416853921b48331 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check whether a ChatEmbed or API client that pauses reading for more than 120 s (for example a suspended mobile tab) should lose the rest of its answer. What this change shipsInventory (6 items) — 6 justifiedIntent: stop one client that has stopped reading from holding a chat slot's queued output and its broadcast to other tabs indefinitely. This is a FIX, reproduced by tests that fail on base.
[FIRST-PRINCIPLES-REVIEWED] 75c2975 Description read[DESCRIPTION-READ] ccd16ee64f87726aaf82408b36b7c216b28915aa5438f5db26399b8f8c497616 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
d364087 to
21cf468
Compare
api_chat's SSE loop and the OpenAI-compatible stream await resp.write with no deadline. Once a client stops reading without closing its socket, the write parks in aiohttp's drain and the loop never ends. While it waits, the loop still holds the slot: every row the turn streams stays in slot._pending, the turn-end release of chunk rows is refused because the reader holds pending_consumer(), and for api_chat the slot's live rows are not broadcast to other tabs because the reader holds _has_reader. Each later turn on that slot is held in full the same way. Every write in both streams now goes through write_sse_or_abort, which gives the write 120 seconds. Past that it aborts the transport and raises ConnectionResetError, so the existing disconnect handling releases the slot: the pending queue is drained, the broadcast resumes, and the deferred release runs.
21cf468 to
75c2975
Compare
|
Folded into the existing commit: the cleanup |
iamwhatever
left a comment
There was a problem hiding this comment.
This bounds every SSE write in api_chat's stream loop and in openai_compat._stream_response / _stream_server_error with write_sse_or_abort (a 120 s deadline). Past the deadline it aborts the transport and raises ConnectionResetError, so the loops' existing disconnect handling releases _pending, _has_reader and pending_consumer(). That fixes the root cause: an un-deadlined resp.write let a client that stops reading hold its slot forever. I checked the side effects. On Python 3.12 (requires-python >=3.12), wait_for adds no task per write. A cancelled write followed by transport.abort() makes aiohttp's later write_eof fail fast. There is no import cycle (chat_handlers does not import openai_compat). A reader that keeps reading sees identical frames, which the control test pins. One edge to be aware of, not a blocker: one very large frame needs the client to drain close to that frame's size within 120 s, not just 48 KiB. That is still far below a working connection. The raw-socket regression test shows _pending, the broadcast and the next turn's release all recover. The OpenAI test proves the abort happens. Checks are green and both AI lanes pass. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: End an SSE chat stream whose client has stopped reading, so the stream stops holding its slot's pending queue and broadcast.
api_chat's SSE loop (dashboard/chat_handlers.py) and the OpenAI-compatible stream (openai_compat._stream_response) send every frame withawait resp.write(...), which has no deadline. Once a client stops reading without closing its socket, aiohttp parks the write in drain and the loop never ends. While it waits, the loop keeps holding the slot:slot._pending. The turn-end release of chunk rows (release_pending_chunks) is refused, because the parked loop still holdspending_consumer(), the guard fix(dashboard): release streamed chunk rows the window purge left behind #4704 added so that a live reader never loses rows.api_chat, the reader also holds_has_reader, which suppresses the slot's broadcast to other tabs. A row added after the turn (a cron or sub-agent notice) is never pushed. The handler's own disconnect path discards it later.Measured on main with aiohttp's test server and the real
api_chathandler, against a raw socket that sends its request and never reads. With a 4,000-chunk turn (about 4 MB), 2,372,102 B stay in_pendingand_has_readerstays True. A row added after the turn is not broadcast within 10 s. A second turn on the slot, sent from a dashboard tab, is held whole (4,120,012 B). A reader that keeps reading holds 0 B.Who meets it: the dashboard's own tabs send with
?ws=1and use the WebSocket path, so they never take these streams. This affects SSE callers of/api/chat(the app SDK'sChatEmbed, API clients) and of/v1/chat/completions.Why it matters
One stalled client keeps a whole turn's streamed output in gateway memory for as long as its socket stays open. Measured: 14.7 MB after a 16,000-chunk turn, and 16.5 MB after the next one. For
/api/chatit also stops every other tab on that slot from receiving live rows, and the rows added during the hold are never pushed. Nothing times the hold out. A peer that stays connected and never reads keeps it until the gateway restarts.Not a goal
slot._pendingwhile a consumer holds it. A reader that is only slow would lose rows from the middle of its answer, where the deadline ends the stream visibly./api/streamand/api/logsuse bounded per-client queues that drop on overflow, and/api/file-watchand the sub-agent output stream hold only their own request.websocket_hub.py) has its own slow-client findings, handled separately.What changed (motivation → approach → change)
chat_handlers.py: a newwrite_sse_or_abort(request, resp, data)awaitsresp.writeunder_SSE_WRITE_DEADLINE_SECS = 120.0. Past the deadline it callsrequest.transport.abort()and raisesConnectionResetError. Every write inapi_chat's SSE loop goes through it: frames,[DONE]and the keepalive. The loop's existingexcept/finallythen drains_pending, clears_has_reader(so the broadcast resumes) and leavespending_consumer(), which retries the deferred release. Aborting the transport also makes aiohttp'swrite_eofafter the handler fail fast instead of waiting on the full socket.openai_compat.py: every write in_stream_responseand_stream_server_errorgoes through the same helper (_stream_server_errornow takes the request). Its existingexcept ConnectionResetError/finallyand the caller'spending_consumer()scope release the slot.test/test_dashboard_chat.py::TestApiChatStalledReader::test_a_stalled_reader_is_cut_off_and_releases_the_slot: a raw socket withSO_RCVBUF=4096posts and never reads, through an 8,000-chunk turn, with the deadline at 0.5 s. It asserts that the reader is released, no chunk rows are held, a later row is broadcast, and the next (ws=1) turn's chunks are released.test_a_reading_client_gets_the_whole_turn(control): a client that reads gets the whole turn ending in[DONE], under the default deadline.test/test_openai_compat.py::test_a_stream_write_the_client_never_drains_aborts_the_connection: a stream write that never completes. It asserts that the request returns and the transport is aborted once.Backwards compatibility
Compatible: a reader that keeps reading sees the same frames. Only a write parked for 120 s ends the stream, which closes the connection the same way a client disconnect does.
Removes nothing: no route, field, config key or test is removed.
_stream_server_erroris module-private and gains arequestparameter; its two callers are updated.Tests
Red: the branch's two test files copied onto main 843abe3, so the new tests run against main's
chat_handlers.pyandopenai_compat.py(-n 0, three runs, the same result each time):The first failure pins the
/api/chathold: the stalled reader keepspending_consumer()and_has_reader, so the turn's rows stay in_pendingand a later row is not broadcast. The second pins the OpenAI-compatible stream: its request never returns and its connection is never aborted. The control passes on both sides. Green on this branch, both files whole (-n 0, three runs):Neighbour files, in one serial
-n 0run: the four that cover the two handlers (test/test_dashboard_chat.py,test/test_openai_compat.py,test/test_chat_handlers_composition_contract.py,test/test_dashboard_chat_handlers_coverage.py), plus every test file that importschat_handlersand namesapi_chat, or importsopenai_compatand names_stream_responseorapi_completions(test_chat_mode_security.py,test_crew_recency.py,test_members_dm_thread.py,test_pruned_crewmate_records.py,test_queue_drain_preserve_attachments.py,test_relay_archive.py,test_slot_chunk_retention.py,test_slot_create_default_agent.py,test_slot_ownership_checkpoint.py,test_unattended_slot_guardrails.py). The run exits 0 across all 14, with 1,906 passed and none failed. The largest aretest_dashboard_chat.py(905),test_chat_mode_security.py(226),test_members_dm_thread.py(191),test_slot_ownership_checkpoint.py(130),test_dashboard_chat_handlers_coverage.py(123) andtest_chat_handlers_composition_contract.py(121).Static gates on the commit: mypy, flake8, isort, black,
scripts/check_black_formatting.py,COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py,./scripts/docs-lint.sh,scripts/check_memory_store_seam.pyandscripts/check_harness_parity.pyall exit 0.Manual verification
pending_consumers=1andrelease_deferred=Trueheld after every turn. So each turn is held whole and replaces the previous one.git merge-treeof the PR head was run against this commit and against its base, and the conflict blocks in the two files were counted. Related Issues lists where this commit adds one. No open PR changesapi_chat's SSE loop.Related Issues
Related: #4704 added the
pending_consumer()guard that defers the chunk release while a reader drains the queue. This change keeps that guard and ends the reader that no longer drains. No open issue or PR reports a stalled SSE reader (searchedpending_consumer,_has_reader,slot._pending,release_pending_chunks, "SSE slow reader", "event-stream backpressure", "stalled SSE").Overlapping open PRs (conflict blocks from
git merge-tree, each PR head against this commit compared with against its base):_stream_responsethrough a new local_writethat prepares the response lazily. It rewrites the same lines, so this commit adds 4 conflict blocks inopenai_compat.py(6 become 10). Whichever lands second must route_writethroughwrite_sse_or_abort, or that stream loses the deadline._is_recap_noticejust above_stream_server_error, whose signature changes here: 1 new block inopenai_compat.py. Keep both.api_chat.write_sse_or_abortsits at the end ofchat_handlers.py, where neither PR edits, so this commit adds no conflict to either. fix(chat): build and file nothing for a create that is refused #15592 still merges cleanly, and fix(dashboard): let an empty chat change member until its first send #14702 keeps exactly the conflicts it has with main.openai_compat.pyimport block; the import added here joins that block and adds no new one.Pattern harvest
Rule candidate: review-prompt
Pattern: a streaming handler awaits a network write with no deadline while it holds shared state (a queue, a flag, a consumer count) that other code waits on.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)