Repository navigation
perf(dashboard): serialize the slot list once per broadcast, not three times - #12711
Conversation
|
Intent: Stop the slots broadcast from serializing every open slot three times per push on the event loop, so a sidebar update no longer stalls the loop for hundreds of milliseconds and the adaptive concurrency controller stops reading that stall as host pressure. |
24adcbf to
bc079cd
Compare
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Post-fix loop-lag re-measurement is still pending ("needs a restart") — confirm the promised number lands on the PR before merge. What this change shipsInventory (5 items) — 5 justifiedIntent: FIX — stop sidebar broadcasts from stalling the event loop, which the adaptive controller misread as host pressure and pinned the subagent cap at 1 (#12710, with measured probe and py-spy evidence).
Sibling count: grepped [FIRST-PRINCIPLES-REVIEWED] 506fb48 |
Design Review (Fable 5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS One pass is still ~200 ms on the loop at 80 tabs — 80% of the 250 ms decrease signal, and the post-fix measurement is still pending. WatchThe fix removes the 3× redundancy but keeps O(n) regex-heavy serialization synchronous on the event loop: the description's own numbers ("one pass is already 192–211 ms", "loop_lag >= 250 ms is a sufficient-alone decrease signal") put the remaining single pass within noise of the controller threshold, so the cap-pinning symptom the PR exists to fix can recur on a modestly larger sidebar, and the description defers proof ("Post-fix loop-lag measurement … will be reported on the PR once done"). Suggestions
[DESIGN-REVIEWED] 506fb48 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
|
Rebased: main moved in import-only overlap files (agent.py, handlers/_shared.py, decisions/*, mcp_discovery.py, vector_memory.py) since this head went green (b2bc03a -> 0a80050); CI re-running. The shard-4 red ( |
bc079cd to
a853ed0
Compare
a853ed0 to
97e2111
Compare
|
Re-pushed 97e2111 as e9de673 (identical tree, amended timestamp only): the four fork lanes on the previous head (GPT 5.6 Review, Opus 4.8 Review, Security Scope Review, Internal Content Scan) failed before producing a verdict on |
97e2111 to
e9de673
Compare
…e times Every slots push serialized every open slot three times -- a bare list for SSE, a dashboard-user list for the WS frame, and an owner list -- each pass re-running the per-slot projection body (markdown strip, credential redaction, options parse) and source-link budgeting, synchronously on the event loop. The three views differ only in each slot's source_links field. Serialize once and re-project that one field per audience. The repeated pass stalled the loop for 0.4-1.1 s per broadcast on a large sidebar; the adaptive concurrency controller read those stalls as loop_lag pressure and kept halving the subagent cap to its floor on an idle host.
e9de673 to
506fb48
Compare
|
Rebased: main moved in import-overlap files (adaptive/controller.py, config/sections.py, eventlog/service.py, subagent.py and tests) since this head (-> 1256a48); CI re-running. No code change. |
|
|
|
bolichen97
left a comment
There was a problem hiding this comment.
Approved after verified readiness: passed, PR Readiness success on this head, mergeable, no open change requests. [operator: bolichen97 batch review 2026-09-23]
…e times (kirodotdev#12711) Every slots push serialized every open slot three times -- a bare list for SSE, a dashboard-user list for the WS frame, and an owner list -- each pass re-running the per-slot projection body (markdown strip, credential redaction, options parse) and source-link budgeting, synchronously on the event loop. The three views differ only in each slot's source_links field. Serialize once and re-project that one field per audience. The repeated pass stalled the loop for 0.4-1.1 s per broadcast on a large sidebar; the adaptive concurrency controller read those stalls as loop_lag pressure and kept halving the subagent cap to its floor on an idle host.
Problem / Motivation
The gateway's event loop stalls for 0.4–1.1 s on every sidebar update. Measured on a live gateway with a probe of a 4 ms static route every 200 ms for four minutes: 8 stalls of 415–1131 ms. Nothing else on the host was busy (315 GB free, load 0.01).
The adaptive concurrency controller reads those stalls as host pressure.
loop_lag >= 250 msis a sufficient-alone decrease signal, so it kept halving the subagent cap.resource_statuson that gateway showedExecution cap: 1/6,decrease=19, increase=19after nine hours, and everyspawn_runfan-out queued behind one slot.Why it matters
A user with a large sidebar gets one subagent at a time on a 128-core machine, and the dashboard's own reason for it ("host pressure") is wrong. The stall also hits every request that shares the loop while a broadcast runs.
What changed (motivation → approach → change)
One serialization pass per slots broadcast instead of three.
py-spy on the gateway (
--gil --threads) put ~55 of the main thread's 79 GIL-holding samples in one chain:push_slots_update → _do_slots_broadcast → serialize_slots → serialize_slot → SlotProjection.to_dict → strip_markdown_preview / redact_credentials → regex._do_slots_broadcastinsrc/kiro_crew/dashboard/state.pyserialized every open slot three times per broadcast: a bare list for SSE, adashboard_user=Truelist for the WS frame, aninclude_check_status=Truelist for owner sockets. Each pass re-ran the per-slot projection body (last-message markdown strip, credential redaction, options parse) and the source-link budgeting on the event loop; the source-link transcript scan itself was already memoized per slot revision, so it was not the repeated cost. One pass is already 192–211 ms in theGET /api/chat/slotshistogram, so a broadcast cost ~600 ms.The three views differ only in each slot's
source_linksfield.include_check_statusanddashboard_userreachSlotProjection.to_dictsolely throughproject_source_links; nothing else in the summary reads them.So the broadcast now calls
serialize_slots()once and derives the other two lists with_reproject_slots, which shallow-copies each payload and replaces onlysource_linksvia the new_ChatSlot.source_links_view. The owner list is built only when an owner socket is registered.serialize_slot_viewsis the seam the broadcast calls;serialize_slotsitself is unchanged, sochat_handlers.py,ws.pyand the tests that stub it keep working. A payload whose key has no live slot is carried over unchanged.Backwards compatibility
Compatible: every frame carries byte-for-byte what a full per-audience
serialize_slots(...)pass produced before (pinned by test). No wire, config or API change.Tests
test/test_dashboard_state_ws.py:test_broadcast_serializes_each_slot_once_and_views_match_full_passes— with three slots (public-repo link, private-repo link with[OPTIONS:], no links) asserts_ChatSlot.to_dictruns once per slot per broadcast, that the derived bare / dashboard-user / owner lists equal three independent full passes, that the owner view is skipped when no owner socket exists, and thatpush_slots_updategoes through the single pass. Fails onmain(noserialize_slot_views).test_reproject_carries_over_payloads_without_a_live_slot— a payload with no registered slot passes through untouched.test_public_repo_status_rides_general_frame_owner_gets_full— rewired from a stubbedserialize_slotsto a real slot with a link, since the audience views are no longer produced by callingserialize_slotswith flags; the assertions are unchanged.Manual verification
Diagnosis on the live gateway: HTTP loop-lag probe and two py-spy profiles (
--idle, then--gil --threads --format raw) attributing the stalls to this call chain. Post-fix loop-lag measurement on a running gateway needs a restart and will be reported on the PR once done.Related Issues
Fixes #12710
Pattern harvest
Rule candidate: review-prompt
Pattern: the same per-item projection computed once per audience on the event loop, when the audiences differ in a single field
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)