Skip to content

perf(dashboard): serialize the slot list once per broadcast, not three times - #12711

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:perf/slots-broadcast-single-pass
Sep 23, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:perf/slots-broadcast-single-pass

Conversation

@Pearcekieser

Copy link
Copy Markdown
Contributor

Problem / Motivation

The gateway's event loop stalls for 0.4–1.1 s on every sidebar update. Measured on a live gateway with a probe of a 4 ms static route every 200 ms for four minutes: 8 stalls of 415–1131 ms. Nothing else on the host was busy (315 GB free, load 0.01).

The adaptive concurrency controller reads those stalls as host pressure. loop_lag >= 250 ms is a sufficient-alone decrease signal, so it kept halving the subagent cap. resource_status on that gateway showed Execution cap: 1/6, decrease=19, increase=19 after nine hours, and every spawn_run fan-out queued behind one slot.

Why it matters

A user with a large sidebar gets one subagent at a time on a 128-core machine, and the dashboard's own reason for it ("host pressure") is wrong. The stall also hits every request that shares the loop while a broadcast runs.

What changed (motivation → approach → change)

One serialization pass per slots broadcast instead of three.

py-spy on the gateway (--gil --threads) put ~55 of the main thread's 79 GIL-holding samples in one chain: push_slots_update → _do_slots_broadcast → serialize_slots → serialize_slot → SlotProjection.to_dict → strip_markdown_preview / redact_credentials → regex. _do_slots_broadcast in src/kiro_crew/dashboard/state.py serialized every open slot three times per broadcast: a bare list for SSE, a dashboard_user=True list for the WS frame, an include_check_status=True list for owner sockets. Each pass re-ran the per-slot projection body (last-message markdown strip, credential redaction, options parse) and the source-link budgeting on the event loop; the source-link transcript scan itself was already memoized per slot revision, so it was not the repeated cost. One pass is already 192–211 ms in the GET /api/chat/slots histogram, so a broadcast cost ~600 ms.

The three views differ only in each slot's source_links field. include_check_status and dashboard_user reach SlotProjection.to_dict solely through project_source_links; nothing else in the summary reads them.

So the broadcast now calls serialize_slots() once and derives the other two lists with _reproject_slots, which shallow-copies each payload and replaces only source_links via the new _ChatSlot.source_links_view. The owner list is built only when an owner socket is registered. serialize_slot_views is the seam the broadcast calls; serialize_slots itself is unchanged, so chat_handlers.py, ws.py and the tests that stub it keep working. A payload whose key has no live slot is carried over unchanged.

Backwards compatibility

Compatible: every frame carries byte-for-byte what a full per-audience serialize_slots(...) pass produced before (pinned by test). No wire, config or API change.

Tests

test/test_dashboard_state_ws.py:

  • test_broadcast_serializes_each_slot_once_and_views_match_full_passes — with three slots (public-repo link, private-repo link with [OPTIONS:], no links) asserts _ChatSlot.to_dict runs once per slot per broadcast, that the derived bare / dashboard-user / owner lists equal three independent full passes, that the owner view is skipped when no owner socket exists, and that push_slots_update goes through the single pass. Fails on main (no serialize_slot_views).
  • test_reproject_carries_over_payloads_without_a_live_slot — a payload with no registered slot passes through untouched.
  • test_public_repo_status_rides_general_frame_owner_gets_full — rewired from a stubbed serialize_slots to a real slot with a link, since the audience views are no longer produced by calling serialize_slots with flags; the assertions are unchanged.

Manual verification

Diagnosis on the live gateway: HTTP loop-lag probe and two py-spy profiles (--idle, then --gil --threads --format raw) attributing the stalls to this call chain. Post-fix loop-lag measurement on a running gateway needs a restart and will be reported on the PR once done.

Related Issues

Fixes #12710

Pattern harvest

Rule candidate: review-prompt
Pattern: the same per-item projection computed once per audience on the event loop, when the audiences differ in a single field

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and Zedmor September 22, 2026 04:50
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Intent: Stop the slots broadcast from serializing every open slot three times per push on the event loop, so a sidebar update no longer stalls the loop for hundreds of milliseconds and the adaptive concurrency controller stops reading that stall as host pressure.
Not a goal: Moving serialization off the event loop, caching per-slot projections across broadcasts, the Issue Radar poller cost, or adding the missing loop-lag metric / decision-history observability (tracked in #12710).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@Pearcekieser
Pearcekieser force-pushed the perf/slots-broadcast-single-pass branch from 24adcbf to bc079cd Compare September 22, 2026 05:04
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 506fb48248c5781d9df43169e79c298e274491f3 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 506fb48

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 506fb48248c5781d9df43169e79c298e274491f3 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Post-fix loop-lag re-measurement is still pending ("needs a restart") — confirm the promised number lands on the PR before merge.

What this change ships

Inventory (5 items) — 5 justified

Intent: FIX — stop sidebar broadcasts from stalling the event loop, which the adaptive controller misread as host pressure and pinned the subagent cap at 1 (#12710, with measured probe and py-spy evidence).

  1. Sidebar broadcasts serialize each slot once instead of three times; the ~600 ms loop stall drops to one pass — justified
  2. New broadcast seam serialize_slot_views; existing serialize_slots callers (chat_handlers.py:1478, ws.py:483) untouched — justified
  3. New _ChatSlot.source_links_view swaps only the per-audience field; premise verified — flags reach SlotProjection.to_dict solely via project_source_links (slot_projection.py:303) — justified
  4. Owner view still built only when an owner socket is registered, decided before the pass; behavior unchanged — justified
  5. test_public_repo_status_rides_general_frame_owner_gets_full rewired from stubbed serialize_slots to a real slot, assertions kept — declared, justified

Sibling count: grepped serialize_slots\( across src — the broadcast was the only multi-pass caller; 0 unfixed siblings.

[FIRST-PRINCIPLES-REVIEWED] 506fb48

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of 506fb48248c5781d9df43169e79c298e274491f3 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

One pass is still ~200 ms on the loop at 80 tabs — 80% of the 250 ms decrease signal, and the post-fix measurement is still pending.

Watch

The fix removes the 3× redundancy but keeps O(n) regex-heavy serialization synchronous on the event loop: the description's own numbers ("one pass is already 192–211 ms", "loop_lag >= 250 ms is a sufficient-alone decrease signal") put the remaining single pass within noise of the controller threshold, so the cap-pinning symptom the PR exists to fix can recur on a modestly larger sidebar, and the description defers proof ("Post-fix loop-lag measurement … will be reported on the PR once done").
Clears when: the promised post-fix loop-lag measurement is posted on the PR showing broadcast stalls below 250 ms on the reporting sidebar.

Suggestions

  • If the measurement still shows lag, the follow-up shape is per-slot to_dict memoization keyed on slot revision — the pattern _source_links_cache already establishes — not a fourth audience seam.
  • The "flags touch only source_links" invariant is pinned only by the equality test's fixtures; a future flag-dependent field needing slot state the fixture lacks slips past it (fail-closed: the field silently vanishes from derived views). Consider asserting the invariant structurally, e.g. a test that greps/introspects SlotProjection.to_dict for flag reads outside project_source_links.

[DESIGN-REVIEWED] 506fb48

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 506fb48248c5781d9df43169e79c298e274491f3 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 506fb48

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in import-only overlap files (agent.py, handlers/_shared.py, decisions/*, mcp_discovery.py, vector_memory.py) since this head went green (b2bc03a -> 0a80050); CI re-running. The shard-4 red (test_running_and_turn_running_slot_readers_are_enumerated, missing _switch_target_busy) reproduces on untouched main and is fixed upstream by #12715/#12716; not patched here.

@Pearcekieser
Pearcekieser force-pushed the perf/slots-broadcast-single-pass branch from bc079cd to a853ed0 Compare September 22, 2026 05:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_pending_boundary_consumer_semantics.py (#12718 fixes the shard-4 slot-reader tripwire that was red on main) since this head (0a80050 -> 194f6d5); CI re-running.

@Pearcekieser
Pearcekieser force-pushed the perf/slots-broadcast-single-pass branch from a853ed0 to 97e2111 Compare September 22, 2026 06:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Re-pushed 97e2111 as e9de673 (identical tree, amended timestamp only): the four fork lanes on the previous head (GPT 5.6 Review, Opus 4.8 Review, Security Scope Review, Internal Content Scan) failed before producing a verdict on API rate limit exceeded for installation (HTTP 403 x3 retries), and a fork contributor cannot gh run rerun them. No code change.

@Pearcekieser
Pearcekieser force-pushed the perf/slots-broadcast-single-pass branch from 97e2111 to e9de673 Compare September 22, 2026 06:52
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 22, 2026
…e times

Every slots push serialized every open slot three times -- a bare list for
SSE, a dashboard-user list for the WS frame, and an owner list -- each pass
re-running the per-slot projection body (markdown strip, credential
redaction, options parse) and source-link budgeting, synchronously on the
event loop. The three views differ only in each slot's source_links field.
Serialize once and re-project that one field per audience.

The repeated pass stalled the loop for 0.4-1.1 s per broadcast on a large
sidebar; the adaptive concurrency controller read those stalls as loop_lag
pressure and kept halving the subagent cap to its floor on an idle host.
@Pearcekieser
Pearcekieser force-pushed the perf/slots-broadcast-single-pass branch from e9de673 to 506fb48 Compare September 22, 2026 07:32
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in import-overlap files (adaptive/controller.py, config/sections.py, eventlog/service.py, subagent.py and tests) since this head (-> 1256a48); CI re-running. No code change.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • remaining single pass still synchronous on the loop; post-fix measurement owed — fixed (measurement below); live-gateway lag reading pending a restart

Synthetic harness on this branch (80 slots x 60 messages with links, markdown and [OPTIONS:], best of 5): three full passes 120.8 ms vs single pass + reproject 42.1 ms, a 2.87x cut in loop time per broadcast. Script: bench_slots.py (DashboardState + real _ChatSlots).
The live gateway that produced the 0.4-1.1 s stalls has not been restarted onto this branch yet (that interrupts every active session on the reporting host), so the in-situ loop-lag reading is still owed; it will be posted here once the operator schedules the restart.
Agreed the remaining pass is still on the loop; per-slot memoization keyed on slot revision is the natural next step and is tracked in #12710 as a follow-up, not folded into this PR.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • follow-up shape: per-slot to_dict memoization keyed on slot revision — accepted-and-deferred

Agreed on the shape (_source_links_cache is the precedent). Recorded on #12710 as the next step if the live measurement still shows lag; this PR stays the minimal 3x-to-1x change so its equivalence proof stays byte-for-byte.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • "flags touch only source_links" invariant pinned only by fixtures — rebutted

The invariant is pinned at the source, not only by the fixture: slot_projection.SlotProjection.to_dict reads include_check_status in exactly one expression (project_source_links(...)) and never reads dashboard_user; _ChatSlot.to_dict binds dashboard_user only into that same callable. A new flag-dependent field would have to be added to that signature, at which point the equality test compares three full passes against the derived views and fails loud, since the full pass would carry the field and the reprojection would not.
The fixture also uses real slots with links, options and no-link cases, so a field that appears on any of them is covered; a field conditional on state none of them has is the only gap, and it is caught by the signature point above.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after verified readiness: passed, PR Readiness success on this head, mergeable, no open change requests. [operator: bolichen97 batch review 2026-09-23]

@bolichen97
bolichen97 merged commit ee906c4 into kirodotdev:main Sep 23, 2026
95 of 96 checks passed
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…e times (kirodotdev#12711)

Every slots push serialized every open slot three times -- a bare list for
SSE, a dashboard-user list for the WS frame, and an owner list -- each pass
re-running the per-slot projection body (markdown strip, credential
redaction, options parse) and source-link budgeting, synchronously on the
event loop. The three views differ only in each slot's source_links field.
Serialize once and re-project that one field per audience.

The repeated pass stalled the loop for 0.4-1.1 s per broadcast on a large
sidebar; the adaptive concurrency controller read those stalls as loop_lag
pressure and kept halving the subagent cap to its floor on an idle host.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slots broadcast serializes every slot 3x on the event loop; loop lag pins the adaptive subagent cap at 1

2 participants