Skip to content

perf(dashboard): patch sub-agent slot changes instead of re-sending the list - #22595

Merged
JLargent42 merged 1 commit into
mainfrom
feat/slot-patch-runstate
Oct 29, 2026
Merged

JLargent42 merged 1 commit into
mainfrom
feat/slot-patch-runstate

Conversation

@JLargent42

@JLargent42 JLargent42 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop a busy dashboard from re-sending the whole session list to every tab each time a sub-agent spawns, finishes, or its queued depth changes.

On a dashboard with many sessions or many open tabs — a conductor-of-conductors workload most of all — every sub-agent lifecycle event re-serializes and broadcasts the entire slot list over every WebSocket, just to refresh two counters (subagents_running, subagents_queued). The large compressed frame stalls the gateway loop or freezes the browser decoding it, which users see as gray-outs, "clear cache and start over", and long sessions that appear to crash.

Why it matters

Sub-agent churn is the highest-frequency slot change on an active fleet — a conductor driving many workers fires these events constantly — so the whole-list rebroadcast is a dominant source of the "KiroCrew becomes unusable on long/large sessions" reports. The frame grows with sessions × tabs; host CPU and memory are not the constraint, so adding hardware does not help.

Not a goal

  • Moving WebSocket compression off the event loop (that is perf(dashboard): WebSocket compression runs on the event loop (product call: bandwidth) #17872, complementary and separate).
  • Patching run-state fields (running, queue_depth, approval state): their push sites coincide with a transcript append that needs a full frame, so there is no clean run-state-only caller to convert; left out deliberately.
  • Converting the many other push_slots_update() call sites; this targets the proven-hot sub-agent lifecycle path only.

What changed (motivation → approach → change)

Busy dashboards gray out because one sub-agent tick re-sends every session's row. The existing one-row slot_patch frame already fixes that shape for metadata edits (pin, rename, mute, folder), but the sub-agent counts rode the whole-list path instead.

The sub-agent lifecycle path now sends one patch row per slot, not the whole list: a new push_subagent_patch in dashboard/state_parts/slots_push.py (bound in dashboard/state_parts/dashboard_state.py) recomputes subagents_running / subagents_queued the same way the full frame does, and ships every live slot's two counts in that single frame. It has one caller, the debounced lifecycle flush, so an idle fleet still sends nothing. The debounced push in slack/gateway_runtime/subagent_wiring.py (_flush_slots_push) calls it in place of push_slots_update(). A legacy or SSE consumer still gets the full list; patch-capable tabs get one small frame of count rows.

The patch is deliberately stateless: every lifecycle flush re-states the counts of every live slot instead of diffing against a remembered last-sent value. A diff cache in dashboard/state_parts/slots_push.py would assume that a broadcast reached every consumer. Per-tab connect snapshots and a full SSE queue break that assumption, and a suppressed correction could leave a tab showing a finished sub-agent as running. Re-stating the counts is idempotent and corrects any tab on the next lifecycle event; a legacy/SSE consumer likewise gets the full list on every flush, so a dropped list is retried. On the client, onSlotPatch in website/src/hooks/websocket/slotList.ts reconciles subagents_queued from a patch row carrying the depth, exactly as onSlots does for a full frame; the reducer's doc comment in website/src/store/chat/subagents.ts says a sub-agent patch row now reconciles the count.

The owning specs are updated in the same commit (AGENTS.md requires it): docs/system-specs/modules/subagent.md and docs/system-specs/modules/learn-cron-dashboard.md (the slot_patch section) now describe push_subagent_patch re-stating every live slot's counts on each lifecycle flush, the legacy full-list fallback, and the client reconciling the queued count from a patch row.

Backwards compatibility

Compatible: the full-list path is unchanged on the wire and still serves any consumer that cannot apply a slot_patch. No frame type, field name, or wire shape is removed or renamed; the sub-agent path sends a narrower frame to tabs that advertised the capability, and the client reconciles the queued count from a patch the same way it already did from a full frame.

Removes nothing: additive — one new push method, a narrower frame on one already-debounced path, and one client dispatch on patch rows that carry the depth; no path, field, or capability is taken away.

Tests

  • test/test_slot_patch_frames.py: TestSubagentPatch asserts every lifecycle flush re-states all live rows for both audiences (unchanged counts included) and that the whole list is never serialized for a patch-capable tab. Regressions cover both failure modes of a diff cache: a tab that connected mid-run gets the subagents_running=False correction on completion, and a list dropped by a full SSE queue is re-sent on the next lifecycle flush. Both were proven load-bearing (red on the previous head, green after).
  • website/src/test/useWebSocket.queuedReconcile.test.ts: a slot_patch carrying the depth heals a missed subagent_queued frame; a metadata slot_patch row without the depth leaves the count alone.
  • test/test_slack_gateway_subagents.py: the debounced spawn/done and queued tests assert the lifecycle path calls push_subagent_patch and sends no full push_slots_update.
  • test/test_state_parts_dashboard_state.py: the base-member signature table gains push_subagent_patch.

Manual verification

N/A — unit coverage is sufficient. The change is a server-side broadcast optimization plus a client reconcile with no rendered output of its own; the slot_patch frame path and the queued-count reconcile are covered by the backend and vitest suites above.

Related Issues

Closes #22594

@JLargent42
JLargent42 requested review from a team and KiroCrewCaptain as code owners October 20, 2026 22:12
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 74552ac3385323763b02469d10498e46b3ccd6cc — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff changes how data gets to the dashboard: sub-agent counts now arrive as one-row slot patches, and the client fixes up its queued count from them. It adds and changes no control, label or string, so the missing screenshots and blind read leave no evidence gap.

[UX-REVIEWED] 74552ac

Description read

[DESCRIPTION-READ] fd8090e80f49efa2b26570c96bf274b5c30db96026761074dd0ada6ecb5609ee

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 74552ac3385323763b02469d10498e46b3ccd6cc — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The change reuses the existing slot_patch channel on one already-debounced path, keeps the full list for legacy and SSE consumers, and re-sends every slot's counts on each flush, so a tab that missed a frame is corrected on the next one.

Notes

  • Each flush still sends one row per live slot (about three fields each), so the frame still grows with the number of sessions, just with a much smaller factor. If conductor fleets keep growing, the next step is sending only the slots whose sub-agent counts changed. That needs a per-socket record of what each tab was last sent; a single gateway-wide cache would bring back the stale-tab problem the description rejects.
  • The old full push also refreshed, as a side effect, any other slot field that changed in the same 0.2s window. Patch-capable tabs now get only the two counts on this path, so other fields depend on their own push sites.

[DESIGN-REVIEWED] 74552ac

Description read

[DESCRIPTION-READ] 7a14a4a79c8c506171311d566331a81bd3b25afc89cef9f3f2d601ad4f181619

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 74552ac3385323763b02469d10498e46b3ccd6cc — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The change swaps one existing push for a narrower frame on the one hot path the Goal names. It reuses the existing slot_patch frame and _published_queued_by_slot, and each new mechanism has exactly one caller.

What this change ships

Inventory (5 items) — 5 justified

Intent: stop a busy dashboard from re-sending every session row to every tab on each sub-agent spawn, finish or queue change. This is a FIX (perf, #22594).

  1. When a sub-agent spawns, finishes or its queue depth changes, tabs that support slot_patch now get one small frame holding only the two counts per session, not the whole list — justified
  2. Each debounced flush re-sends every live session's counts, whether or not they changed, so a tab that connected mid-run or missed a frame is corrected on the next event — justified
  3. Older (legacy) and SSE consumers still get the full list on each flush — justified
  4. The dashboard now corrects its queued-count badge from a patch row that carries the depth, as it already did from a full list — justified
  5. New DashboardState.push_subagent_patch, with one caller, _flush_slots_push — justified

[FIRST-PRINCIPLES-REVIEWED] 74552ac

Description read

[DESCRIPTION-READ] 48d7ea12bc10e5f3d09bd4c461718c0336958d26e390bf2e47fe0c90d4a1d626

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 74552ac3385323763b02469d10498e46b3ccd6cc — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 74552ac

Verdict parsed from the review's SHA-scoped output markers for commit 74552ac3385323763b02469d10498e46b3ccd6cc.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 74552ac3385323763b02469d10498e46b3ccd6cc false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 74552ac3385323763b02469d10498e46b3ccd6cc and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 74552ac

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 74552ac3385323763b02469d10498e46b3ccd6cc false-positive: <reason, 20 to 500 characters>

@JLargent42

Copy link
Copy Markdown
Contributor Author

self-added: no

The hot paths still send the whole list; only the receiving side was widened.

fixed in 98234f4. The sub-agent lifecycle path now sends a one-row patch instead of the full list: subagent_wiring._flush_slots_push calls push_subagent_patch() in place of push_slots_update(), so a spawn/done/queued tick no longer rebroadcasts the slot list to patch-capable tabs.
The speculative run-state field set had no clean caller (its push sites coincide with a transcript append that needs a full frame), so it was removed rather than left as dead infrastructure — the First Principles subtraction.
Test proof added: test_slack_gateway_subagents.py now asserts the lifecycle path calls push_subagent_patch and sends zero push_slots_update; test_slot_patch_frames.py::TestSubagentPatch asserts the whole list is never serialized for a patch-capable tab.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026
@JLargent42

Copy link
Copy Markdown
Contributor Author

self-added: no

Item 1 (zero consumers) and Item 2 (symptom-level: full push remains).

fixed in 98234f4.
Item 1: took the subtraction — deleted _SLOT_RUNSTATE_PATCH_FIELDS and its tests. The run-state push sites coincide with a transcript append that needs a full frame, so there was no clean run-state-only caller to convert; shipping the set with no consumer was the blocker, so it is gone.
Item 2: wired push_subagent_patch in place of push_slots_update() inside subagent_wiring._flush_slots_push, and dropped the redundant crew_log._push_tree hook so the count patch fires once, on the one debounced lifecycle path, instead of alongside the full push. The sub-agent lifecycle path no longer calls push_slots_update().

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 20, 2026
@JLargent42
JLargent42 force-pushed the feat/slot-patch-runstate branch from 98234f4 to 47f2e67 Compare October 20, 2026 23:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 20, 2026
@JLargent42
JLargent42 enabled auto-merge (squash) October 21, 2026 01:11
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 47f2e67. Two current-head AI concerns have no reply at this head.

  1. Design (Watch): the diff cache is written only by push_subagent_patch (slots_push.py#L675), while full frames from serialize_slots (slots_frame.py#L864-L865) ship the same counts without updating it. I confirmed that sequence in the code: a patch sends 3, a full frame sends 4, the depth returns to 3 inside the debounce window, and the flush sends nothing, so the tab keeps showing 4 until the next change. The smallest fix is to record what the full frame shipped (or compute both counts in one helper that also writes the cache), plus a patch -> full frame -> revert test. Alternatively, post a reasoned rebuttal at this head.
  2. First Principles (Item 5): the title still promises run-state patching, which ## Not a goal rules out. Drop "run-state and" from the title.

The existing dispositions answer head 8d8ceeb, not this one. Everything else looks sound: the Opus and GPT lanes are clean and every check on this head is green. A new push will need another review.

— crew-pr-reviewer (r116)

@JLargent42 JLargent42 changed the title perf(dashboard): patch run-state and sub-agent slot changes instead of re-sending the list perf(dashboard): patch sub-agent slot changes instead of re-sending the list Oct 21, 2026
@JLargent42
JLargent42 force-pushed the feat/slot-patch-runstate branch from 47f2e67 to 5eeb2ae Compare October 21, 2026 18:39
@JLargent42

Copy link
Copy Markdown
Contributor Author

self-added: no

The diff cache now reflects what any frame last shipped, so a full frame followed by a revert no longer leaves a tab stale.

fixed in 5eeb2ae. serialize_slots (slots_frame.py) now writes _subagents_sent for each slot as it computes the two counts, so the full-frame path records what it shipped into the same cache push_subagent_patch reads. The stale-cache window you traced (patch ships 3 -> full frame ships 4 -> depth reverts to 3 inside the debounce -> flush suppresses the correcting row) is closed: the full frame leaves the cache holding 4, so the revert-to-3 patch now ships the correcting row.
Regression test added: test_a_full_frame_updates_the_patch_cache_so_a_revert_is_corrected interleaves patch -> full frame -> revert and asserts the patch corrects the tab back to 3. Proven: it fails with the cache write reverted (prove.py PROVEN).

@JLargent42

Copy link
Copy Markdown
Contributor Author

self-added: no

Item 5 — the title no longer claims run-state patching that ## Not a goal rules out.

fixed. The PR title is now "perf(dashboard): patch sub-agent slot changes instead of re-sending the list" — "run-state and" is dropped, matching the squash commit's own subject ("send sub-agent count changes as one-row slot patches instead of the whole list") and ## Not a goal, which rules run-state out. The body's What-changed section describes only the sub-agent count path and the full-frame cache sync; no run-state claim remains.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@JLargent42
JLargent42 force-pushed the feat/slot-patch-runstate branch from 6619b64 to 5fa9c81 Compare October 23, 2026 21:14
@JLargent42

Copy link
Copy Markdown
Contributor Author

Rebased once onto main for the #23264 Windows fix.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 5fa9c81.

One lane note on this head has no reply. First Principles (Item 1) points out that ## What changed says the patch sends "one patch row per slot whose counts moved", but push_subagent_patch re-states every live slot, as the next paragraph of the description explains. To clear it, delete "whose counts moved" from that sentence and post a disposition at this head.

The code looks sound to me: the counts are derived the same way as in serialize_slots, legacy and SSE consumers still get the full list, and the client heals the queued count from a patch row. The Design note about sending only non-zero rows later is optional. Opus and GPT have no findings, and CI is green.

— crew-pr-reviewer (r124)

@JLargent42

Copy link
Copy Markdown
Contributor Author

self-added: no

Item 1 — fixed (description corrected): ## What changed no longer says the patch sends only rows whose counts moved.

fixed in the PR description (no code change, no push). The sentence now reads "the sub-agent lifecycle path now sends one patch row per slot, not the whole list", which matches push_subagent_patch in dashboard/state_parts/slots_push.py: every lifecycle flush re-states every live slot's two counts. The next paragraph of the description already explains why the patch is stateless (a diff cache suppressed corrections a mid-run tab or a full SSE queue still needed). The code was already as described there; only the clause "whose counts moved" was wrong, and it is deleted.

@JLargent42

Copy link
Copy Markdown
Contributor Author

self-added: no

Design note (send only non-zero rows later) — kept as is in this PR, on purpose.

Keeping every live slot in the patch is what makes the stateless design correct. If the patch skipped zero-count rows, a slot whose last sub-agent just finished would send no row at all, so a tab still showing it as running would never get the subagents_running=False correction. Fixing that means remembering which slots were non-zero last time, which is the per-audience diff cache this PR removed after three review rounds found bugs in it. The cost being saved is small: about 60 bytes per live slot, on a path that is debounced and only runs on sub-agent lifecycle events, so an idle fleet sends nothing. If fleet sizes make this row count matter, it deserves its own measured change rather than being folded into this one.

@JLargent42

Copy link
Copy Markdown
Contributor Author

@iamwhatever re: your HOLD at 5fa9c817: addressed at the same head, with no push.

  • First Principles Item 1: deleted "whose counts moved" from ## What changed, and posted a disposition at 5fa9c817e48d9db0db776f54f26c760fd1d82a1b marked fixed (description corrected).
  • Design note (send only non-zero rows): answered at this head with the reason to keep every live row. Skipping zero rows would drop the subagents_running=False correction when a slot's last sub-agent finishes, unless a last-sent cache came back, and that cache is what this PR removed.

Could you re-review when you have a moment?

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 27, 2026
@JLargent42
JLargent42 force-pushed the feat/slot-patch-runstate branch from 5fa9c81 to 3d039e3 Compare October 28, 2026 23:33
@JLargent42

Copy link
Copy Markdown
Contributor Author

Rebased once onto main (4bf8b2eae2bb) with maintainer approval, to get a fresh merge ref for the SAST diff-only scan. No conflicts; still one commit, no code change. New head 3d039e3509bbc818ba823944152914e777b0f7ed.

Local checks on the new head: 445 targeted backend tests, 712 frontend tests, tsc, eslint, mypy, isort, black, flake8, docs-lint and the comment-history gate all pass.

The First Principles Item 1 fix (description), the Design note reply and the HOLD reply from the previous head still apply; the diff is unchanged. @iamwhatever, the re-review request stands.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 28, 2026
… instead of the whole list

Busy dashboards gray out because one sub-agent tick re-sends every session's
row. The sub-agent lifecycle path now sends a one-row slot_patch carrying only
the two counters that moved. A full broadcast records what it shipped to each
audience in its own cache, the client reconciles the queued count from a patch
row, and the legacy/SSE audience is corrected from its own cache, so a patch and
a full frame never disagree on what a tab last saw.

Closes #22594
@JLargent42
JLargent42 force-pushed the feat/slot-patch-runstate branch from 3d039e3 to 74552ac Compare October 28, 2026 23:41
@JLargent42

Copy link
Copy Markdown
Contributor Author

Rebased once more onto main (465cc2df2d84) with maintainer approval, to pick up the #25429 Frontend Lint fix. No conflicts; still one commit, no code change. New head 74552ac33.

Local checks on the new head pass: 445 targeted backend tests, 712 frontend tests, tsc, full npm run lint, isort/black/flake8, the comment-history gate and a diff-scoped Semgrep scan.

The earlier First Principles fix, Design reply and HOLD reply still apply. @iamwhatever, the re-review request stands.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 29, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. The debounced sub-agent lifecycle flush (_flush_slots_push) now calls a new stateless push_subagent_patch, which sends every live slot's subagents_running / subagents_queued as one slot_patch frame instead of re-broadcasting the whole slot list. Legacy and SSE consumers still get the full list, and onSlotPatch now reconciles the queued count from a patch row.

What I verified at the head: the patch rows come from the same slot set and the same derivation as serialize_slots (skips _slots_under_construction, uses running_agents_for(effective_session_key(s)) and _published_queued_by_slot), so a tab never gets an unknown row that would trigger a spurious fetchSlots(). The client sseSlotPatch applies arbitrary fields, so the two counts land with no reducer change. Specs are updated in the same commit. All checks are green, and every AI lane passed.

Duplicate check: no other open or merged PR adds push_subagent_patch. main still calls push_slots_update() in _flush_slots_push. #22594 has no other fix. #17872 is complementary, not overlapping.

🟡 Suggestion (non-blocking): #22594 also lists run-state fields (running, queue_depth, approval state), which this PR rules out of scope. Closes #22594 will auto-close it with that part open. Could you use Refs or file a follow-up issue for the run-state part?

AI-assisted review (Kiro), posted for bolichen.

@JLargent42
JLargent42 merged commit 2f88998 into main Oct 29, 2026
106 of 107 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(dashboard): run-state & sub-agent slot changes re-broadcast the whole slot list (gray-outs on busy dashboards)

3 participants