Skip to content

fix(acp): give interactive chat starts priority over subagent starts - #16020

Merged
bolichen97 merged 1 commit into
mainfrom
fix/chat-start-priority
Oct 3, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/chat-start-priority

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A session start that a person is waiting on never queues first-in-first-out behind subagent, cron or pool-fill starts.

Three in-process queues bound every agent start: the SessionManager cold-start semaphore, the gateway-wide spawn admission and the session/new gate. All three were FIFO. When a subagent fan-out was running, a person who opened a chat, typed into Slack or pressed Refine waited behind every start already queued, and nobody was waiting on any of those.

Why it matters

Fan-outs are routine. One parent can queue dozens of starts, and the gate admits two at a time. Behind that queue, a person's first message, or a Slack reply, sat for the whole fan-out, and from the person's side it looked like a hang ("chats throttling each other"). It hits everyone who runs subagents and uses the dashboard or a channel at the same time.

Not a goal

  • gatewayd's daemon-wide MCP SpawnGate, which only exists when mcp_gateway.stub_servers is configured. Tracked as fix(mcp-gateway): order gatewayd SpawnGate waiters by start priority #15830 (G11b).
  • Restructuring _pool_fill_lock. That lock is only held with session.pool_size > 0, and the hold already existed. Residual tracked as fix(session): pool refill should not hold _pool_fill_lock across the start-permit wait #15831.
  • Boosting an in-flight background respawn of the shared _bg runtime when a person starts waiting on _bg_runtime_lock. The person's own session/new is ordered; the respawn already under way runs at its starter's priority. Listed as a known residual in acp-client.md.
  • Changing the background width of any queue, agent.session_start_concurrency, or the collector headroom. The one capacity change is deliberate: _start_sem gains one person-only permit (outer ceiling 4 -> 5, new_cold_start_semaphore). Background still holds at most 4, and the extra start is a person's chat that was going to start anyway. The gateway-wide spawn admission (_COLD_START_MAX_CONCURRENT = 2) is unchanged, so on a memory-tight host no more than two runtimes spawn and initialize at once, as before.
  • The per-line cold start of live meeting translation. That stream stays BACKGROUND.

What changed (motivation → approach → change)

  • Root cause: all three start queues were plain FIFO semaphores, and nothing recorded who was waiting on a start.
  • Primitive: one PrioritySemaphore in the neutral kiro_crew/start_priority.py (application code may not import kiro_crew.acp). FIFO within a priority, FOREGROUND first, direct hand-off to the woken waiter. A queued BACKGROUND waiter is passed by at most _FOREGROUND_BYPASS_LIMIT FOREGROUND grants, so background is never starved. release(priority) rejects a release naming a class that holds nothing. drain() is the identity sweep's barrier: it outranks both priorities and hands back what it holds, even when cancelled.
  • Who is FOREGROUND: every start is BACKGROUND unless its caller claims FOREGROUND where a person is waiting. Nothing is derived from a session key, because automation runs on interactive keys too. Claimers: the dashboard runner for a turn a person sent (plus the eager respawn and canary that turn arms); channel messages a transport received from a person (person_origin); Slack events and interactions unless a trusted bot posted; the CLI chat; STT endpointing through run_bg_oneliner -> get_bg_session; and every dashboard handler, which reads who asked through owner_start_priority (an app token is BACKGROUND, because the person-only reserve must not go to an app). Issue Radar's label-recommendation route now reuses _run_oneshot_model, so its claim is made in one place.
  • Drained channel replays: a queued entry records its own person_origin (queue_drain.person_tag; absent reads False), and a drain ORs the flags of the entries it collapses, so a replay never takes the finished turn's opener's flag.
  • Outer reserve: _start_sem is held across the provider's inner queue waits, so it gets one additive person-only permit. Background keeps its width of 4.
  • Startup clock: a subagent's clock pauses while it waits in a queue (dedicated path: all three; shared path: the gate, plus the companion runtime's per-parent lock and spawn admission, never the spawn's own work). Waits accumulate, are measured on the watchdog's own clock so host sleep is not charged, and reset when _exec_started is re-stamped. _startup_deadline now budgets two handshake rounds of initialize plus session/new, and the paused total is capped by _START_QUEUE_MAX_SECS, past which a start ends as "Never started: start queues saturated".
  • Telemetry: the queue logs carry priority=, per-class depth and the bypass count. A queue wait under START_QUEUE_LOG_MIN_MS (100 ms) is no longer logged: the subagent session/new line used to fire on any wait above zero, and an uncontended acquire measures a few microseconds, so every start logged a wait nobody had. The start clock still receives every wait; only the log line is gated.
  • Checked and left as is: run_bg_oneliner's timeout bounds the drive only, because cancelling the acquisition kills the shared _bg (re)spawn and leaks a session whose session/new was already sent. A test pins this.
Area Change
start_priority.py, acp/runtime.py, acp/runtime_start.py, providers/acp.py, providers/base.py the semaphore; spawn admission and session/new gate take a priority
session.py, session_allocation.py, session_background.py, session_lifecycle.py, session_pool.py _start_sem reserve, priority threading, drain() sweep barrier
subagent.py, subagent_manager/run.py, subagent_manager/monitoring.py, subagent_manager/terminal.py paused clock, deadline, saturation cap
dashboard/request_priority.py, dashboard/chat_runner.py, dashboard/chat_handlers.py, dashboard/chat_threads.py, dashboard/handlers/*, dashboard/stt_stream.py, llm_helpers.py dashboard claimers
src/kiro_crew/apps (builtins/issue_radar, builtins/meetings, builtins/code_review_sage) Issue Radar claim and route reuse; meetings stays BACKGROUND; a test fake
messaging/transport.py, messaging/dispatch.py, messaging/queue_drain.py person_origin, ChannelTurn.start_priority, queued-entry flag
src/kiro_crew/slack, src/kiro_crew/discord, src/kiro_crew/teams, src/kiro_crew/telegram, src/kiro_crew/webex, src/kiro_crew/feishu, src/kiro_crew/imessage, src/kiro_crew/wecom, src/kiro_crew/weixin, src/kiro_crew/whatsapp each transport's receive sets person_origin; dispatchers and drains pass it on
cli_chat.py, task_planner.py, src/kiro_crew/taskrunner.py, workflows/service.py, config/sections.py, config/loader.py, src/kiro_crew/constants.py CLI claim, Task Runner and workflow authoring threading, help text, INITIALIZE_TIMEOUT_SECS

Specs updated in this commit: acp-client.md § Start priority (claimers, guarantee, residuals, clock rule), session.md, subagent.md, messaging.md, taskrunner.md, prompt-optimizer.md, dynamic-subagent-sizing.md and resource-protection.md (deadline formula), plus config-baseline.json.

Backwards compatibility

Compatible. Every new parameter defaults to BACKGROUND, which is how starts behaved before except that a later FOREGROUND start can now pass them, within the bounded bypass. Widths, config keys and wire formats are unchanged.

The start-queue callbacks (on_gate_queued / on_gate_acquired) now also receive the queue's name, and only internal callers use them. SessionManager._start_sem must now be a PrioritySemaphore; only tests assign it.

Tests

  • test/test_start_priority.py (new):
    • The primitive: ordering, the no-starvation bound, the no-stale-streak cases, cancellation in queue and after a grant, wrong-priority release, constructor validation, held().
    • The additive reserve: background keeps its full width, the first person gets a permit at once, a second person goes ahead of background, and a freed reserve is never handed to background.
    • drain(): it outranks both priorities, gives back what it holds when cancelled, is not re-entrant, and the identity sweep completes under a foreground stream.
    • The dashboard origin rule, through the real queue actor rule (Resume press, runner requeue, cron, sub-agent).
    • The priority that reaches each queue, and that a raising callback returns the permit.
    • The real nested inversion: background holds the outer permits while stuck at the inner gate, and the chat still goes next.
    • Call-site pins for every claimer, every transport's receive, and every channel dispatcher.
  • test_session_start_gate.py: with 2 permits held by child starts and 3 more child starts queued, a chat create_session is the next session/new.
  • test_acp_provider.py: the priority reaches both runtime queues, including the resume-died respawn.
  • test_subagent_startup_pressure.py / test_subagent_startup_watchdog.py: the clock pauses and accumulates across all three queues, and a stall is reaped at the right total. A start parked past the cap gets the saturation terminal. The companion acquisition brackets only its lock and admission waits, and a hung companion spawn is reaped at the startup deadline.
  • test_subagent_startup_budget.py: the deadline formula, including the two-round case.
  • test_session_sharing.py: a task run a person opened that falls back to the companion runtime (no bootstrap provider) spawns that runtime FOREGROUND, both through _get_or_bootstrap_run_runtime and get_subagent_runtime directly.
  • Channel drains (test_webex_dispatch.py, test_teams_midturn.py, Discord, Telegram): a drained replay round-trips through the real enqueue both ways (a person's message starts FOREGROUND, a queued wake BACKGROUND) under any opener.
  • Issue Radar: the start priority is read from who asked (owner FOREGROUND, app token or no user BACKGROUND).
  • Existing tests were adapted for the new widths and callback arity.
  • Also pinned: the background singleton's three queues read the caller's priority; run_bg_oneliner does not cancel the acquisition; the startup window covers the initialize budget; and every site that re-stamps _exec_started clears the paused-clock fields.
  • Every new wait is bounded. With the reserve deliberately mutated to be stricter, every affected test fails by name and none hangs.

Manual verification

N/A. The behaviour is queue ordering inside one process, and tests drive it through the real semaphores, the real create_session gate and the real SessionManager cold-start path. A live repro needs a concurrent subagent fan-out and a chat, and that adds timing noise rather than evidence.

Related Issues

Follow-ups: #15830 (gatewayd SpawnGate priority, G11b). #15831 (pool refill lock) was fixed on main by #15857; this PR's rebase moves its four new TestFillLockReleasedAcrossStart tests onto a PrioritySemaphore and drops the residual from acp-client.md.

Pattern harvest

Rule candidate: review-prompt
Pattern: "a bounded queue that both people and automation wait in is FIFO, so a person queues behind work nobody is waiting on. Bounded start queues take an explicit, caller-claimed priority, never one derived from a session key."

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@bolichen97
bolichen97 requested a review from a team as a code owner October 1, 2026 22:33
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of a1db36f082acec65aa69052b05e58884dfd9d4e7 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the grown startup watchdog deadline (two added initialize rounds, ~+180s) still lets a genuinely wedged dedicated start die on an inner budget first.

[DESIGN-REVIEWED] a1db36f

Description read

[DESCRIPTION-READ] c9616e8309aff006e4309ab4cfce4e35f3ccb6f4fd68e8afbeb69544d5504960

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of a1db36f082acec65aa69052b05e58884dfd9d4e7 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a1db36f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a1db36f082acec65aa69052b05e58884dfd9d4e7: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of a1db36f082acec65aa69052b05e58884dfd9d4e7 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the one capacity change: the outer cold-start ceiling rises 4→5 (new_cold_start_semaphore), so confirm the memory-floor math still holds — spawn admission (2) bounds spawn+initialize only, not the later phases a fifth in-flight start now occupies.

Not justified as shipped

  • Item 9 — rides along: the Issue Radar label route's rewiring onto _run_oneshot_model is a refactor riding on the FIX, though it deletes ~20 duplicated lines so the priority claim lives in one place.

What this change ships

Inventory (10 items) — 9 justified

Intent: a FIX — a session start a person is waiting on no longer queues FIFO behind subagent/cron/pool-fill starts (pinned by test_a_chat_start_acquires_the_gate_ahead_of_queued_child_starts, which fails on base's plain asyncio.Semaphore queues).

  1. A start a person waits on (chat turn, channel message, CLI, dictation) is served ahead of queued background starts at all three start queues — justified
  2. One extra person-only cold-start permit; outer ceiling 4→5, background width unchanged — justified
  3. A queued background start is passed by at most 4 foreground grants, then served — justified
  4. Subagent startup clock pauses while queued; deadline now also budgets two initialize rounds, so dead starts are reaped later — justified
  5. New subagent terminal: queued >30 min total ends as "Never started: start queues saturated" (new tombstone cause) — justified
  6. Queue-wait logs only above 100 ms, now carrying priority, per-class depth and bypass count — justified
  7. Inbound messages record person_origin; queued entries persist it and a drained replay takes its entries' own flags — justified
  8. Dashboard handlers run at the asker's priority; an app token stays background even on its own slot — justified
  9. Issue Radar label-recommendation route reuses _run_oneshot_model — rides along: a dedup refactor, not the fix itself
  10. Internal API: start_priority threaded through provider/spawn/create_session/get_or_create; gate callbacks gain a queue name; _start_sem must be a PrioritySemaphore — justified

Lens 4/5 evidence: no prior priority primitive exists (grepped PrioritySem|PriorityQueue|heapq — only unrelated hits); start_priority has 239 uses across 59 src files; drain(), describe(), held_count each have named src consumers; the three unfixed sibling queues are declared and tracked (#15830, #15831, _bg_runtime_lock residual in acp-client.md).

[FIRST-PRINCIPLES-REVIEWED] a1db36f

Description read

[DESCRIPTION-READ] 6aca50c3a3607c8cc0549bad695775cb5f0e709c3a968807305820230cad4d4b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed a1db36f082acec65aa69052b05e58884dfd9d4e7 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a1db36f

Verdict parsed from the review's SHA-scoped output markers for commit a1db36f082acec65aa69052b05e58884dfd9d4e7.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a1db36f082acec65aa69052b05e58884dfd9d4e7: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the fix/chat-start-priority branch from 7607243 to 87ecb80 Compare October 2, 2026 00:05
@bolichen97
bolichen97 enabled auto-merge (squash) October 2, 2026 00:51
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97
bolichen97 force-pushed the fix/chat-start-priority branch from 87ecb80 to 6c87ac7 Compare October 2, 2026 21:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 2, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

This push (6c87ac7), rebased onto origin/main 0777838, still one commit:

  • Webex/Discord/Teams/Telegram drains: a queued entry records its own person_origin, and a drained replay takes the OR of the collapsed entries' flags rather than the opener's (Opus finding on _reply_envelope).
  • Subagent startup clock: the shared path pauses only while it waits for the companion runtime (its per-parent lock and spawn admission), never across the spawn's own work. _startup_deadline budgets two handshake rounds.
  • Issue Radar's AI routes read who asked (owner_start_priority). The label-recommendation route reuse is now named in the description.
  • Wording: the agent.session_start_concurrency help names the bounded background bypass, and the saturation terminal says the cap is a total across waits (both GPT findings). config-baseline.json is regenerated.
  • Docs updated in the same commit: acp-client.md, subagent.md, messaging.md, prompt-optimizer.md, dynamic-subagent-sizing.md, resource-protection.md. The _bg_runtime_lock respawn priority is listed as a known residual.
  • One small cleanup: slack/events._dispatch_queued now reads from_trusted_bot once.

The previous red lanes were not caused by this PR: Backend Lint (main's F811, fixed by #16032) and Backend Tests shard 4 / Coverage Gate (the autonudge lock flake, fixed by #16040). Both fixes are included through the rebase.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • span=1b88ed28a3ad fixed in 6c87ac7

fixed: the agent.session_start_concurrency help (config/sections.py) now says that starts a person is waiting on are served first, except that a waiting background start is let through after a bounded run of them so it is never starved, FIFO within each class.
config-baseline.json was regenerated from the same text, so both copies agree.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • span=726408e9068d fixed in 6c87ac7

fixed: the saturation terminal (subagent_manager/terminal.py) now reads "Never started: start queues saturated (over Ns queued for start permits in total, behind other starts)", which matches _start_queue_saturated_secs: it sums the separate waits plus the current one.
The "Never started: start queues saturated" prefix is unchanged, because the specs and test_subagent_startup_watchdog quote it.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • span=332b9d7b0964 fixed in 6c87ac7

fixed: webex _reply_envelope now takes person_origin as a required keyword and sets it on the envelope it returns, whatever base it was built on, so a gateway-built opener no longer leaks its False onto a drained replay.
The flag passed is the queued entries' own (messaging/queue_drain.person_tag at enqueue, OR of entry_person_origin across the collapsed entries), not an assumed True, because a gateway wake can be queued too. Discord, Teams and Telegram drains do the same.
Pinned by test_webex_dispatch TestQueueAndDrain and test_teams_midturn TestDrain, round-tripping through the real enqueue both ways under any opener.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • span=74a36ad6a809 fixed in 6c87ac7

fixed (description): the PR body now names the Issue Radar label-recommendation route moving onto _run_oneshot_model, with the reason: that route is one of the start-priority claimers, and sharing the helper makes the claim (owner_start_priority(request)) in one place instead of two copies that could disagree.
The code change stays. It deletes the route's duplicate get_or_create/stream/release/destroy block, and test_the_start_priority_is_read_from_who_asked pins the shared path.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • span=74a36ad6a809 fixed in 6c87ac7

fixed (description): the PR body now names the Issue Radar label-recommendation route moving onto _run_oneshot_model, with the reason: that route is one of the start-priority claimers, and sharing the helper makes the claim (owner_start_priority(request)) in one place instead of two copies that could disagree.
The code change stays. It deletes the route's duplicate get_or_create/stream/release/destroy block, and test_the_start_priority_is_read_from_who_asked pins the shared path.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97
bolichen97 force-pushed the fix/chat-start-priority branch from 6c87ac7 to d68ba49 Compare October 2, 2026 23:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 2, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in config-baseline.json, docs/system-specs/modules/session.md, src/kiro_crew/acp/runtime.py, src/kiro_crew/config/loader.py, src/kiro_crew/config/sections.py, src/kiro_crew/providers/acp.py, test/test_acp_provider.py and test/test_acp_runtime.py since this head went green (0777838 -> f0be829); CI re-running.

  • The rebase was clean, still one commit, authorship unchanged (6c87ac7 -> d68ba49). No code changed.
  • The one red on 6c87ac7, Backend Tests (Windows) (1), was test_issue_radar_crew_runtime.py::TestSeedingSurvivesABriefLogLock::test_the_held_lock_is_real_contention. That test came from test: touch the crew log off the event loop in every async test #16102 on main, and this PR does not touch it or the crew log. It failed with a raw OSError from the single-shot on-loop acquire in _durable_end, inside _prepare_write -> _fold_checkpoint. That point is before the test's holder thread starts, so the lock was held by the crew log's own reader thread, not by the test's holder. This is a wall-clock race (testing-conventions class 2, "Interleavings: name the point"), not a defect in this diff. main's own CI on the same test is green.
  • PR description: the Telemetry bullet now states the START_QUEUE_LOG_MIN_MS (100 ms) log threshold, which First Principles listed as undeclared.
  • Local: black/encoding/isort/flake8/mypy clean; targeted tests on the overlapping files (1905) pass; local-gate's related backend chunks 1-3/6 (90,815 tests) pass.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • Item 10 — undeclared: START_QUEUE_LOG_MIN_MS stops logging gate waits under 100 ms fixed in the PR description

fixed (description): the Telemetry bullet now declares that a queue wait under START_QUEUE_LOG_MIN_MS (100 ms) is no longer logged. The base logged the subagent session/new wait whenever it was above zero, so an uncontended acquire, which measures microseconds, wrote a line for every start.
The code stays as it is. Only the log line is gated: _on_gate_acquired still adds every wait to _start_queue_wait_ms, so the paused start clock and the saturation cap see the full total.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • Verify the widened startup deadline (695s at defaults, from ~425s) kept as designed

kept: the window is wider on purpose. _startup_deadline (subagent.py) is now _STARTUP_HANDSHAKES * (session_start_timeout_secs + INITIALIZE_TIMEOUT_SECS) plus the collect grace and launch margin. The start clock now runs through the spawn's own handshakes and pauses only while queued, so a retried start that uses its full initialize and session/new budgets twice is still a live start. Under the old ~425s window the watchdog would reap it while those budgets had not run out, and its terminal would blame the start for a timeout the start had not reached.
The cost is that a truly hung start is reaped about 4.5 minutes later. That bound is still fixed and does not grow with the crowd, the queued time is capped separately by _START_QUEUE_MAX_SECS, and test_subagent_startup_budget pins the formula, including the two-round case.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 3, 2026
A start a person was waiting on queued FIFO behind every subagent,
cron and pool-fill start at three bounded queues: SessionManager's
cold-start semaphore, the gateway-wide spawn admission and the
session/new gate. A subagent fan-out held the person typing behind
work nobody was waiting on.

All three are now one PrioritySemaphore (kiro_crew.start_priority):
FIFO within a priority, FOREGROUND first, and a background waiter is
passed by at most _FOREGROUND_BYPASS_LIMIT foreground grants. Every
start is BACKGROUND unless its caller claims FOREGROUND where a person
waits: the dashboard runner for a turn a person sent, eager spawns
from the owner's slot actions, channel messages a transport received
from a person, side chat, threads, the CLI, the optimizer, Issue
Radar, Refine, Plan, owner workflow authoring and STT endpointing.
Nothing is derived from a session key, because automation runs on
interactive keys too.

_start_sem gains an additive person reserve, so background keeps its
width and the first concurrent person start never waits behind
background holders stuck in the inner queues. The identity sweep
drains it ahead of both priorities. A subagent's startup clock now
pauses while queued at any of the queues, and the paused total is
capped by _START_QUEUE_MAX_SECS.
@bolichen97
bolichen97 force-pushed the fix/chat-start-priority branch from d68ba49 to a1db36f Compare October 3, 2026 02:14
@bolichen97

Copy link
Copy Markdown
Contributor Author

Pushed a1db36f (one commit, rebased onto origin/main c0671f5). It replaces d68ba49, which conflicted with main.

  • Rebase. The only textual conflict was the get_or_create row in docs/system-specs/modules/session.md. I kept main's new sentence about the Tool Search replay and added start_priority. Main's fix(session): release pool fill lock across each start-permit wait #15857 (fixes fix(session): pool refill should not hold _pool_fill_lock across the start-permit wait #15831) added four TestFillLockReleasedAcrossStart tests that assign asyncio.Semaphore(n) to _start_sem. This PR's setter rejects that type, so those tests now use PrioritySemaphore(n). I also deleted the fix(session): pool refill should not hold _pool_fill_lock across the start-permit wait #15831 residual from acp-client.md, because fix(session): release pool fill lock across each start-permit wait #15857 fixed it.
  • Fix. If no bootstrap provider is configured, _get_or_bootstrap_run_runtime falls back to get_subagent_runtime, and that call dropped the resolved start_priority. So when a person opened a task run, the companion runtime's cold start queued BACKGROUND at the spawn admission. get_subagent_runtime now takes start_priority (on the protocol, the implementation and the SessionManager facade) and passes it to spawn, and both fallback call sites pass it on. The new test test_session_sharing.py::test_the_companion_spawn_keeps_the_callers_start_priority fails without the fix. I updated the spec line in acp-client.md to match.
  • Opus 5 Review on d68ba49 failed for an infrastructure reason, not a verdict. The validation step hit HttpError: API rate limit exceeded for installation while it was checking permissions, so it posted no findings. The fix above comes from that run's unposted discovery candidate 1. Its other candidates were low confidence or design-level. The validation pass on the new head will rule on them.

Gate: black (touched files), isort, flake8, mypy (clean), docs-lint, harness parity. The targeted tests (every test file that touches the start-priority and companion paths, plus the test files main changed since the old base) gave 11831 passed and 33 skipped.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • Verify the grown startup watchdog deadline (two added initialize rounds, ~+180s) still lets a genuinely wedged dedicated start die on an inner budget first kept, as designed

kept: this is the same item that was ruled on for 6c87ac7 (comment "Verify the widened startup deadline"), and nothing has changed since. _startup_deadline (subagent.py) is _STARTUP_HANDSHAKES (2) * (session_start_timeout_secs + INITIALIZE_TIMEOUT_SECS (90s)) plus the collect grace and launch margin. So each wedged handshake hits its own initialize or session/new timeout before the watchdog fires, and the watchdog only catches a start whose inner budgets have all run out.
A start that is truly hung is still bounded by a fixed number that does not grow with the crowd. Queued time is capped separately by _START_QUEUE_MAX_SECS. test_subagent_startup_budget pins the formula, including the two-round case.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • Verify the one capacity change: the outer cold-start ceiling rises 4->5 (new_cold_start_semaphore) kept, premise checked

kept: the fifth permit is FOREGROUND_COLD_START_RESERVE, and only FOREGROUND can hold it (PrioritySemaphore foreground_reserve). Background still holds at most MAX_CONCURRENT_COLD_STARTS = 4, and test_start_priority pins that a freed reserve permit never goes to background. The per-loop spawn admission _ColdStartAdmission (_COLD_START_MAX_CONCURRENT = 2, acp/runtime_start.py) still bounds the memory-heavy phase, process spawn plus initialize, and this PR does not change it.
In the phases after that, the fifth holder is a process that has already launched for the session a person is waiting on. That session would be resident once its start finished anyway. So the peak rises by at most that one process, and only while a person is actually waiting. This is the same answer given for 6c87ac7.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • Item 9 -- rides along: the Issue Radar label route moves onto _run_oneshot_model kept, and declared

kept: this is the same ruling as on 87ecb80 and 6c87ac7. The label-recommendation route is one of the routes that claims a start priority. Sharing _run_oneshot_model puts its owner_start_priority(request) claim in one place, not in two copies that could disagree. The move is a net deletion of the route's duplicate get_or_create/stream/release/destroy block, and the PR description names it under "Who is FOREGROUND".

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@bolichen97
bolichen97 merged commit 1423dfb into main Oct 3, 2026
99 of 105 checks passed
@bolichen97
bolichen97 deleted the fix/chat-start-priority branch October 3, 2026 04:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants