Skip to content

Orchestration: parallel foreground subagents with a cap, per-child budget and cost, child cards (#32) - #98

Merged
derekhearst merged 11 commits into
mainfrom
feat/32-orchestration
Sep 24, 2026
Merged

Orchestration: parallel foreground subagents with a cap, per-child budget and cost, child cards (#32)#98
derekhearst merged 11 commits into
mainfrom
feat/32-orchestration

Conversation

@derekhearst

@derekhearst derekhearst commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Summary

I finished #32 with option A from the triage: delegated children run in the foreground and in parallel, and no migration was needed. I picked up the interrupted attempt's uncommitted work rather than redoing it.

The new module src/lib/engine/delegation-gate.ts runs from the engine's PreToolUse hook. It forces run_in_background:false and removes isolation and mode from every call; on a gateway parent it also removes model. It refuses a fifth live child with a "wait for the running children, then delegate the rest" message (MAX_CONCURRENT_SUBAGENTS = 4). It also refuses a delegation made inside a child. Before a child starts, it runs the same budget check a chat turn uses, scoped to the child's own agent (src/lib/chat/stream-delegation.server.ts). Any error inside the gate refuses the call. Workflow is disabled, and Agent, Task and Workflow are blocked inside children, so nothing can bypass the gate.

Each delegation gets one collapsed SubagentBlockCard, opened at the parent's Agent call. It shows the agent, status, tokens, cost and duration, and expands to the child's ordered transcript built from the subagent_* frames. The delegation no longer also appears as a separate tool card.

Each completed child writes one llm_usage row: source 'subagent', runId = the parent run, agentId = the child's agent. In the installed SDK the child's reported usage covers only its last model call, while the turn total already includes the children. So each child row is subtracted from the parent's row rather than added to it, and the turn total stays the same.

Pressing Stop on the parent turn stops the children too. I confirmed in the bundled CLI that foreground children share the parent turn's abort controller, and perTaskStopAffordance is never set.

I also fixed a regression the previous attempt left: chat hooks now fire before_tool/after_tool for a delegation as Agent.

check (0 errors) and build pass. On both projects, 590+94 tests passed. The 4 failures were the two known ECONNREFUSED 4173 cost.linkage cases, plus hooks.chat-run, which I then fixed and which now passes. The port patch was reverted.

After review: This round fixed all five should_fix items and the one claimed-but-not-fixed item from the #32 review. The branch is at 8 commits on origin/main (0bb97b6), the worktree is clean and there is no migration.

  1. Stop now shows stopped, not failed. The engine records that it sent Stop to the CLI. An error result for a delegated child that arrives after that, or that carries the CLI's "[Request interrupted by user" text, closes the child's card as stopped.

  2. A child only gives back its slot when it has really finished. A child the CLI still runs in the background (because a trusted project's agent definition says background: true) now keeps its slot. Its card stays open until its task_notification, which closes the card with how the child ended and frees the slot. So the limit of 4 holds.

  3. A child's cost is now everything its model calls used. The new src/lib/engine/subagent-usage.ts adds up each call once (one count per message id, larger report wins). The SDK's typed result is folded in as the last call. This total goes on the persisted block, the done frame, the ledger row and the card's token count. Failed and stopped children now get a ledger row for what they spent.

  4. Each child's ledger row is written the moment the child finishes, not at the end of the turn. So the budget check for the next child sees earlier children from the same turn.

  5. Spec cleanup. The live spec now waits for and deletes the review item that a blocked child creates.

I also switched off SendMessage, which can restart a finished agent without passing the gate. I confirmed this in the bundled CLI: it can say "Resuming agent …".

Checks: bun run check has 0 errors, including on the two middle commits. bun run build passes. The changed spec files also type-check with tsc. 15 relevant spec files passed on both desktop and mobile (366 passed).

Closes #32

What was addressed

Audit findings (from the 2026-09-22 subsystem audit) were re-checked against the code before any change; "not a bug" means the claim did not hold up.

# Item Verdict Note
should_fix-1 stream.server.ts:895 Stop shows children as failed should_fix-1 stream.server.ts:895 Stop shows children as failed fixed The engine now sets a flag when it sends Stop. finishSubagentBlock takes that flag and also recognises the CLI's '[Request interrupted by user' text. Either one closes an error result as 'stopped' with STOPPED_REASON. Specs now cover: the real CLI interrupt result arriving after handle.interrupt(); the CLI's made-up cancellation wording after Stop; the interrupt text alone; a real error with no Stop still showing 'failed'.
should_fix-2 stream.server.ts:895 background placeholder frees the slot should_fix-2 stream.server.ts:895 background placeholder frees the slot fixed settle is now called only in closeChild, which runs when a card actually closes. A launch placeholder no longer closes anything. The child's task_notification (matched by tool_use_id) closes the card through closeBackgroundedSubagent and frees the slot. If the notification arrives before the placeholder, it is stored and applied when the placeholder comes. I confirmed in the CLI that shouldRunAsync = call wish OR agent.background (only CLAUDE_CODE_DISABLE_BACKGROUND_TASKS can override it, and that would also switch off the app's background Bash). The optional 'better still' suggestion, refusin
should_fix-3 subagent-ledger.ts:97 per-child cost is only the last call should_fix-3 subagent-ledger.ts:97 per-child cost is only the last call fixed createSubagentUsageTally adds up message.usage from each forwarded child assistant message, plus message_start/message_delta events. It counts each message.id once and takes the larger figure for each field. AgentOutput.usage is folded in as the last call, and is used on its own only when no call was seen (usageBasis 'final_call'). The ledger row uses the tally (usageBasis 'model_calls', modelCalls, status), and so does the card's token count ('48k tokens' instead of the old last-call 12k).
should_fix-4 stream-delegation.server.ts:38 budget check blind to spend earlier in the same turn should_fix-4 stream-delegation.server.ts:38 budget check blind to spend earlier in the same turn fixed I used the reviewer's alternative: each child's ledger row is written as soon as its card closes (onSubagentDone, awaited before the stream loop reads the next message, which goes to createSubagentLedger.record). enforceBudgetGuard reads the ledger, so later waves are checked against the waves that have finished. What is still not counted (running siblings, the parent's own calls this turn, $0 rows on the subscription) is stated in docs/cost/spec.md.
should_fix-5 costs.subagent-ledger-live.spec.ts:159 review item left behind should_fix-5 costs.subagent-ledger-live.spec.ts:159 review item left behind fixed The test now checks that the policy_override_request item exists. The finally block waits up to 5s for it to appear, then deletes it with payload->>'conversationId' = ids.conversationId.
claimed-not-fixed 32-cancel claimed-not-fixed 32-cancel fixed Same fix as should_fix-1. The docs' claim that Stop shows children as stopped is now true for the stream the real CLI sends.
notes-unverified-1 SendMessage resumes a child outside the gate notes-unverified-1 SendMessage resumes a child outside the gate implemented SendMessage is now in DISALLOWED_BUILTIN_TOOLS and SUBAGENT_DISALLOWED_TOOLS, and a spec asserts it. I confirmed in claude.exe 2.1.278 that SendMessage can wake an agent ('Resuming agent …'). The app has no agent teams that need it.
notes-weak gate.live() assertions passed because of reset() notes-weak gate.live() assertions passed because of reset() fixed The slot checks now read gate.live() inside the scripted stream, before the end-of-turn reset: the foreground rewrite test, both background-child tests, and the cap/re-issue test.
notes-unverified-2 project hooks and agent definitions can undo the rewrite notes-unverified-2 project hooks and agent definitions can undo the rewrite partially-fixed A definition's background:true can no longer get past the limit, because the slot is held until the child ends. A definition's own isolation, and a trusted project's own PreToolUse updatedInput hooks, are that project's configuration. They can already run arbitrary commands, so I documented this rather than blocking it.

Verification

  • bun run check: 0 errors
  • bun run build: passes
  • Specs: E2E_PORT=4514 E2E_SKIP_EXTERNAL_CHECKS=1 E2E_NO_MODEL_CREDENTIALS=1 E2E_QUARANTINE=1 bunx playwright test tests/engine.delegation-stream.spec.ts tests/engine.subagent-usage.spec.ts tests/engine.delegation-gate.spec.ts tests/costs.subagent-ledger.spec.ts tests/costs.subagent-ledger-live.spec.ts tests/chat.subagent-card.spec.ts tests/hooks.chat-run.spec.ts tests/engine.stream-routing.spec.ts tests/engine.agent-definitions.spec.ts tests/engine.stream-approvals.spec.ts tests/engine.run-result.spec.ts tests/engine.tool-surface.spec.ts tests/chat.agent-tool-policy.spec.ts tests/engine.tool-decision.spec.ts tests/costs.settings-budget.spec.ts --workers=3 (desktop + mobile): 366 passed, 0 failed. The e2e-port patch was applied and reverted, and tests/server-env.ts is unchanged. The changed spec files also type-check with tsc.
  • Independent review: approve (0 must-fix, 5 should-fix), addressed in a follow-up round.

Docs updated

  • docs/cost/spec.md
  • docs/chat/spec.md
  • docs/agents/agents.md
  • docs/agents/spec.md
  • docs/structure/claude-desktop-parity.md
  • docs/structure/parity-reaudit-2026-09-22.md
  • README.md
  • docs/hooks/hooks.md

Follow-ups / residual risk

  1. Why I did not refuse unknown agent types (should_fix-2's 'better still' suggestion). A trusted project's own .claude/agents/ is a deliberate feature (Per-project instructions and knowledge #23 and setting-sources.ts). A project file can also reuse a built-in agent's name, so a name allowlist would not be airtight. Holding the slot until the child really ends is what keeps the limit enforced whatever a definition asks for. A definition's own isolation, and a trusted project's own PreToolUse hooks, are that project's configuration and can already run arbitrary commands. This is documented in docs/agents/spec.md.

  2. Output tokens need a live check. I confirmed in the bundled CLI that each split assistant message is emitted when its content block ends, and that the call's final usage is written onto those same objects only when message_delta arrives. I did not confirm whether the copy sent out to our stream carries the final output count. If it does not, and child stream events are not forwarded either, output tokens for a child's earlier calls are undercounted; they are a floor. Input and cache tokens are exact, and so is the last call (the typed result is folded in). Either way the carve-out keeps the turn's total exact. Worth checking live once model credentials are available: look at metadata.modelCalls and tokens_out on a 'subagent' row.

  3. Budget blind spots that remain. The check does not yet count siblings still running beside the child being checked, or the parent's own model calls this turn (its row is written at the end). So one wave can go past a limit by what that wave spends. On the Claude subscription every row costs $0, so a dollar limit never blocks there. All of this is stated in docs/cost/spec.md.

  4. Unchanged from last round, both noted by the reviewer as not blocking. A delegation that needs approval still reserves its slot and runs the budget check before the user approves. A limit refusal for an 'ask' delegation still leaves an approval card that only the later tool result closes.

  5. The Stop behaviour has not been tried live. In a real stopped run, the child cards should read 'stopped' and carry a usage figure.

🤖 Generated with Claude Code

derekhearst and others added 11 commits September 23, 2026 20:11
@derekhearst @claude
feat(engine): gate delegations in the PreToolUse hook, one card per c…
90accf7 
…hild

Fan-out (#32) is the model calling the SDK's `Agent` tool (formerly
`Task`) several times in one message; the CLI runs them as a parallel
batch. Nothing bounded how many ran, what they cost or how deep the tree
went, and canUseTool cannot: the Agent tool answers its own permission
check with "allow" in the modes we run. The engine's PreToolUse hook
fires for every call and is binding, so admission lives there.

$lib/engine/delegation-gate (pure) decides each delegation:
- rewrites run_in_background to false. A background child in a one-shot
  query is held back and killed at the print-mode ceiling and reports
  only a token total; a foreground child runs in parallel with its
  siblings and returns a typed AgentOutput;
- strips isolation and mode, and model when the parent is on the gateway;
- refuses a child past MAX_CONCURRENT_SUBAGENTS (4), with "wait for the
  running children to finish, then delegate the rest". Refused, not
  queued: a hook that waited would time out and let the call through.
  The slot is reserved before the budget check awaits;
- refuses a call made inside a child (agent_id on the hook input);
- runs an injected per-child budget check, bounded at 10s;
- fails closed on any surprise.
Agent, Task and Workflow join every definition's disallowedTools, and
Workflow (the CLI's scripted fan-out, which would bypass the gate) joins
DISALLOWED_BUILTIN_TOOLS. The roster now also maps each offered key to
its agents row, for the budget check and the ledger.

The child's card opens at the parent's Agent call ($lib/engine/
subagent-block), so a refused child still has one, and the delegation
gets no tool block or tool_call frame beside it. The child's messages,
routed by parent_tool_use_id, build an ordered, capped transcript
($lib/engine/subagent-transcript). The delegation's typed result closes
the card with SubagentDetails (tool-result-details now distils Agent and
Task); a launch placeholder does not; a card still open when the turn
ends is closed as stopped. New block fields are optional jsonb: no
migration. readTurnUsage now says whether its figure already includes
the children, for the ledger.

Stop needs nothing new: a foreground child runs on the parent turn's own
abort controller (bundled CLI 2.1.278), so interrupt() ends it with the
turn, and perTaskStopAffordance stays undeclared.

Chat hooks read the delegation off its card's frames, so before_tool /
after_tool still fire for it (as `Agent`, with the child's report).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
feat(costs): budget-check each delegated child and give it a ledger row
ab6c893 
The chat stream now hands every turn a delegation gate
($lib/chat/stream-delegation.server). Its budget check is the same
enforceBudgetGuard a chat turn passes, scoped to the child's own agent,
so a fan-out cannot walk through a limit set on the agents it delegates
to; a blocked child records the same alert and review item, and the
model is told not to retry. A child with no agents row of ours (the
SDK's built-in agents) is checked as the parent's agent.

Each child that completes gets one llm_usage row: source 'subagent',
runId = the parent run, agentId = the child's agent (else the parent's),
model = the model it ended on, cost 0 on the subscription and priced
from the catalogue on the gateway ($lib/costs/subagent-ledger).

The SDK's per-child usage (AgentOutput.usage) is the child's LAST model
call, not its total (read in the bundled CLI 2.1.278), while the turn's
modelUsage already includes every child. So a child row is carved out
of the parent's row rather than added: tokens field by field, cost once
the child row is priced. The turn's total is unchanged and nothing is
counted twice; the rest of a child's spend stays on the parent, where
all of it was before. Where the turn figure is main-loop only (a resumed
session with no baseline) the child rows are additions. The message and
conversation totals still show the whole turn, and each child's cost is
stamped on its card before the blocks persist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
feat(chat): each delegated child is a collapsed card with its own tra…
44f4bc8 
…nscript

SubagentBlockCard now starts collapsed, live and reloaded alike, because
a fan-out opens several at once. Collapsed it shows the child agent, how
it ended (working, done, failed, refused, stopped), its task, and its
tokens, cost, duration and tool count, leaving out what the child did
not report ($lib/chat/subagent-card, pure). Expanded it shows the
child's own transcript in order: what it said, and each tool it called
with a short hint at what it touched and whether it worked. A refusal
or failure shows its reason.

The live page builds the same transcript from the subagent_* frames
(streaming-blocks), subagent_done carries the typed result and how the
child ended, subagent_start is idempotent for a resumed stream, and a
partial save on Stop keeps the card and marks a running child stopped.
Blocks saved before this still render from their text and tool names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
docs: parallel delegation, per-child budget and ledger rows (#32)
72af3ad 
agents.md gains the delegation rules and the fan-out flow in plain
English; agents/spec.md replaces the category-routing section, which was
never built, with how delegation works and what was verified in the
installed SDK. cost/spec.md explains the subagent rows and the
carve-out; chat/spec.md the child cards and Stop; hooks.md how a
delegation is reported. The parity table's orchestration row moves to
even, and the re-audit's #32 section is marked shipped, correcting its
canUseTool, background and worktree claims.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
fix(engine): Stop reads as stopped, a backgrounded child keeps its sl…
dd01401 
…ot (#32)

Review follow-ups for delegation.

- Stop: the CLI answers every delegation still running with an error
  result of its own ("[Request interrupted by user for tool use]", or a
  synthetic cancellation) before the turn's result, which closed each
  card as failed. The engine now records that it sent the interrupt,
  and an error result after that, or one carrying the CLI's interrupt
  text, closes the card as stopped.
- The slot is freed when the child's card closes, not when its call is
  answered. A trusted project's agent definition with background: true
  is backgrounded by the CLI whatever the call says; its launch
  placeholder no longer frees the slot, and its task_notification
  closes the card and hands the slot back, so the cap holds.
- Each child's spend is added up over its own model calls (forwarded
  assistant messages and message_start/message_delta events, one count
  per message id, largest report wins), with the typed result folded in
  as the last call. The total rides on the card's done frame and on
  the persisted block as `usage`.
- onSubagentDone: the caller is told as each card closes, before the
  loop reads on.
- SendMessage is off for parents and children: it can resume a finished
  agent outside any Agent call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
fix(costs): book each child as it finishes, for everything it spent (#32
670ad64 
)

- A child's ledger row now carries the sum over its model calls
  (usageBasis 'model_calls'), falling back to the SDK's last-call figure
  only when none of its calls was seen ('final_call'). A child that
  failed or was stopped partway gets a row for what it spent.
- The row is written the moment the child's card closes, through the
  engine's onSubagentDone, instead of after the turn. The budget check
  the next child passes reads the ledger, so a turn that fans out in
  waves is checked against the waves that already finished.
  createChatDelegation hands the route the gate and the ledger together;
  settle() books any child not booked yet and returns the parent's share.
- The live spec no longer leaves the blocked child's review item in the
  shared database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
feat(chat): a child's card counts every token its calls used (#32)
75f7edc 
The card's "N tokens" was the SDK's totalTokens, the child's context at
the end plus its final answer. It is now the child's spend added up over
its model calls, the same figure as its ledger row, live and after a
reload. A card saved before falls back to the old figure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
docs: stopped children, background slots, summed child spend (#32)
1701c4f 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
Merge feat/35-live-shell into feat/32-orchestration
d538d8d 
Brings #14, #9, #22, #24, #4 and #35 (live background-shell output)
under the orchestration work (#32).

Resolutions:
- stream.server.ts: kept #32's delegation routing (PreToolUse admission
  gate, child cards, per-child spend, stopped children) beside #4's
  AskUserQuestion answered in canUseTool, #35's background shells and
  #24's transcript tail. Imports combined (UsageCoverage kept). On the
  result message, background shells end first, then readTurnUsage keeps
  includesSubagents and unfinished children are closed; the no-result
  path does both as well.
- chat/[id]/stream/+server.ts: the delegation (gate + ledger) is created
  once per turn and passed into the #24 runWithResumeFallback attempt
  (a fallback only retries an attempt that produced nothing, so no child
  ran in it). The parent's gateway cost is priced by #9's
  priceGatewayTurn over #32's carved parentUsage, logged with
  ledgerCostOverride and the backend/costBasis metadata; messageCost
  still adds the children's rows back.
- tool-result-details.ts: SubagentDetails (#32) and AskUserQuestionDetails
  (#4) both in ToolResultDetails; SHELL_TOOLS is #35's Bash-only set,
  SUBAGENT_TOOLS kept; both distillers reachable in toolResultDetails.
- streaming-blocks.ts: imports combined (#4 ask-user, #35 shell tail and
  status, #32 SubagentDetails/SubagentSpend/transcript helpers).
- agent-definitions.ts: SUBAGENT_DISALLOWED_TOOLS refuses
  AskUserQuestion (#4) plus Agent/Task/Workflow/SendMessage (#32).
- options.server.ts: #32's perTaskStopAffordance note kept above #4's
  toolConfig/settings for AskUserQuestion.
- runs.schema.ts: AskQuestion import (#4) beside the subagent transcript,
  spend and SubagentRunStatus (#32).
- console-state.svelte.ts: took #14's trimmed store; the streaming-block
  types #32 widened with 'stopped' are gone and unused.
- claude-desktop-parity.md: #35's terminal row, #32's git-worktrees row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
docs(llm): a gateway turn that delegated prices its children on their…
d793f3d 
… own rows (#9, #32)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst @claude
Merge main into feat/32-orchestration
809f618 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekhearst derekhearst mentioned this pull request Sep 24, 2026
Merged
@derekhearst
derekhearst merged commit b7ed619 into main Sep 24, 2026
1 check passed
@derekhearst derekhearst mentioned this pull request Sep 24, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Multi-agent orchestration: fan out, run in parallel, show the tree

1 participant

@derekhearst