Orchestration: parallel foreground subagents with a cap, per-child budget and cost, child cards (#32)

README.md

@@ -26,7 +26,7 @@ Creation workflows are chat-led: New Agent and New Skill actions launch a fresh
 
 ### Agents
 
-Autonomous agents with custom roles, system prompts, and model assignments. Agents are created and managed via the chat orchestrator. The agents page lists every agent with its status — Available or Paused — and lets you pause or resume a custom agent: a paused agent is not offered for delegation and its automations and monitors are skipped, but you can still chat with it. Agent detail pages allow editing the assigned model, system prompt and hook bindings; hooks run for an agent's chats as well as its automations (see [docs/hooks/hooks.md](docs/hooks/hooks.md)). The Plan and Research agents hand an approved plan to another agent — usually Chat or Autonomous — and find its id with the read-only `list_agents` tool. See [docs/agents/agents.md](docs/agents/agents.md).
+Autonomous agents with custom roles, system prompts, and model assignments. Agents are created and managed via the chat orchestrator. The agents page lists every agent with its status — Available or Paused — and lets you pause or resume a custom agent: a paused agent is not offered for delegation and its automations and monitors are skipped, but you can still chat with it. Agent detail pages allow editing the assigned model, system prompt and hook bindings; hooks run for an agent's chats as well as its automations (see [docs/hooks/hooks.md](docs/hooks/hooks.md)). The Plan and Research agents hand an approved plan to another agent — usually Chat or Autonomous — and find its id with the read-only `list_agents` tool. A built-in agent can also fan work out to several custom agents at once: up to four children run in parallel inside the turn, one level deep, each budget-checked against its own agent before it starts and charged to it in the usage ledger; each child shows in the reply as a collapsed card that expands to its own transcript, and Stop stops them all. See [docs/agents/agents.md](docs/agents/agents.md).
 
 ### Skills
 

docs/agents/agents.md

@@ -20,6 +20,17 @@ There are three kinds of agent:
 
 A built-in agent can hand part of a task to a custom agent. The built-in agent is told which agents it can hand work to, and it asks for one by name. At most 12 custom agents are offered at once, newest first. Built-in agents and evaluators are never offered.
 
+The agent that hands work out is the **parent**; each agent it hands work to is a **child**. A parent can hand out several pieces of work at once (a **fan-out**). The children then work side by side, and each one reports back when it is done.
+
+| Rule | What it means |
+| ---- | ------------- |
+| At most 4 children at once | A fifth request while four are still working is refused. The parent is told to wait for the running children, then hand out the rest. |
+| One level deep | A child cannot hand work to another agent. The request is refused. |
+| Each child passes the budget check | Before a child starts, the same budget check a chat passes is run for that child's own agent. A child whose agent is over its limit is refused before it spends anything. The check counts the children of the same turn that have already finished, so handing out work in wave after wave cannot run past a limit. |
+| Children finish inside the turn | The app asks for every child to run inside the turn, and children report back before the parent's reply ends. A trusted project can define an agent that always runs in the background; it still counts toward the limit of four until it ends. Nothing keeps running after the parent's turn is over. |
+| Stop stops everything | Pressing **Stop** on the parent's turn stops every child that is still working, and each of their cards shows `stopped`. |
+| Each child is charged to its own agent | Every child that spent anything gets its own row in the usage ledger, for everything its model calls used, marked as sub-agent spend and charged to the child's agent. This includes a child that failed or was stopped partway. See [../cost/spec.md](../cost/spec.md). |
+
 ### Status: Available or Paused
 
 Every agent is either **Available** or **Paused**.
@@ -92,6 +103,18 @@ The assistant can also pause and resume agents itself with its `pause_agent` and
 
 While an agent is running, its card on `/agents` and its own page show the latest text it has written, with a **Watch live** link to the conversation. A run that has started but not written anything yet shows an empty preview. (Both pages used to crash the moment an agent started running.)
 
+### Hand work out to several agents at once
+
+1. The user asks a built-in agent for something that splits into independent parts, for example "review these six files".
+2. The agent asks for several children in one step, one per part.
+3. Up to four children start at once. Each shows as its own card in the chat, collapsed, with the child agent's name, what it was asked to do, and whether it is still working.
+4. As each child finishes, its card shows how it ended (done, failed, refused or stopped), how many tokens it used, what it cost, how long it took, and how many tools it called.
+5. Opening a card shows the child's own transcript: what it said and which tools it called, in order.
+6. If the agent asked for more than four, the extra requests are refused. The agent waits for the running children to report back, then asks again for the rest.
+7. The parent reads every child's report and writes the final answer.
+
+A refused child still gets a card, which says why it was refused. A child that was still working when the user pressed **Stop** is shown as stopped, not failed, with the tokens it used before it stopped.
+
 ### Hand a plan over from Plan or Research
 
 1. The user asks the Plan agent to plan a change, or the Research agent to research a question.
@@ -132,7 +155,7 @@ Research and Plan cannot run shell commands, edit files in place, push code or o
 
 ## Integrations
 
-- **Claude Agent SDK.** The custom agents a run may delegate to are passed to the SDK as its list of agents. The SDK describes each one in its delegation tool, `Agent`. The built-in agents' instructions point there instead of keeping a list of their own, so the two cannot disagree.
+- **Claude Agent SDK.** The custom agents a run may delegate to are passed to the SDK as its list of agents. The SDK describes each one in its delegation tool, `Agent`. The built-in agents' instructions point there instead of keeping a list of their own, so the two cannot disagree. Every delegation request passes through a check that runs before the tool does. That check enforces the rules in the table under **Delegation**, and it makes every child run to completion inside the turn instead of in the background. [spec.md](spec.md) has the details, including what was confirmed in the SDK.
 - **Automations and monitors.** Both check the agent's status before running it. See [../automations/spec.md](../automations/spec.md) and [../monitors/spec.md](../monitors/spec.md).
 - **Hooks.** The hook bindings saved on an agent's page run for its chats as well as its automations. See [../hooks/hooks.md](../hooks/hooks.md).
 - **Audit trail.** Every change of status is recorded as `agent.status.changed`.
@@ -149,3 +172,9 @@ Research and Plan cannot run shell commands, edit files in place, push code or o
 - A startup never overwrites a built-in agent's hook bindings, research settings, system prompt or identity skill. Only its name, role, switch reminder and tool list follow the code.
 - `request_plan_approval` only accepts a full agent id. `list_agents` is how the model finds one.
 - Promoting an agent to an identity skill re-uses the agent's existing identity skill when there is one.
+- At most 4 children work at once for one turn. A request past that is refused, not queued, and the parent is told to try again once the running children have reported back.
+- A child can never hand work to another agent.
+- A child whose agent is over a blocking budget limit is refused before it starts. A child with no agent row of its own (the SDK's built-in helper agents) is checked, and charged, as the parent's agent.
+- A child never gets its own git worktree or a separate permission mode. It works in the parent's workspace under the parent's permission mode.
+- Scripted fan-out (the CLI's `Workflow` tool) is switched off, so every child goes through the same checks. So is the CLI's `SendMessage` tool, which could wake a child that had already finished without any of those checks.
+- A child's usage row is written the moment the child finishes, not at the end of the turn. That is what lets the budget check for the next child see it.

docs/agents/spec.md

@@ -144,15 +144,37 @@ Priority when both DB and repo file exist: controlled by `AGENT_SOURCE_PRIORITY=
 
 `config.memory.enabled` controls whether memory recall is injected at the start of the agent's runs. Defaults are profile-based (for example: on for main/coding agents, off for evaluator binding) and can be overridden per agent.
 
-### Category-based subagent routing
+### Delegation and fan-out (#5, #32)
 
-When the main agent requests a subagent, it provides a category (`coding`, `ui_design`, `research`, etc.) rather than a hard-coded agent ID. Resolver order:
+There is no category routing. (An earlier draft of this spec described an `agentCategoryBindings` resolver; it was never built.) An orchestrator run is handed up to 12 custom agents as the SDK's `Options.agents` (`loadSubagentRoster` in `src/lib/engine/agent-definitions.server.ts`), and the model delegates by calling the SDK's `Agent` tool (called `Task` by older CLIs) with the agent's key as `subagent_type`. A fan-out is several `Agent` calls in one assistant message; the CLI runs them as one parallel batch and returns their results in order.
 
-1. Project-scoped `agentCategoryBindings`
-2. Workspace-scoped `agentCategoryBindings`
-3. Main agent as fallback
+**Admission.** Every delegation passes `src/lib/engine/delegation-gate.ts`, called from the engine's PreToolUse hook. The hook is used rather than `canUseTool` because the CLI's `Agent` tool answers its own permission check with "allow" in the modes we run, so `canUseTool` is not consulted for it; a PreToolUse hook fires for every call and its answer is binding. The gate:
 
-Subagent runs are ephemeral execution instances, but they reuse persistent agent definitions by `agentId`.
+| Rule | Behaviour |
+| ---- | --------- |
+| Foreground | `run_in_background` is rewritten to `false`. A background child in a one-shot query is held back and killed at the CLI's print-mode ceiling, and reports only a token total. |
+| Cap | At most `MAX_CONCURRENT_SUBAGENTS` (4) children hold a slot at once. The next call is denied with "wait for the running children to finish, then delegate the rest". It is refused rather than queued: a hook that waited would be timed out by the CLI, and a timed-out hook lets the call through. The slot is reserved before the budget check awaits, so a parallel batch cannot all see the same free slot. |
+| One level | A call whose hook input carries `agent_id` (made inside a child) is refused. `Agent`, `Task`, `Workflow` and `SendMessage` are also in every definition's `disallowedTools`. |
+| Budget | The child is checked with `enforceBudgetGuard` scoped to its own agent (`src/lib/chat/stream-delegation.server.ts`) before it starts, bounded at 10 seconds. A block, a throw or a timeout is a refusal. |
+| Isolation, mode | `isolation` is stripped: `worktree` would branch the run's checkout into a copy nothing merges back or cleans up, outside what the containment guard and the approval cards know about, and `remote` always runs in the background. `mode` is stripped (the SDK documents it as ignored). |
+| Model | Stripped when the parent runs on the gateway: the tool only takes Claude aliases, which the gateway cannot serve. A Claude parent keeps it. |
+| Plan mode | `Agent` is classified as a mutation (`permission-mode.ts`), so plan mode refuses the call before it takes a slot. |
+
+Any surprise inside the gate is a refusal with a reason, never a call waved through. `Workflow`, the CLI's scripted fan-out, is in `DISALLOWED_BUILTIN_TOOLS`, since its agents are not `Agent` calls and would bypass the gate. So is `SendMessage`: in the bundled CLI it also resumes an agent that has finished or been stopped ("Resuming agent …"), which would restart a child outside any `Agent` call, with no slot, no budget check and no card.
+
+**When a slot comes back.** Only when the child's card closes: its typed result, an error result, or the turn ending. A launch placeholder (`async_launched`, `remote_launched`) does not free it. The rewrite sets the call's `run_in_background`, but the bundled CLI also backgrounds a child whose agent definition says `background: true` (it computes "run async" as the call's wish **or** the definition's `background` flag), and a trusted project's `.claude/agents/` can say so. That child keeps its slot, and its card stays open, until its `task_notification` (matched by `tool_use_id`) says how it ended. The cap therefore holds whatever a definition asks for. A definition's own `isolation` is honoured by the CLI the same way; it is the trusted project's own configuration, and only the call's `isolation` is stripped. Refusing every `subagent_type` outside our roster was considered and rejected: a trusted project's own agents are a feature (#23), and a project file can reuse a built-in agent's name anyway.
+
+**The budget check within a turn.** Each child's ledger row is written the moment its card closes (the engine's `onSubagentDone`, wired to `createSubagentLedger` in `src/lib/costs/subagent-ledger.server.ts`), so the next child's `enforceBudgetGuard` sees every child of the turn that has already finished. Siblings still running and the parent's own calls in the turn are not in the figure until they are booked.
+
+**Verified in the installed SDK (0.3.278, bundled CLI 2.1.278).** From the typings: `AgentInput` (`sdk-tools.d.ts`) carries `subagent_type`, `model`, `run_in_background`, `isolation` and `mode`; `AgentOutput` is `completed` (with `content`, `totalTokens`, `totalDurationMs`, `totalToolUseCount`, `usage`, `resolvedModel`) or an `async_launched` / `remote_launched` placeholder; `PreToolUseHookInput` carries `agent_id` only inside a subagent; `result.usage` is the main loop only while `modelUsage` covers subagents. From the bundled CLI: a PreToolUse `updatedInput` with no `permissionDecision` is applied as a plain input change; `AgentOutput.usage` and `totalTokens` describe the child's last model call, not its whole spend; a foreground child runs on the parent turn's own abort controller while a background one gets a fresh one; on an interrupt, the CLI answers every tool call still running (a child included) with an error result of its own (`[Request interrupted by user for tool use]`, or a synthetic cancellation from `StreamingToolExecutor.createSyntheticErrorMessage`) before the turn's `result`; and each assistant message of a model call (one per content block) is yielded as soon as its block ends, carrying the usage known at that moment, while the call's final output count is patched onto those same objects when `message_delta` arrives. Whether a forwarded copy carries the final count therefore depends on when it is written out, which is why the tally below takes the larger report rather than the first or the last.
+
+**Cancellation.** Stop calls `interrupt()` on the run's handle, which aborts the parent turn and, through the shared abort controller, every foreground child; the engine's `close()` then ends the CLI process. `perTaskStopAffordance` is never declared, so even a background task would be killed by the interrupt. The engine records that it sent the interrupt, and an error result for a delegation after that (or one carrying the CLI's interrupt text) closes the card as `stopped` rather than `failed`.
+
+**The card.** Each delegation opens a `subagent` stream block at the parent's `Agent` call (`src/lib/engine/subagent-block.ts`), so a refused child still has a card. The delegation gets no `tool` block of its own. The child's messages, routed by `parent_tool_use_id`, build an ordered transcript on the block (`src/lib/engine/subagent-transcript.ts`, capped at 200 entries and 20,000 characters). The delegation's typed result closes it with `details` (`SubagentDetails` in `tool-result-details.ts`); a card still open when the turn ends is closed as `stopped`. The block's `usage` is what the child spent over all its model calls (`src/lib/engine/subagent-usage.ts`): each forwarded child assistant message and each `message_start` / `message_delta` stream event is counted per `message.id`, every field taking the larger of the reports for the same call (so a call split over several messages counts once, and a provisional output count is replaced by the final one when either reaches the stream), and the typed result's `usage` is folded in as the last call. The card's token count and the ledger row both read it. All new block fields are optional jsonb, so no migration.
+
+**Cost.** See [../cost/spec.md](../cost/spec.md): one `subagent` ledger row per child that spent anything, for all its calls, written when its card closes and carved out of the parent's row.
+
+**Not built.** Child `chat_runs` rows (and with them a run-tree view), a per-child stop control, and worktree isolation for a child, which needs a merge-back and cleanup story first.
 
 ### Subagent output is data, not instructions