diff --git a/README.md b/README.md index 4d445923..230c3f4f 100644 --- a/README.md +++ b/README.md @@ -1663,6 +1663,7 @@ quest agent start --harness claude --model claude-opus-5-5 --effort high --permi quest agent resume --harness claude --model claude-opus-5-5 --effort high --permission-mode auto .worktrees/foo "Take the recommendation." quest agent stop .worktrees/foo quest agent stop --all +quest agent admit ``` Codex starts and resumes only in a checkout registered by `quest worktree create`, @@ -1676,6 +1677,20 @@ pruned dependencies when the next turn needs them. Claude continues to run directly in the supplied checkout. Codex desktop threads use the desktop app's native archival instead. +Start and resume first wait until the host can carry one more agent, before +claiming the worktree, and a coordinator runs `quest agent admit` before it +starts or resumes a sub-agent of its own. The host has room while `quest gate` +would admit a command at once, counting every command queued for it and, as one +recent admission each, every agent admitted in the last 5 minutes: an agent adds +no gated load until its first build, and three quarters of those that build at +all reach it by then. An agent `start` or `resume` runs stops counting when it +exits. As in the gate, a host loaded only by work no gate holds still starts one +agent at a time. + +Admission only delays the next agent: none already running is paused or +stopped, and `quest agent stop` does not see a launch still waiting, which ends +when its command is cancelled. + Options go before the worktree, and the prompt or message follows it, so text starting with `-` is never read as an option. The agent runs in the worktree with the user's environment minus the coordinating session's harness variables diff --git a/assets/AGENTS.md b/assets/AGENTS.md index a5a2b322..199341d6 100644 --- a/assets/AGENTS.md +++ b/assets/AGENTS.md @@ -192,7 +192,8 @@ Relay no interim report: resume a sub-agent that ended its turn on one, telling A long-running coordinator lets compaction bound its context: what must outlast it lives in the forge and the quest tree, and a PR's state is read afresh before acting, not trusted from earlier turns. Start each sub-agent in the background with its own isolated worktree, since agents cannot write to a worktree their parent created. -It reports only when done or blocked, and never waits on GitHub CI, reviews, or merges; stay silent on its interim notifications. +Before starting or resuming one, run `quest agent admit` in the background and go on once it exits 0, handling reports meanwhile: it waits until the host has room for another agent. `quest agent start` and `resume` admit themselves. +A sub-agent reports only when done or blocked, and never waits on GitHub CI, reviews, or merges; stay silent on its interim notifications. As each reports, explain its result in a few lines: its PR, whether it is still a draft, and its open decisions with recommendations. Once two agents report the same blocker, such as a check failure that reproduces on the base or a reviewer outage, name it with its evidence in each later agent's brief, and report it once. diff --git a/assets/CONTEXT.md b/assets/CONTEXT.md index ec3e8439..77d6033a 100644 --- a/assets/CONTEXT.md +++ b/assets/CONTEXT.md @@ -44,7 +44,7 @@ Before ending a task, pausing for a decision, or handing off, commit a checkpoin Finish with a clean checkout: intended changes committed and pushed, incidental generated changes restored, and pre-existing or unrelated changes preserved. Run CPU-heavy commands, such as builds, tests, model checkers, and browsers, through `quest gate `, which queues until the host has room; model calls like `quest review` and servers stay out. -Never work around the queue by polling the load average or capping agents yourself; cancel a gated command only when you no longer need its result. +Never work around the queue by polling the load or capping agents, `quest agent admit`'s job; cancel a gated command only once its result is unneeded. Run it in the foreground and read its exit code directly, not through a pipe. Never end a turn or report while your background work runs, and never poll it with `sleep` or status checks: block on it with the harness's own wait for that task, at the longest interval it allows, saying nothing between waits. To wait on a PR's CI or reviews, use `quest wait`, re-running one that times out with the `--since` it printed, so activity during the first wait is still reported. diff --git a/assets/skills/run.md b/assets/skills/run.md index 942f969c..573d4588 100644 --- a/assets/skills/run.md +++ b/assets/skills/run.md @@ -16,7 +16,7 @@ A brief gives the task, its reason and context, such as the quest or PR and the Starting quests asks the user nothing: each agent works headless, and its decisions surface in its PR. -`quest unblocked --unclaimed --json` ranks the unclaimed ready quests, reading the upstream tree afresh; start them in that order, as many at once as the harness allows. +`quest unblocked --unclaimed --json` ranks the unclaimed ready quests, reading the upstream tree afresh; start them in that order, each once `quest agent admit` admits it, as the guide's Coordination section describes. It lists each skipped claim on stderr, with how long ago it was last active when it has an open PR; name any that look stale in the summary, for the user to take over. Start every quest you would recommend: well planned, with no blockers, and no conflict with work already started. For one waiting on an outside condition, first check whether it has cleared. diff --git a/docs/context.md b/docs/context.md index 37285247..6324350c 100644 --- a/docs/context.md +++ b/docs/context.md @@ -44,7 +44,7 @@ Before ending a task, pausing for a decision, or handing off, commit a checkpoin Finish with a clean checkout: intended changes committed and pushed, incidental generated changes restored, and pre-existing or unrelated changes preserved. Run CPU-heavy commands, such as builds, tests, model checkers, and browsers, through `quest gate `, which queues until the host has room; model calls like `quest review` and servers stay out. -Never work around the queue by polling the load average or capping agents yourself; cancel a gated command only when you no longer need its result. +Never work around the queue by polling the load or capping agents, `quest agent admit`'s job; cancel a gated command only once its result is unneeded. Run it in the foreground and read its exit code directly, not through a pipe. Never end a turn or report while your background work runs, and never poll it with `sleep` or status checks: block on it with the harness's own wait for that task, at the longest interval it allows, saying nothing between waits. To wait on a PR's CI or reviews, use `quest wait`, re-running one that times out with the `--since` it printed, so activity during the first wait is still reported. diff --git a/quest/m0/agent-admission.md b/quest/m0/agent-admission.md deleted file mode 100644 index f1e3bcd8..00000000 --- a/quest/m0/agent-admission.md +++ /dev/null @@ -1,96 +0,0 @@ -# [L] Coordinators start an agent only when the host and the account can carry it - -## Goal - -Every skill that starts agents, such as `/quest-run`, `/quest-spawn`, -`/quest-complete`, and `/quest-audit`, and `quest agent start` and -`quest agent resume`, starts or resumes an agent only while the host's gate -would admit work at once and the account's remaining usage covers the agents -already in flight. A fan-out, including the resumes after a usage limit -resets, never dies en masse on a usage limit, and a started agent rarely -stalls in the gate with its prompt cache going cold. Agents already running -are never paused or stopped by this; it only delays starting the next one. - -Non-goals: capping the user's own top-level sessions, querying a -particular credential gateway, and changing how the gate admits commands. - -## Plan - -Found 2026-11-12 from transcripts on the maintainer's three Linux hosts, -2026-10-22 to 11-12: - -- Fan-outs peaked at 20 to 26 concurrent sub-agents per session (Claude - Code caps native sub-agents at 20 per session, "Concurrent subagent limit - reached"), and up to 59 top-level sessions were active on one host within - five minutes. Nothing caps headless `quest agent start` agents or Codex. -- The shared quota is the main harm: 157 "All credentials ... cooling - down" errors on one host, 143 of them on the day of the 59-session peak; - weekly and session limits hit on several days; 156 background agents - ended "terminated early due to an API error", 26 of them within seconds - in one moq session. -- Gate waits: of 2,482 gated commands on one host, 191 waited over 10 - minutes, 77 over 30, and 11 were still queued at the 55-minute command - timeout. A waiting agent's turn blocks, so its cache can expire. -- Agent processes themselves are cheap: about 100 to 250 MB and 1% CPU - each. - -Decided 2026-11-12 (maintainer, /quest-plan): - -- What outcome? ✅ Admission on both quota and host (rejected: quota only, - which leaves the gate waits; host only, which lets several hosts drain one - account; a fixed smaller fan-out in skill text, which is blind to quota - and load). -- What says the host can carry one more agent? ✅ The gate would admit a - command at once (rejected: a fixed agent count per host, a guess; a - count derived from cores and memory, a poor proxy for gated load). It - targets the measured harm and adds no constant. -- What says the account can? ✅ The usage each harness reports, such as - Codex's `rate_limits.used_percent` in its session log and Claude's - 5-hour and weekly utilization where its login or gateway exposes it. - Stop starting once the budget left cannot cover what the agents in flight - are expected to spend, estimated from `quest usage`'s per-size history - (rejected: asking the credential gateway's management API, which ties the - core to one gateway against AGENTS.md; starting in small waves with no - signal). Decide at a coordinator's start which setups give a signal, and - say once when one gives none, rather than trying and falling back. -- Milestone: ✅ m0 (rejected: m1), since both harms are live. - -What the decisions above require, found in review: - -- "Would admit at once" is the gate's own admission decision (load, - memory, and the headroom recent admissions take), not an empty queue: a - host can be full with nothing queued, and an agent just started adds no - gated load until its first build minutes later. So agents started but - not yet building must count against the headroom too, as the gate counts - its recent admissions, or a whole fan-out passes the check at once. -- Harnesses report a percentage of a window, while `quest usage` reports - tokens and list cost, and neither reports the window's capacity. Convert - through the account's own history, such as tokens spent per percentage - point over recent sessions, or an absolute budget where a harness exposes - one. - -Agent's choices, for the implementer to revisit: - -- Admission waits in the coordinator before it starts or resumes an agent, - never inside a started agent, so only one cheap context waits. `quest - agent start` and `quest agent resume` can admit themselves; native - sub-agents need a step the coordinator runs first, since Claude Code's - `SubagentStart` hook fires once the sub-agent exists and runs `quest - hook` by default (`quest worktree hook` only with `quest setup - --worktrees`). Keep the coordinator able to handle reports while it - waits. -- Verify whether Claude's utilization reaches a session through the - maintainer's credential-pool gateway (`ANTHROPIC_BASE_URL` on a local - port); if not, that setup reports no signal rather than a wrong one. -- `assets/CONTEXT.md` tells agents "never work around the queue by - polling the load average or capping agents yourself". Keep the ban on - agents improvising caps, and point to the admission instead. The - "as many at once as the harness allows" line in `assets/skills/run.md` - changes with it. -- Measure on a real `/quest-spawn` fan-out: limit errors, agents lost, - gate wait times, and total wall time, against the numbers above. - -## Related - -- [The gate records each run for the process that started the agent](/quest/m0/gate-record.md) - per-agent gate waits, to measure this against -- [Cut agent token waste](/quest/m0/token-waste/README.md) - kept the 1-hour cache TTL and capped commands just under it, which a long gate wait can still outlast diff --git a/quest/m0/gate-memory.md b/quest/m0/gate-memory.md deleted file mode 100644 index d349bced..00000000 --- a/quest/m0/gate-memory.md +++ /dev/null @@ -1,62 +0,0 @@ -# [S] A gated command cannot exhaust the host's memory - -## Goal - -On Linux hosts where the gate owns each command in a systemd scope, a -gated command that grows past its memory ceiling is throttled and then -killed as a whole group, instead of the kernel's OOM killer picking a -victim elsewhere on the host. Other gated commands, agent sessions, and -host services keep running. - -Non-goals: host-wide swap or oomd settings, which kixel-ai/nixos's -quest tree plans for the maintainer's host. - -## Plan - -Found 2026-11-12 in read-only audits of the maintainer's three Linux -hosts: the kernel OOM killer fired 5 times in 10 days on one host, killing a -47 GB `rust2-ts` inside a gate scope, a 36 GB `lean`, and a 38 GB gated -`java` while a different process was the one allocating. earlyoom killed -34 processes in a week on another, including HOL Light, lean, and an -Apalache run with a 16 GB heap. Gate scopes start with no memory limit -(`systemd-run` in `src/cli/supervise.ts`). - -Decided 2026-11-12 (maintainer, /quest-plan): - -- ✅ The gate's scopes get a memory ceiling and kill the whole group on - OOM (rejected: leaving it to host-wide oomd alone, which picks victims by - pressure, not by which command overran; rules telling agents to cap - heaps themselves). -- Milestone: ✅ m0 (rejected: m1). - -What the decision requires, found in review: - -- The ceilings of commands running at once must fit the host together: - per-command ceilings each sized from the memory available at admission - overcommit, since several commands admitted together each see the same - free memory. Partition one host budget (memory less a reserve for - sessions and services) across admitted commands, as the gate's memory - admission already counts each command's share (`COMMAND_MEMORY` in - `src/cli/gate.ts`), and admit a command only when its ceiling fits. -- A nested gate's scope is a sibling bound to its enclosing scope - (`BindsTo`), not a child cgroup, so it does not inherit the parent's - limit: its usage must come out of the enclosing command's share. - -Agent's choices, for the implementer to revisit: - -- Set `MemoryHigh` below `MemoryMax`, so a command slows under reclaim - before it is killed, and `OOMPolicy=kill`, so an OOM kill inside the - scope stops the whole command rather than leaving its wrappers running. -- Size a command's share from its measured peak once the gate has seen - it, rather than a constant (moq-dev/quest#673 dropped the matching CPU - plan); a command - that needs more fails loud with the scope's OOM event in the gate's - error, so the agent learns why. - -## Hosts - -- linux - systemd scopes carry the memory limits - -## Related - -- [Coordinators start an agent only when the host and the account can carry it](/quest/m0/agent-admission.md) - the agent-level half of host admission diff --git a/src/cli/admission.ts b/src/cli/admission.ts new file mode 100644 index 00000000..4fc0d169 --- /dev/null +++ b/src/cli/admission.ts @@ -0,0 +1,94 @@ +// Agent admission: a coordinator starts or resumes an agent only while the host's gate would admit +// work at once. It only delays the next start: agents already running are never paused or stopped. +import { randomUUID } from "node:crypto"; +import { basename, dirname, join } from "node:path"; + +import { + type Admission, + type Host, + type Limits, + SETTLE_MS, + type Waiting, + claimDirectory, + fileHost, + hostLimits, + processStart, + room, +} from "./gate"; + +/** + * How long an admitted agent counts against the host's headroom, as one recent admission of the + * gate's: an agent adds no gated load until its first build. Of 263 sub-agents on one host over three + * weeks, the 126 that gated anything reached their first gated command in a median of 133 seconds, + * and three quarters within 265; the rest never gated, so no later event ends the count. + */ +export const PENDING_MS = 5 * 60_000; + +/** What `admit` reads and writes, replaceable in tests. */ +export interface AdmissionDependencies { + limits: Limits; + /** The gate's ledger and the host's load and memory. */ + gate: Host; + /** The agents' ledger; only its clock, ledger, liveness, and sleep are used. */ + agents: Host; + /** A line for the user. */ + notice(text: string): void; +} + +const waits: Record = { + load: "the load average to fall", + memory: "free memory", + queue: "gated commands queued on this host to start", +}; + +/** + * Wait until the host can carry one more agent, then record it. The host has room while the gate + * would admit a command at once (see `room`), counting each agent admitted in the last PENDING_MS + * whose launcher, if it has one, still runs. + * + * `held` is the process that runs the agent, as `quest agent start` does; without it, as for a + * coordinator's own sub-agent, nothing reports the agent's end. + */ +export async function admit(deps: AdmissionDependencies, signal: AbortSignal, held?: number): Promise { + const holder = held === undefined ? {} : { pid: held, start: processStart(held) }; + const shown = new Set(); + for (;;) { + signal.throwIfAborted(); + const ledger = deps.agents.ledger(); + const now = deps.agents.now(); + // A clock stepped backward dates an entry in the future; restart it from now, so it still expires. + if (ledger.admissions.some((entry) => entry.at > now)) { + deps.agents.publish( + ledger, + ledger.admissions.map((entry) => (entry.at > now ? { ...entry, at: now } : entry)), + ); + continue; + } + const pending = ledger.admissions.filter( + (entry) => entry.at > now - PENDING_MS && (entry.pid === undefined || deps.agents.running(entry)), + ); + const reason = room(deps.limits, deps.gate, pending.length); + if (reason === null) { + const entry: Admission = { id: randomUUID(), at: now, ...holder }; + if (deps.agents.publish(ledger, [...pending, entry])) return; + continue; + } + if (!shown.has(reason)) { + shown.add(reason); + deps.notice(`waiting for ${waits[reason]} before starting an agent`); + } + await deps.agents.sleep(SETTLE_MS, signal); + } +} + +/** The agents' ledger of this host, beside the gate's. */ +const agentsDirectory = (gate: string) => join(dirname(dirname(gate)), "agents", basename(gate)); + +/** The admission `quest agent` runs with: the host's own limits and ledgers. */ +export function processAdmission(notice: (text: string) => void): AdmissionDependencies { + const gate = claimDirectory(); + return { limits: hostLimits(), gate: fileHost(gate), agents: fileHost(agentsDirectory(gate)), notice }; +} + +/** The admission every `quest agent` command runs, held in an object so a test can replace it. */ +export const admission = { admit }; diff --git a/src/cli/agent.ts b/src/cli/agent.ts index 63c4e1a5..4883e125 100644 --- a/src/cli/agent.ts +++ b/src/cli/agent.ts @@ -24,7 +24,9 @@ import { constants, tmpdir } from "node:os"; import { join } from "node:path"; import { setTimeout as sleep } from "node:timers/promises"; +import { admission, processAdmission } from "./admission"; import { discoveryEnv, git } from "./git"; +import { cancelled, interrupts } from "./interrupt"; import { LIMIT } from "./limit"; import { alive, programCommand, programEnvironment, requireProgram, signalProcess } from "./program"; import { systemdScopes } from "./supervise"; @@ -386,14 +388,39 @@ export interface AgentDependencies { base: string; stdout(text: string): void; stderr(text: string): void; + /** Wait until the host can carry one more agent, run by process `held` if any (see admission.ts). */ + admit(signal: AbortSignal, held?: number): Promise; } export const processDependencies = (out: Pick): AgentDependencies => ({ env: process.env, base: agentDirectory(), ...out, + admit: (signal, held) => + admission.admit( + processAdmission((text) => out.stderr(`quest: ${text}\n`)), + signal, + held, + ), }); +/** + * Admit one agent before it starts, for `quest agent admit` and every launch. Returns undefined once + * admitted, else the signal's exit status when interrupted while waiting. + */ +export async function admitAgent(deps: AgentDependencies, held?: number): Promise { + const cancel = interrupts(); + try { + await deps.admit(cancel.signal, held); + return undefined; + } catch (error) { + if (cancel.signal.aborted) return cancelled(cancel.signal); + throw error; + } finally { + cancel.close(); + } +} + /** How much of an agent's output is kept for diagnostics; the rest is read as it streams. */ const TAIL = 64 * 1024; const tail = (value: string) => (value.length > TAIL ? value.slice(-TAIL) : value); @@ -432,7 +459,8 @@ export function boundedLines(limit: number, read: (line: string) => void) { } /** - * Run one agent turn in the foreground: a new session with `prompt`, or + * Run one agent turn in the foreground, once the host can carry it + * (see `admitAgent`): a new session with `prompt`, or * `session` resumed with it. Prints the session id as soon as it is known, * then the outcome and final reply. Returns the exit status. */ @@ -467,6 +495,9 @@ export async function runAgent( const program = requireProgram(name, deps.env); const command = checkouts ? undefined : programCommand(program, args, deps.env); const dir = spawnDirectory(deps.base, worktree); + // Waited out before the worktree is claimed, so a queued launch holds nothing. + const refused = await admitAgent(deps, process.pid); + if (refused !== undefined) return refused; const reader = harness.reader(); // Claude Code takes the id this command assigns; Codex announces its own. diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 2fdaa6d6..4e20fa73 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -413,6 +413,44 @@ export function fileHost(directory: string, now = Date.now): Host { }; } +/** + * The load and memory a caller is admitted against while the `running` admissions hold the host: + * the load counts as at least the CPUs they kept busy, and without one the raw load counts as + * having room (see `admit`). A priority caller counts no load. + */ +function standing(limits: Limits, host: Host, running: Admission[], priority = false) { + const used = priority ? 0 : running.reduce((sum, admission) => sum + (admission.cores ?? 0), 0); + const load = priority ? 0 : running.length > 0 ? Math.max(host.load(), used) : Math.min(host.load(), limits.load); + return { used, load, memory: host.memory() }; +} + +/** + * Why a command arriving now would wait, or null when the gate would admit it at once even with + * `reserved` more recent admissions counted against the headroom: work admitted elsewhere that has + * not reached the gate yet, such as an agent still reading (see admission.ts). It counts every + * queued caller as ahead, and never the spaced admissions `admit` grants on raw room, which would + * pass the reserved work one by one. Like the gate, it admits a lone arrival whatever the load: + * with nothing gated running, recent, queued, or reserved, the load is work no gate can wait out. + */ +export function room(limits: Limits, host: Host, reserved: number): Waiting | null { + const ledger = host.ledger(); + const now = host.now(); + const live = ledger.admissions.filter((admission) => host.running(admission)); + const queued = live.filter( + (admission) => admission.queued && (admission.start !== undefined || admission.at > now - RECENT_MS), + ).length; + const admissions = ledger.admissions.filter((admission) => !admission.queued); + const running = admissions.filter((admission) => live.includes(admission)); + const recent = admissions.filter((admission) => admission.at > now - RECENT_MS).length; + const { used, load, memory } = standing(limits, host, running); + if (load > limits.load || used >= limits.load) return "load"; + if (memory < limits.memory) return "memory"; + const taken = recent + queued + reserved; + if (running.length === 0 && taken === 0) return null; + if (load + taken >= limits.load) return queued > 0 ? "queue" : "load"; + return memory - taken * limits.memory >= limits.memory ? null : queued > 0 ? "queue" : "memory"; +} + /** * Wait until the host has room for a command, then record its admission. The * load average and available memory lag new work, so each admission from the @@ -510,13 +548,7 @@ export async function admit( // Recorded like any admission, so it still counts against the others' headroom. if (now >= deadline) return admitted(now); // Priority ignores load, whether or not a gated command runs; memory still applies. - const used = precedence.priority ? 0 : [...running].reduce((sum, admission) => sum + (admission.cores ?? 0), 0); - const load = precedence.priority - ? 0 - : running.size > 0 - ? Math.max(host.load(), used) - : Math.min(host.load(), limits.load); - const memory = host.memory(); + const { used, load, memory } = standing(limits, host, [...running], precedence.priority); let wait = SETTLE_MS; if (load > limits.load || used >= limits.load) waiting?.("load"); else if (memory < limits.memory) waiting?.("memory"); diff --git a/src/cli/main.ts b/src/cli/main.ts index 1cd95c8b..bbfc2115 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -23,8 +23,18 @@ import { onMaintainer, waiting, } from "../core"; -import { AgentInputError, HARNESSES, type Launch, processDependencies, runAgent, stopAgent, stopAll } from "./agent"; +import { + AgentInputError, + HARNESSES, + type Launch, + admitAgent, + processDependencies, + runAgent, + stopAgent, + stopAll, +} from "./agent"; import { held, openDrafts, reach, target, upstream, upstreamRefs } from "./claims"; +import { PENDING_MS } from "./admission"; import { gate, QUICK_MS } from "./gate"; import { gitHubDependencies, @@ -1022,6 +1032,8 @@ export function main(argv: string[], out: Output): number | Promise { "The agent runs in with the user's environment minus every CLAUDE* (except CLAUDE_CONFIG_DIR), " + "ANTHROPIC*, CODEX_* (except CODEX_HOME), OP_*, OPENAI_API_KEY, QUEST_GATE_ADMITTED, QUEST_GATE_PRIORITY, " + "QUEST_LAND_BASE, and MBX_SCHEDULER_PRIORITY when QUEST_GATE_PRIORITY=1, so it uses the CLI's stored login rather than the coordinating session's.\n\n" + + "Start and resume first wait until the host can carry the agent, as agent admit describes, " + + "before claiming the worktree.\n\n" + "Prints `session ` as soon as the id is known, then `outcome finished|failed|limited|logged-out|stopped`, " + "a blank line, and the agent's final reply. Exits 0 when the agent finished, 3 when its account hit a " + "usage limit or ran out of credit, 4 when the CLI is logged out (after printing its login command), 2 for " + @@ -1070,6 +1082,21 @@ export function main(argv: string[], out: Output): number | Promise { ).action((worktree: string, session: string, message: string, options: LaunchOptions) => launch(worktree, message, session, options), ); + agent + .command("admit") + .summary("Wait until the host can carry one more agent") + .description( + "Wait until this host can carry one more agent, for a coordinator to run before it starts or resumes " + + "an agent of its own, such as a sub-agent; start and resume do the same themselves. The host has room " + + "while quest gate would admit a command at once, counting every queued command and, as one recent " + + `admission each, every agent admitted in the last ${PENDING_MS / 60_000} minutes, since an agent adds ` + + "no gated load until its first build; an agent start or resume runs counts only until it exits. Agents " + + "already running are never paused.\n\n" + + "Exits 0 once admitted, 128 plus the signal number when interrupted, and 1 otherwise.", + ) + .action(() => { + pending = admitAgent(agentDependencies()).then((status) => status ?? 0, agentFailure); + }); agent .command("stop") .summary("Stop the agent in a worktree, or end the spawn with --all") diff --git a/test/admission.test.ts b/test/admission.test.ts new file mode 100644 index 00000000..c09d0252 --- /dev/null +++ b/test/admission.test.ts @@ -0,0 +1,124 @@ +import { expect, test } from "vitest"; +import { type AdmissionDependencies, PENDING_MS, admit } from "../src/cli/admission"; +import { type Admission, type Host, RECENT_MS, SETTLE_MS, fileHost, processStart } from "../src/cli/gate"; +import { tempDir } from "./fixture"; + +const GiB = 2 ** 30; +const signal = new AbortController().signal; +const me = { pid: process.pid, start: processStart(process.pid) }; + +/** A host of 8 CPUs with real ledgers, and load, memory, a clock, and process liveness the test controls. */ +function host() { + const state = { + now: 10 * PENDING_MS, + load: 0, + memory: Infinity, + dead: new Set(), + stop: Infinity, + }; + const sleeps: number[] = []; + const notices: string[] = []; + const ledger = (directory: string): Host => ({ + ...fileHost(directory, () => state.now), + load: () => state.load, + memory: () => state.memory, + running: (admission) => admission.pid !== undefined && !state.dead.has(admission.pid), + sleep: async (ms) => { + // A wait that outlasts the test's patience ends it, as a cancelled caller's would. + if (state.now >= state.stop) throw new Error("still waiting"); + sleeps.push(ms); + state.now += ms; + }, + }); + const deps: AdmissionDependencies = { + limits: { load: 8, memory: 4 * GiB }, + gate: ledger(tempDir()), + agents: ledger(tempDir()), + notice: (text) => notices.push(text), + }; + const gated = (admissions: Admission[]) => deps.gate.publish(deps.gate.ledger(), admissions); + /** Whether one more agent is admitted without waiting. */ + const atOnce = async (held?: number) => { + state.stop = state.now; + try { + await admit(deps, signal, held); + return true; + } catch (error) { + if ((error as Error).message !== "still waiting") throw error; + return false; + } + }; + return { state, sleeps, notices, deps, gated, atOnce }; +} + +test("a fan-out fills the host's headroom, and each agent frees its share once it would have built", async () => { + const { state, deps, sleeps, notices, atOnce } = host(); + state.load = 5; + // Load 5 plus 0, 1, and 2 agents not yet building stays under 8 CPUs; a fourth would not. + expect([await atOnce(), await atOnce(), await atOnce(), await atOnce()]).toEqual([true, true, true, false]); + // The fourth waits as the gate's callers do, and starts once the first three no longer count. + state.stop = Infinity; + await admit(deps, signal); + expect(sleeps.reduce((sum, ms) => sum + ms, 0)).toBe(PENDING_MS); + expect(new Set(sleeps)).toEqual(new Set([SETTLE_MS])); + // Each waiting call names what it waits for once, however long it waits. + expect(notices.filter((text) => text.startsWith("waiting"))).toEqual( + Array(2).fill("waiting for the load average to fall before starting an agent"), + ); +}); + +test("each agent not yet building takes one command's memory", async () => { + const { state, atOnce } = host(); + state.memory = 9 * GiB; + expect([await atOnce(), await atOnce(), await atOnce()]).toEqual([true, true, false]); +}); + +test("a queued or running gated command holds agents back as it would another command", async () => { + const { state, gated, atOnce, notices } = host(); + state.load = 6; + // One command queued for the gate: an arrival would wait behind it only once the host is short. + gated([{ id: "queued", at: state.now, ...me, queued: true }]); + expect([await atOnce(), await atOnce()]).toEqual([true, false]); + expect(notices.at(-1)).toContain("gated commands queued"); + // Running commands that kept every CPU busy hold the host whatever the load average says. + state.load = 0; + state.now += PENDING_MS; + gated([{ id: "build", at: state.now - RECENT_MS, ...me, cores: 8 }]); + expect(await atOnce()).toBe(false); + // A recent gated admission counts once, as the gate counts it. + gated([{ id: "build", at: state.now, ...me, cores: 1 }]); + state.load = 6; + expect([await atOnce(), await atOnce()]).toEqual([true, false]); +}); + +test("a host loaded by work no gate holds still starts one agent at a time", async () => { + const { state, atOnce } = host(); + state.load = 40; + expect([await atOnce(), await atOnce()]).toEqual([true, false]); + state.now += PENDING_MS; + expect(await atOnce()).toBe(true); + // Memory is never waived. + state.now += PENDING_MS; + state.memory = GiB; + expect(await atOnce()).toBe(false); +}); + +test("an agent whose launcher exited no longer counts, and one still running counts only while it may yet build", async () => { + const { state, atOnce } = host(); + state.load = 7; + expect([await atOnce(4242), await atOnce()]).toEqual([true, false]); + state.dead.add(4242); + expect(await atOnce()).toBe(true); + state.now += PENDING_MS; + expect([await atOnce(4243), await atOnce()]).toEqual([true, false]); + state.now += PENDING_MS; + expect(await atOnce()).toBe(true); +}); + +test("an agent dated in the future by a clock stepped backward counts from now, and still expires", async () => { + const { state, deps, sleeps } = host(); + state.load = 7; + deps.agents.publish(deps.agents.ledger(), [{ id: "ahead", at: state.now + 12 * PENDING_MS }]); + await admit(deps, signal); + expect(sleeps.reduce((sum, ms) => sum + ms, 0)).toBe(PENDING_MS); +}); diff --git a/test/agent.test.ts b/test/agent.test.ts index c175554a..30e26637 100644 --- a/test/agent.test.ts +++ b/test/agent.test.ts @@ -12,8 +12,9 @@ import { writeFileSync, } from "node:fs"; import { delimiter, join } from "node:path"; -import { afterEach, expect, test, vi } from "vitest"; +import { afterEach, beforeEach, expect, test, vi } from "vitest"; +import { admission } from "../src/cli/admission"; import { boundedLines, spawnDirectory, strippedEnvironment } from "../src/cli/agent"; import { batchArgument, programCommand, programEnvironment, signalProcess, which } from "../src/cli/program"; import { type Output, main } from "../src/cli/main"; @@ -21,6 +22,11 @@ import { tempDir } from "./fixture"; import { systemdScopes } from "../src/cli/supervise"; import { Worktrees } from "../src/cli/worktree"; +// Launches here are admitted at once, whatever the host running the tests carries. +beforeEach(() => { + vi.spyOn(admission, "admit").mockResolvedValue(); +}); + afterEach(() => { vi.unstubAllEnvs(); vi.restoreAllMocks(); @@ -365,6 +371,29 @@ test("an account at its usage limit exits 3, and the session resumes once it res expect(call.stdin).toBe("continue"); }); +test("a launch waits for admission before it claims the worktree", async () => { + const s = setup(); + vi.spyOn(admission, "admit").mockImplementation(async (deps, signal, held) => { + expect([held, signal.aborted]).toEqual([process.pid, false]); + // Nothing holds the worktree while the launch waits. + const state = join(s.runtime, "quest", "agent"); + expect(readdirSync(join(state, readdirSync(state)[0], "agents"))).toEqual([]); + deps.notice("waiting for the load average to fall before starting an agent"); + }); + const started = await quest("agent", "start", ...launch("claude"), s.worktree, "go"); + expect(started).toMatchObject({ + code: 0, + stderr: "quest: waiting for the load average to fall before starting an agent\n", + }); +}); + +test("agent admit exits 0 once admitted", async () => { + const admit = vi.spyOn(admission, "admit").mockResolvedValue(); + expect(await quest("agent", "admit")).toEqual({ code: 0, stdout: "", stderr: "" }); + // No process runs a coordinator's own sub-agent, so nothing holds its admission. + expect(admit.mock.calls[0][2]).toBeUndefined(); +}); + test("a spent free usage allowance exits 3 as limited", async () => { const s = setup(); vi.stubEnv("FAKE_MODE", "free-limited");