From 8e9425ba57ab6d25df3de6495da9c0411605f341 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 19:31:18 -0800 Subject: [PATCH 01/13] chore(quest): claim quest/m0/gate-record.md Quest-Claim: a89988caa7d992c8 From a122961a377c92da8c564ea0680d450406747663 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 19:53:11 -0800 Subject: [PATCH 02/13] feat(gate): record each outermost run to QUEST_GATE_RECORD With QUEST_GATE_RECORD naming an absolute path, every outermost gate on Linux and macOS appends one JSON line per run, however it ends: command, directory, started/admitted/ended timestamps, CPU time, peak memory, what measured them, exit status, cancellation, and the gate's own error. A systemd scope is now delegated: its handshake shell stays at the root as a holder and runs the command in a child cgroup, so the scope's cpu.stat and memory.peak outlive the command until the owner has stopped its leftovers and read them. A process group owner starts its command through Bun.spawn for the command's rusage. Co-Authored-By: Claude Opus 5.5 --- README.md | 41 ++++- quest/m0/agent-admission.md | 4 +- quest/m0/gate-record.md | 114 ------------- quest/m1/gate-record-windows.md | 12 +- src/cli/gate.ts | 277 ++++++++++++++++++++++++-------- src/cli/main.ts | 9 +- src/cli/supervise.ts | 185 +++++++++++++++++---- src/cli/worktree.ts | 2 +- test/agent.test.ts | 2 +- test/gate.test.ts | 231 +++++++++++++++++++++++--- test/worktree.run.test.ts | 4 +- test/worktree.test.ts | 10 +- 12 files changed, 635 insertions(+), 256 deletions(-) delete mode 100644 quest/m0/gate-record.md diff --git a/README.md b/README.md index a578d9fc1..15395ce28 100644 --- a/README.md +++ b/README.md @@ -1238,7 +1238,8 @@ gate exits with 128 plus the signal's number. How far ownership reaches depends on the host: - Linux with a systemd 254+ user manager, unified cgroup v2, and Linux 5.14+ - runs the command in its own systemd user scope in `background.slice`, whose + runs the command in a child cgroup of its own delegated systemd user scope + in `background.slice`, whose low CPU weight keeps the desktop session ahead of gated work under contention, and owns every descendant, including ones that start their own process group or session. Only a process that asks another service manager to start it @@ -1287,6 +1288,44 @@ arrive together, it takes a third; a repeated Ctrl-C, which Just does not forward, kills at the second. There is no timeout except `--quick`'s: any other command waits its turn for as long as the host stays busy. +### Record each run + +A process that starts an agent, such as a job runner or a CPU audit, can read +back what the agent's gated commands waited for and cost by setting +`QUEST_GATE_RECORD` in the agent's environment to an absolute path. Every +outermost `quest gate` run on Linux or macOS, including those the agent's +parallel shells and subagents start, then appends one JSON line to that file, +however the run ends: + +```json +{"command":["cargo","test"],"directory":"/home/me/src/app","started":"2026-11-12T19:40:01.120Z","admitted":"2026-11-12T19:40:31.452Z","ended":"2026-11-12T19:42:10.008Z","cpuMs":512340,"peakMemory":3221225472,"measured":"cgroup","exit":0,"cancelled":false,"error":null} +``` + +- `started`, `admitted`, and `ended` give the wait and the wall time. `ended` is + when the command's own process exited, before the gate stopped what it left + behind. A run cancelled before admission has a null `admitted`. +- `cpuMs` and `peakMemory` (bytes) are null when the command never ran, and + `peakMemory` also where the host cannot say. `measured` names what they + count: + - `cgroup`: the command's systemd scope, so every process it started. Peak + memory is the scope's `memory.peak`, which includes page cache charged to + it and needs Linux 5.19. + - `children`: on macOS, and Linux without a systemd user manager, the + command's own resource usage, a lower bound: it covers the descendants the + command waited for, not those the gate kills at the end, such as a leftover + background server, and its peak memory is the largest single process's. +- `exit` is the status the gate returns, `cancelled` whether it was cancelled, + and `error` the gate's own failure, such as a command that cannot start. + +Each line is one write to a file opened for appending, so concurrent gates +leave only whole lines; give each agent its own file to tell them apart. Nested +gates write nothing, since their outermost gate already counts their command. +An unset or empty variable writes nothing, a relative path is refused, and a +file the gate cannot open fails it before the command runs. A line it cannot +write fails the gate too, though a cancelled run keeps its cancelled status. +Windows refuses the variable for now. The agent's own processes write the +file, so a record is only as trustworthy as the agent. + ## Route work by size `quest` ranks the harnesses, models, and efforts that write work of each size, diff --git a/quest/m0/agent-admission.md b/quest/m0/agent-admission.md index 4a408ecf7..bd2735a19 100644 --- a/quest/m0/agent-admission.md +++ b/quest/m0/agent-admission.md @@ -92,10 +92,10 @@ Agent's choices, for the implementer to revisit: "as many at once as the harness allows" line in `assets/skills/run.md` changes with it. - Measure on a real `/quest-spawn` fan-out: limit errors, agents lost, - gate wait times, and total wall time, against the numbers above. + gate wait times (each agent's `QUEST_GATE_RECORD` file), and total wall + time, against the numbers above. ## Related - [The gate admits work through mbx's pool](/quest/m0/gate-mbx-pool.md) - changes how the gate admits commands, which this reads -- [The gate records each run for the process that started the agent](/quest/m0/gate-record.md) - per-agent gate waits, to measure this against - [Cut agent token waste](/quest/m0/token-waste/README.md) - kept the 1-hour cache TTL and capped commands just under it, which a long gate wait can still outlast diff --git a/quest/m0/gate-record.md b/quest/m0/gate-record.md deleted file mode 100644 index 175bae96d..000000000 --- a/quest/m0/gate-record.md +++ /dev/null @@ -1,114 +0,0 @@ -# [M] The gate records each run for the process that started the agent - -## Goal - -When `QUEST_GATE_RECORD` names a file, every outermost `quest gate` run -on Linux or macOS appends one JSON line to it, however the run ends, -with the command, its directory, how long it waited for admission, its -CPU time, wall time, peak memory, and exit status. When it is unset, the -gate writes nothing. - -A process that starts an agent, such as a job runner or a CPU audit, -sets the variable in that agent's environment and reads back what the -agent's gated commands waited for and cost, including those its parallel -shells and subagents ran. Today that takes matching agent transcripts -against systemd journal lines, which hold no wait time and exist only on -Linux. - -Non-goals: how the gate admits commands, which -[moq-dev/quest#667](https://github.com/moq-dev/quest/pull/667) plans -through mbx's pool; Windows, which -[its own quest](/quest/m1/gate-record-windows.md) covers; and any view of -the record in Quest itself. - -## Plan - -Decided 2026-11-12 (maintainer, `/quest-plan`): - -- ✅ An environment variable names the file, and the reader owns it - (rejected: every gate always appending to a per-host log under - Quest's cache directory, tagged with an ID the reader filters by, - which needs rotation and retention for history nothing reads). A - reader that wants each agent apart gives each agent its own file. -- ✅ Linux and macOS here, Windows in m1 (rejected: Windows here too, - since m0 leaves native Windows and its suite to m1; recording only - wait, wall time, and exit status on Windows with no plan for the - rest). The first answer was every platform; review found Windows - conflicts with m0's scope and needs a new channel out of the job stub, - so the maintainer split it. -- ✅ Where the gate owns the command only as a process group (macOS, and - Linux without a systemd user manager), record a labeled lower bound: - the CPU of the processes the gate reaped, and the largest single - process's peak memory (rejected: sampling the group every few seconds, - which costs more code and still misses short-lived processes). The - README says these exclude processes the gate kills at the end, such as - a leftover background server. -- ✅ Only outermost runs are recorded. A gate inside an admitted command - (`QUEST_GATE_ADMITTED`) runs at once and is measured as part of it, - so recording it too would count its CPU twice. -- ✅ m0 (rejected: m1), with the rest of the gate's work. - -Agent's choices, for the implementer to revisit: - -- One line per run: `command` (argv), `directory` (absolute), `started`, - `admitted` (null when the run never was), and `ended` as ISO - timestamps, `cpuMs`, `peakMemory` in bytes (null when the host cannot - say), `measured` (`cgroup` or `children`, naming the source above; the - Windows quest adds `job`), - `exit` (the status the gate returns, 128 plus the signal's number for - a signalled command, as the owners already map it), `cancelled`, and - `error` (the gate's own failure, such as a command that cannot start - or a scope it cannot confirm stopped). Wait and wall time follow from - the timestamps. Nothing is in production, so the format carries no - version. -- `ended` is when the command's own process exits, before the gate stops - descendants it left behind, so cleanup does not count as wall time. -- On Linux with a systemd scope, the counters must outlive the command: - the scope runs with `--collect`, and `vacated` in - `src/cli/supervise.ts` already succeeds when systemd has removed the - emptied cgroup, so a read after the command's tree exits can find - nothing, and the 5-second CPU samples miss a short command and the - last interval. Recommended: the handshake shell stays in the scope as - a holder instead of becoming the command, and runs the command in a - child cgroup of the scope, which is delegated (`Delegate=yes`) with no - controllers enabled below it, so the scope's own `cpu.stat` - `usage_usec` and `memory.peak` cover everything the command started. - The owner stops the command's leftovers with the child's - `cgroup.kill`, reads the scope's counters while the holder keeps it - populated, then releases the holder and returns the counters with the - exit status. The holder writes the command's exit status and the time - it exited to the handshake's report pipe as soon as the command exits, - since the owner otherwise sees only the holder's exit, after cleanup. An implementer may prove another way instead, such as - holding a D-Bus reference to the unit and reading systemd's - `CPUUsageNSec` and `MemoryPeak`; the short-command test below decides. - `memory.peak` needs Linux 5.19, so on 5.14 to 5.18 `peakMemory` is - null. -- On a process group, `cpuMs` and `peakMemory` come from the command's - rusage at exit, which `wait4` reports for it and the descendants it - waited for. Node's `child_process` exposes none; Bun's - `Subprocess.resourceUsage()` does, so the group owner may need to - start its command through `Bun.spawn`. -- Append each line in one write to a file opened for append, since an - agent's parallel shells and subagents share it. -- A cancelled run, including one cancelled while waiting for admission, - still writes its line. A line that cannot be written fails the gate: - it names the file on stderr and exits nonzero, keeping a cancellation's - status when the run was cancelled. -- The record holds what the agent's own processes wrote. The agent's - shell already has the variable and can write the file directly, so - removing it from gated commands' environment would protect nothing; - the README says a record is as trustworthy as the agent. -- Document the variable, the line format, and what each platform counts - in the README's gate section. - -Tests: an outermost run under the variable appends one line whose wait, -CPU, wall time, and exit match a stub command's, including one that ends -before the first CPU sample; a nested gate adds none; an unset variable -writes nothing; a run cancelled while queued for admission is recorded -with no admission; an unwritable file fails the gate; a process group's -line is labeled `children`; and concurrent gates leave only whole lines. - -## Hosts - -- linux - the systemd scope's cgroup accounting -- macos - the process group's rusage diff --git a/quest/m1/gate-record-windows.md b/quest/m1/gate-record-windows.md index 5565b6fbc..8eec780d1 100644 --- a/quest/m1/gate-record-windows.md +++ b/quest/m1/gate-record-windows.md @@ -9,9 +9,10 @@ starts. ## Plan -Decided 2026-11-12 (maintainer, `/quest-plan`): split from the -[gate run record](/quest/m0/gate-record.md), since m0 leaves native -Windows and its suite to m1. +Decided 2026-11-12 (maintainer, `/quest-plan`): split from the gate run +record, since m0 leaves native Windows and its suite to m1. The README's +"Record each run" section gives the variable and the line format; until +this lands, `quest gate` on Windows refuses `QUEST_GATE_RECORD`. The Job Object lives in the Bun stub that `src/cli/job.ts` runs, and it ends with the stub, so the gate never holds it. The stub reads the job's @@ -30,8 +31,3 @@ without a job is labeled `children`. ## Hosts - windows - the Job Object accounting - -## Required - -- [Gate run record](/quest/m0/gate-record.md) - the variable, the line - format, and the Linux and macOS measurement this extends diff --git a/src/cli/gate.ts b/src/cli/gate.ts index d37848194..c4ffac787 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -3,14 +3,44 @@ // the command's descendants until they stop, and holds nothing else while it runs. import { spawn, spawnSync } from "node:child_process"; import { randomUUID } from "node:crypto"; -import { linkSync, lstatSync, mkdirSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { + closeSync, + linkSync, + lstatSync, + mkdirSync, + openSync, + readdirSync, + readFileSync, + rmSync, + statSync, + writeFileSync, + writeSync, +} from "node:fs"; import { availableParallelism, constants, cpus, hostname, loadavg, totalmem, userInfo } from "node:os"; -import { dirname, join, relative, resolve, sep } from "node:path"; -import type { Readable } from "node:stream"; +import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; import { setTimeout as sleep } from "node:timers/promises"; import { type Cancellation, cancelled, follow, forwards, GROUP, interrupts, received, requested } from "./interrupt"; import { stubbed } from "./job"; -import { questScope, scopeGroup, startTime, systemdScopes } from "./supervise"; +import { type Exit, questScope, scopeGroup, startTime, systemdScopes } from "./supervise"; + +/** + * The part of Bun's process API the group owner needs: Node's `child_process` reports no resource + * usage of a child, and Quest always runs on Bun. + */ +declare const Bun: { + spawn( + command: string[], + options: { env: NodeJS.ProcessEnv; stdio: ["inherit", "inherit", "inherit", "pipe"]; detached: true }, + ): { + pid: number; + stdio: [null, null, null, number]; + exited: Promise; + exitCode: number | null; + signalCode: NodeJS.Signals | null; + resourceUsage(): { cpuTime: { total: number | bigint }; maxRSS: number } | undefined; + }; + file(fd: number): { stream(): ReadableStream }; +}; /** * One admitted command: a unique id, the wall-clock time it was admitted, and the process ID of @@ -607,12 +637,13 @@ const ADMITTED = "QUEST_GATE_ADMITTED"; export const PRIORITY = "QUEST_GATE_PRIORITY"; /** - * Runs an admitted command with inherited streams and resolves with its exact exit status. + * Runs an admitted command with inherited streams and resolves with its exact exit status, when it + * exited, and what it used where the owner measures that. * Asking to cancel delivers the requested signal to every process the owner holds, unless they * already have it (see `forwards`), so each one's own handling of it runs; killing stops them * outright. */ -export type Owner = (command: string[], env: NodeJS.ProcessEnv, cancel: Cancellation) => Promise; +export type Owner = (command: string[], env: NodeJS.ProcessEnv, cancel: Cancellation) => Promise; /** * Run the command as a plain child. `held` says whether an enclosing owner that signals everything @@ -638,7 +669,7 @@ function plain(held: boolean): Owner { }); child.once("close", (code, killed) => { cleanup(); - resolve(code ?? (killed ? 128 + constants.signals[killed] : 1)); + resolve({ status: code ?? (killed ? 128 + constants.signals[killed] : 1), ended: Date.now() }); }); }); } @@ -693,24 +724,29 @@ export const job: Owner = (command, env, cancel) => direct(...stubbed(command, e * it or a parent closes it, as Bun and Node do for what they start. Its read end ends once every * holder has exited, so each one, such as a leaf that outlives the wrappers around it, finishes its * own cleanup before the rest of the group is killed, unless a second signal kills it first. + * + * Its usage is the command's own as `wait4` reports it, which covers the descendants it waited for + * but not those the group kills at the end, and as peak memory the largest single process's. */ export const group: Owner = (command, env, cancel) => new Promise((resolve, reject) => { - const child = spawn("nice", ["-n", "10", ...command], { - env: { ...env, [GROUP]: "1" }, - stdio: ["inherit", "inherit", "inherit", "pipe"], - detached: true, - }); - const token = child.stdio[TOKEN] as Readable; - const released = new Promise((resolve) => { - token.once("end", resolve); - token.once("close", resolve); - token.once("error", () => resolve()); - }); - token.resume(); // Holders' writes are discarded; only the end matters. + let child: ReturnType; + try { + child = Bun.spawn(["nice", "-n", "10", ...command], { + env: { ...env, [GROUP]: "1" }, + stdio: ["inherit", "inherit", "inherit", "pipe"], + detached: true, + }); + } catch (error) { + return reject(error); + } + const token = child.stdio[TOKEN]; + // Read only once a cancellation waits for the end; holders' writes are discarded. + let reader: ReadableStreamDefaultReader | undefined; + let released: Promise | undefined; const signal = (name: NodeJS.Signals) => { try { - process.kill(-child.pid!, name); + process.kill(-child.pid, name); } catch (error) { const code = (error as NodeJS.ErrnoException).code; if (code === "ESRCH") return; @@ -725,33 +761,47 @@ export const group: Owner = (command, env, cancel) => if (live) throw error; } }; - let signalled = false; const stop = () => { - signalled = true; + if (!reader) { + const stream = Bun.file(token).stream().getReader(); + reader = stream; + released = (async () => { + try { + while (!(await stream.read()).done); + } catch { + // A failed read ends the wait as the pipe's end would. + } + })(); + } signal(requested(cancel.signal)); }; const terminate = () => signal("SIGKILL"); - child.once("error", reject); - if (child.pid === undefined) return; // It never started. const cleanup = follow(cancel, stop, terminate); const killed = new Promise((resolve) => { if (cancel.kill.aborted) resolve(); else cancel.kill.addEventListener("abort", () => resolve(), { once: true }); }); - child.once("exit", async (code, name) => { + void child.exited.then(async () => { + const ended = Date.now(); + const status = child.exitCode ?? 128 + constants.signals[child.signalCode!]; + const usage = child.resourceUsage(); // Every process in the group got the signal, so let each holder finish its own cleanup. - if (signalled) await Promise.race([released, killed]); + if (released) await Promise.race([released, killed]); cleanup(); - token.destroy(); try { + await reader?.cancel(); + // Bun leaves a pipe it handed out open. + closeSync(token); // The group outlives its leader while any member remains, so its ID is not reused yet. A // group that emptied during the drain could only be mistaken once the host cycled through // every process ID. terminate(); + if (!usage) throw new Error("gate cannot read its command's resource usage"); } catch (error) { return reject(error); } - resolve(code ?? 128 + constants.signals[name!]); + const cpuMs = Math.round(Number(usage.cpuTime.total) / 1000); + resolve({ status, ended, usage: { cpuMs, peakMemory: usage.maxRSS, measured: "children" } }); }); }); @@ -787,56 +837,151 @@ const notices: Record = { queue: "quest gate: waiting for commands queued earlier to start", }; +/** + * Names a file to which every outermost gate appends one JSON line describing its run, however it + * ends (see `gate`). A nested gate runs inside a command its outermost gate already measures. + */ +export const RECORD = "QUEST_GATE_RECORD"; + +/** Open the file `RECORD` names for appending, before anything runs; undefined while it is unset or empty. */ +function recordFile(env: NodeJS.ProcessEnv): { path: string; fd: number } | undefined { + const path = env[RECORD]; + if (!path) return undefined; + if (process.platform === "win32") throw new Error(`gate cannot record its run on Windows yet (${RECORD})`); + // The command's shells change directory, so a relative path would scatter one reader's lines. + if (!isAbsolute(path)) throw new Error(`${RECORD} must be an absolute path: ${path}`); + try { + return { path, fd: openSync(path, "a") }; + } catch (error) { + throw new Error(`gate cannot record its run in ${path}: ${message(error)}`); + } +} + +const message = (error: unknown) => (error instanceof Error ? error.message : String(error)); + +/** How `gate` admits its command. */ +export interface GateOptions { + /** Wait at most QUICK_MS (see `Precedence`). */ + quick?: boolean; + /** The admission ledger and host measures; this user's on this host when omitted. */ + host?: Host; +} + +/** What one gate observed of its run, for its record. */ +interface Observed { + /** Wall-clock milliseconds when the command was admitted. */ + admitted?: number; + exit?: Exit; +} + /** * Run from the CLI. A failure to measure the command is fatal even when the command succeeded, * since every other gate on the host relies on that accounting; a cancelled command still - * reports its cancellation. + * reports its cancellation. With `RECORD` set, an outermost gate appends its run's line in one + * write, so concurrent gates leave only whole lines; a line it cannot write fails the gate, except + * that a cancelled run keeps its cancelled status. */ -export async function gate(command: string[], options: { quick?: boolean } = {}): Promise { +export async function gate(command: string[], options: GateOptions = {}): Promise { + const outermost = process.env[ADMITTED] !== "1"; + const record = outermost ? recordFile(process.env) : undefined; + const started = Date.now(); const cancel = interrupts(); + const observed: Observed = {}; try { - const { run, unit } = owner(); - let admitted: { host: Host; id: string } | undefined; - // A nested invocation executes inside the already admitted command. - if (process.env[ADMITTED] !== "1") { - const shown = new Set(); + let status: number; + let failure: unknown; + try { + status = await gated(command, options, outermost, cancel, observed); + } catch (error) { + failure = error; + // The status the CLI exits with for a gate's own failure. + status = 1; + } + if (record) { + const iso = (ms: number) => new Date(ms).toISOString(); + const usage = observed.exit?.usage; + const line = `${JSON.stringify({ + command, + directory: process.cwd(), + started: iso(started), + admitted: observed.admitted === undefined ? null : iso(observed.admitted), + ended: iso(observed.exit?.ended ?? Date.now()), + cpuMs: usage?.cpuMs ?? null, + peakMemory: usage?.peakMemory ?? null, + measured: usage?.measured ?? null, + exit: status, + cancelled: cancel.signal.aborted, + error: failure === undefined ? null : message(failure), + })}\n`; try { - const notice = (reason: Waiting) => { - if (shown.has(reason)) return; - shown.add(reason); - process.stderr.write(`${notices[reason]}\n`); - }; - const precedence = { - priority: process.env[PRIORITY] === "1", - quick: options.quick, - }; - const host = fileHost(claimDirectory()); - admitted = { host, id: await admit(hostLimits(), host, cancel.signal, notice, precedence) }; + const bytes = Buffer.from(line); + const written = writeSync(record.fd, bytes); + if (written !== bytes.length) throw new Error(`wrote ${written} of ${bytes.length} bytes`); } catch (error) { - if (cancel.signal.aborted) return cancelled(cancel.signal); - throw error; + const unrecorded = `gate cannot record its run in ${record.path}: ${message(error)}`; + if (failure === undefined && cancel.signal.aborted) { + process.stderr.write(`quest ${unrecorded}\n`); + return status; + } + throw new Error(failure === undefined ? unrecorded : `${message(failure)}; ${unrecorded}`); + } finally { + closeSync(record.fd); } } - const stop = new AbortController(); - // Held as a value until the command ends, so a failure never goes unhandled meanwhile. - const measuring = - admitted !== undefined && unit !== undefined - ? measure(admitted.host, { id: admitted.id, unit }, stop.signal).then( - () => undefined, - (error: unknown) => ({ error }), - ) - : undefined; - let result: number; + if (failure !== undefined) throw failure; + return status; + } finally { + cancel.close(); + } +} + +/** Admit the command unless nested, then run it, noting what the record needs in `observed`. */ +async function gated( + command: string[], + options: GateOptions, + outermost: boolean, + cancel: ReturnType, + observed: Observed, +): Promise { + const { run, unit } = owner(); + let admitted: { host: Host; id: string } | undefined; + // A nested invocation executes inside the already admitted command. + if (outermost) { + const shown = new Set(); try { - result = await run(command, { ...process.env, [ADMITTED]: "1" }, cancel); - } finally { - stop.abort(); + const notice = (reason: Waiting) => { + if (shown.has(reason)) return; + shown.add(reason); + process.stderr.write(`${notices[reason]}\n`); + }; + const precedence = { + priority: process.env[PRIORITY] === "1", + quick: options.quick, + }; + const host = options.host ?? fileHost(claimDirectory()); + admitted = { host, id: await admit(hostLimits(), host, cancel.signal, notice, precedence) }; + observed.admitted = Date.now(); + } catch (error) { + if (cancel.signal.aborted) return cancelled(cancel.signal); + throw error; } - const failed = await measuring; - if (cancel.signal.aborted) return cancelled(cancel.signal); - if (failed) throw failed.error; - return result; + } + const stop = new AbortController(); + // Held as a value until the command ends, so a failure never goes unhandled meanwhile. + const measuring = + admitted !== undefined && unit !== undefined + ? measure(admitted.host, { id: admitted.id, unit }, stop.signal).then( + () => undefined, + (error: unknown) => ({ error }), + ) + : undefined; + try { + observed.exit = await run(command, { ...process.env, [ADMITTED]: "1" }, cancel); } finally { - cancel.close(); + stop.abort(); } + const failed = await measuring; + if (cancel.signal.aborted) return cancelled(cancel.signal); + if (failed) throw failed.error; + return observed.exit.status; } diff --git a/src/cli/main.ts b/src/cli/main.ts index d7cc9b499..af4713396 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -142,11 +142,14 @@ export function main(argv: string[], out: Output): number | Promise { "that lands PRs should set, the command waits for memory but never for load.\n\n" + "Once the command exits, or is cancelled and exits, its remaining descendants are killed; a second " + "signal kills them all at once. Linux with a systemd 254+ user manager owns every descendant through " + - "a systemd user scope, or inside a Quest scope through a child cgroup of its own. macOS, and Linux " + + "a systemd user scope, inside a Quest scope one bound to it. macOS, and Linux " + "without one, own the command's process group in a new session without a controlling terminal, so a " + "descendant that leaves the group or session escapes. Windows owns every descendant through a " + "kill-on-close Job Object. Exits with the command's status, or 128 plus the signal's number once " + - "cancelled.", + "cancelled.\n\n" + + "With QUEST_GATE_RECORD naming an absolute path, an outermost gate on Linux or macOS appends one JSON " + + "line per run to that file, however the run ends: its command, directory, when it started, was " + + "admitted, and ended, its CPU time, peak memory, and exit status. A file it cannot write fails the gate.", ) .option("--quick", `Start a cheap command after at most ${QUICK_MS / 1000} seconds of waiting`) .argument("", "Executable to run") @@ -900,7 +903,7 @@ export function main(argv: string[], out: Output): number | Promise { "Run a command in the checkout, restoring it first when released, without the repository-location " + "variables, such as GIT_DIR, that a Git hook exports. The command and its descendants " + "run in a new, delegated systemd user scope; once the command exits, descendants still in the scope " + - "are killed. A `quest gate` inside it owns its command through a child cgroup of the scope. " + + "are killed. A `quest gate` inside it runs its command in a scope of its own, bound to this one. " + "When no other supervised run remains, the scratch declared at creation is pruned. Put the whole job, " + "such as an agent session or a script that installs, builds, and tests, in one run.\n\n" + "A supervisor that crashes leaves its run recorded; the next worktree command drops it once its scope " + diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index feac04a22..a8ea8476f 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -1,7 +1,9 @@ // Linux process ownership for supervised checkout commands and gated ones. Each run gets its own // systemd user scope, named once and never reused, so its cgroup holds every descendant that does // not ask another service manager to start it. Other hosts fail explicitly; only the gate settles -// for a process group there (see `group` in gate.ts). +// for a process group there (see `group` in gate.ts). The scope is delegated: a holder shell stays +// at its root while the command runs in a child cgroup, so the scope, and with it the counters that +// cover everything the command started, outlives the command until the owner has read them. import { type ChildProcessWithoutNullStreams, execFileSync, spawn } from "node:child_process"; import { closeSync, existsSync, constants as fs, openSync, readdirSync, readFileSync, watch, writeSync } from "node:fs"; import { constants, hostname } from "node:os"; @@ -51,6 +53,30 @@ export function alive(owner: Owner): boolean { } } +/** What an owner measured of its command, including what that command started. */ +export interface Usage { + /** CPU time, in milliseconds. */ + cpuMs: number; + /** Peak memory in bytes, or null where the host cannot say. */ + peakMemory: number | null; + /** + * `cgroup`: a scope's counters, which cover every process in it. `children`: the command's own + * resource usage, which covers only the descendants it waited for, and as peak memory the largest + * single process's. + */ + measured: "cgroup" | "children"; +} + +/** How an owned command ended. */ +export interface Exit { + /** Its exit status, 128 plus the signal's number when signalled, or the cancelled status. */ + status: number; + /** Wall-clock milliseconds when the command's own process exited, before its leftovers were stopped. */ + ended: number; + /** Undefined where the owner measures nothing, or lost the count to an enclosing owner's kill. */ + usage?: Usage; +} + /** How a scope runs its command. */ export interface RunOptions { /** The command's environment; Quest's own when omitted. */ @@ -86,10 +112,11 @@ export interface Scopes { * Run a command in a new scope, then stop its remaining descendants: at once, or, when a * cancellation signalled the whole scope, once the scope empties or the run is killed, so each * process can finish its own cleanup. Resolves with the command's exit status, or the cancelled - * status once asked to stop, only once the scope is empty. Rejects with `Retained` when that - * cannot be confirmed, and otherwise only when the command never started. + * status once asked to stop, only once the scope is empty, with the scope's counters. Rejects with + * `Retained` when that cannot be confirmed, and otherwise only when the command never started or + * its counters cannot be read. */ - run(unit: string, cwd: string, command: string[], options?: RunOptions): Promise; + run(unit: string, cwd: string, command: string[], options?: RunOptions): Promise; } /** The command may have started and its scope may still hold processes, so its run stays recorded. */ @@ -99,15 +126,24 @@ export const UNSUPPORTED = "quest worktree run needs Linux 5.14 or later with a systemd user manager (systemd 254 or later) and unified cgroup v2"; /** - * The shell proves it runs in the scope's own v2 cgroup, then waits for the supervisor before it - * executes the command. A supervisor that died never acknowledges, so the command never starts. + * The shell proves it runs in the scope's own v2 cgroup, creates the child cgroup `command`, and + * waits for the supervisor before it starts the command there. A supervisor that died never + * acknowledges, so the command never starts. The shell stays at the scope's root as its holder, + * ignoring the signals forwarded to every process in the scope, which reach the command through the + * default handlers a subshell restores. It reports `started` once the command is in its cgroup and + * then the command's exit status, and keeps the scope populated until the supervisor, having stopped + * the command's leftovers and read the scope's counters, closes the acknowledgement pipe. */ const handshake = "IFS=: read -r hierarchy controllers path &3; ' + - "read -r admitted <&4 || exit 125; " + - 'exec 3>&- 4<&-; exec "$@"'; + 'if ! read -r admitted <&4; then rmdir "$inner"; exit 125; fi; ' + + "trap : INT TERM HUP; " + + '(echo 0 >"$inner/cgroup.procs" || exit 125; echo started >&3; exec 3>&- 4<&-; exec "$@"); ' + + 'status=$?; printf "%s\\n" "$status" >&3; ' + + 'read -r released <&4; rmdir "$inner" 2>/dev/null; exit "$status"'; function populated(group: string): boolean { try { @@ -194,6 +230,7 @@ export function kill(group: string): void { */ function signalGroup(group: string, signal: NodeJS.Signals): void { const signalled = new Set(); + let failure: unknown; for (let fresh = true; fresh;) { fresh = false; for (const pid of members(group)) { @@ -203,10 +240,28 @@ function signalGroup(group: string, signal: NodeJS.Signals): void { try { process.kill(pid, signal); } catch (error) { - if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; + // Such as EPERM from a process running as another user: still signal the rest. + if ((error as NodeJS.ErrnoException).code !== "ESRCH") failure ??= error; } } } + if (failure) throw failure; +} + +/** + * The scope's CPU time and peak memory, which cover every process it held, as long as the holder + * keeps it from being removed. `memory.peak` needs Linux 5.19 and the memory controller. + */ +function counters(group: string): Usage { + const usage = /^usage_usec (\d+)$/m.exec(readFileSync(join(group, "cpu.stat"), "utf8"))?.[1]; + if (usage === undefined) throw new Error(`cannot read the CPU use of ${group}`); + let peakMemory: number | null = null; + try { + peakMemory = Number(readFileSync(join(group, "memory.peak"), "utf8")); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + return { cpuMs: Math.round(Number(usage) / 1000), peakMemory, measured: "cgroup" }; } /** The processes in the cgroup and every cgroup below it; none once it is removed. */ @@ -296,6 +351,8 @@ export const systemdScopes: Scopes = { "--scope", "--quiet", "--collect", + // The holder creates the command's child cgroup (see `handshake`). + "--property=Delegate=yes", `--unit=${unit}`, ...bound, ...(options.slice ? [`--slice=${options.slice}`] : []), @@ -340,8 +397,8 @@ interface Containment { /** * Start the handshake shell through `launch`, admit the command once the shell reports a cgroup - * that `owns` accepts, and once the command exits, kill whatever is left in that cgroup and wait - * until it is empty (see `Scopes.run`). + * that `owns` accepts, and once the command exits, kill whatever is left of it, read the scope's + * counters, release the holder, and wait until the cgroup is empty (see `Scopes.run`). */ function contain( launch: string[], @@ -350,7 +407,7 @@ function contain( cwd: string, command: string[], options: RunOptions, -): Promise { +): Promise { return new Promise((resolve, reject) => { const own = options.cancel ? undefined : interrupts(); const cancel = options.cancel ?? own!; @@ -363,10 +420,16 @@ function contain( options.spawned?.(child as ChildProcessWithoutNullStreams); const report = child.stdio[3] as Readable; const admission = (child.stdio as unknown as (Writable | null)[])[4]!; + // The scope's cgroup, and the command's child cgroup in it. let group: string | undefined; + let inner: string | undefined; let admitted = false; + let started = false; let interrupted = 0; let failure: Error | undefined; + // Set once the holder reports the command's exit; settles once its leftovers are stopped and counted. + let exited: { status: number; ended: number } | undefined; + let finishing: Promise<{ usage: Usage } | { error: unknown }> | undefined; // Set once every process in the cgroup was asked to stop; aborted once they no longer get time to. let signalled = false; const drain = new AbortController(); @@ -376,16 +439,12 @@ function contain( // The shell reads end-of-file and never starts the command. if (!admitted) return admission.destroy(); if (!forwards(cancel.signal)) return; - const name = requested(cancel.signal); signalled = true; try { - signalGroup(group!, name); + // The holder ignores it; every other process runs its own handling. + signalGroup(group!, requested(cancel.signal)); } catch (error) { - // Such as EPERM from a process running as another user: signal what this process can. - process.stderr.write( - `quest: cannot signal every process in ${describe.name}, only its command: ${error}\n`, - ); - child.kill(name); + process.stderr.write(`quest: cannot signal every process in ${describe.name}: ${error}\n`); drain.abort(); } }; @@ -394,7 +453,9 @@ function contain( // The enclosing owner kills this cgroup at its own second signal (see `bound`). if (describe.bound && received(cancel.signal)) return; drain.abort(); - if (group) kill(group); + // Once the command runs, the holder outlives it to report its status and counters. + const target = started ? inner : group; + if (target) kill(target); }; const unfollow = follow(cancel, stop, terminate); const cleanup = () => { @@ -402,26 +463,74 @@ function contain( own?.close(); }; + /** Stop what the command left in its cgroup, count the whole scope, then release the holder. */ + const finish = async (): Promise<{ usage: Usage } | { error: unknown }> => { + try { + try { + // Every process got the signal, so let each finish its own cleanup, even one that + // outlives the wrappers that exited first, until a second signal kills them. + if (signalled) await vacated(inner!, drain.signal); + kill(inner!); + await vacated(inner!); + } catch (error) { + return { + error: new Retained( + `cannot confirm that ${describe.name} stopped: ${error instanceof Error ? error.message : error}`, + ), + }; + } + try { + return { usage: counters(group!) }; + } catch (error) { + return { error }; + } + } finally { + admission.end(); + } + }; + admission.on("error", (error: NodeJS.ErrnoException) => { if (error.code !== "EPIPE") failure = error; }); let text = ""; + let line = 0; report.setEncoding("utf8"); report.on("data", (data: string) => { text += data; - if (group !== undefined || !text.endsWith("\n")) return; - const path = text.slice(0, -1); - // A unique name can never name a later cgroup, so this path stays this run's. - if (!path.startsWith("/") || path.includes("\n") || !owns(path)) { - failure = new Error(`${describe.unsupported}: the command ran outside its cgroup (${path})`); - admission.destroy(); - return; + for (let end = text.indexOf("\n"); end >= 0; end = text.indexOf("\n")) { + const value = text.slice(0, end); + text = text.slice(end + 1); + reported(value, line++); } - group = join("/sys/fs/cgroup", path); - if (interrupted) return; - admitted = true; - admission.end("admitted\n"); }); + /** The holder reports the scope's cgroup, then `started`, then the command's exit status. */ + const reported = (value: string, index: number) => { + if (index === 0) { + // A unique name can never name a later cgroup, so this path stays this run's. + if (!value.startsWith("/") || !owns(value)) { + failure = new Error(`${describe.unsupported}: the command ran outside its cgroup (${value})`); + admission.destroy(); + return; + } + group = join("/sys/fs/cgroup", value); + inner = join(group, "command"); + if (interrupted) return; + admitted = true; + admission.write("admitted\n"); + } else if (index === 1 && value === "started") started = true; + else if (/^\d+$/.test(value) && exited === undefined) { + exited = { status: Number(value), ended: Date.now() }; + if (started) finishing = finish(); + else { + // The holder could not move the command into its cgroup, so it never ran. + if (!interrupted) + failure ??= new Error(`${describe.unsupported}: the command never entered its cgroup`); + admission.end(); + } + } else { + failure ??= new Error(`${describe.launcher} reported an unexpected line: ${value}`); + } + }; child.once("error", (error) => { cleanup(); @@ -433,12 +542,13 @@ function contain( }); child.once("exit", async (code, signal) => { try { + const counted = await finishing; // Descendants that outlive the command are orphans of this run: stop them. if (group) { try { - // Every process got the signal, so let each finish its own cleanup, even one that - // outlives the wrappers that exited first, until a second signal kills them. - if (signalled) await vacated(group, drain.signal); + // A holder killed before it reported, as by the enclosing owner's kill, leaves + // the signalled processes their cleanup here. + if (signalled && !finishing) await vacated(group, drain.signal); kill(group); await vacated(group); await describe.emptied(); @@ -448,12 +558,17 @@ function contain( ); } } + if (counted && "error" in counted) throw counted.error; if (failure) throw failure; if (!admitted && !interrupted) throw new Error( `${describe.unsupported}: ${describe.launcher} exited with ${code ?? signal} before the command started`, ); - resolve(interrupted || (code ?? 128 + constants.signals[signal!])); + resolve({ + status: interrupted || (exited?.status ?? code ?? 128 + constants.signals[signal!]), + ended: exited?.ended ?? Date.now(), + usage: counted && "usage" in counted ? counted.usage : undefined, + }); } catch (error) { reject(error); } finally { diff --git a/src/cli/worktree.ts b/src/cli/worktree.ts index 3bd7b4454..dfc1b1517 100644 --- a/src/cli/worktree.ts +++ b/src/cli/worktree.ts @@ -935,7 +935,7 @@ export class Worktrees { // The command runs in this checkout, where a location inherited from a Git hook names the wrong repository. const env = discoveryEnv(options?.env); const result = await this.scopes.run(run.unit, record.path, command, { ...options, env }).then( - (code) => ({ code }), + ({ status }) => ({ code: status }), (error: unknown) => ({ error }), ); // Waits out a busy lock rather than failing: the run's record keeps its scratch until this lands. diff --git a/test/agent.test.ts b/test/agent.test.ts index efa3643a8..610a3ef2d 100644 --- a/test/agent.test.ts +++ b/test/agent.test.ts @@ -193,7 +193,7 @@ async function managed() { }); options!.spawned!(child); child.once("error", reject); - child.once("close", (code) => resolve(code ?? 1)); + child.once("close", (code) => resolve({ status: code ?? 1, ended: Date.now() })); }), ); writeFileSync(join(s.worktree, ".gitignore"), "cache/\n.worktrees/\n"); diff --git a/test/gate.test.ts b/test/gate.test.ts index b5210c858..651b53eb0 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -739,7 +739,7 @@ function cancellation() { test("a command runs in place with the given environment and its exact exit status", async () => { const script = "process.exit(process.env.QUEST_GATE_ADMITTED === '1' ? 7 : 1)"; const env = { ...process.env, QUEST_GATE_ADMITTED: "1" }; - expect(await direct([process.execPath, "-e", script], env, cancellation().cancel)).toBe(7); + expect(await exitStatus(direct)([process.execPath, "-e", script], env, cancellation().cancel)).toBe(7); }); test("a nested gate runs at once and keeps the exact exit status", async () => { @@ -770,7 +770,11 @@ test.runIf(process.platform === "linux")("a gated command's piped output blocks" test("a command cancelled before listener registration still stops", async () => { const { stop, cancel } = cancellation(); stop.abort(); - const result = await direct([process.execPath, "-e", "setInterval(() => {}, 1000)"], process.env, cancel); + const result = await exitStatus(direct)( + [process.execPath, "-e", "setInterval(() => {}, 1000)"], + process.env, + cancel, + ); expect(result).not.toBe(0); }); @@ -778,7 +782,11 @@ test("a command cancelled before listener registration still stops", async () => test.skipIf(process.platform === "win32")("cancellation forwards the received signal to the command", async () => { const { stop, cancel } = cancellation(); stop.abort("SIGHUP"); - const result = await direct([process.execPath, "-e", "setInterval(() => {}, 1000)"], process.env, cancel); + const result = await exitStatus(direct)( + [process.execPath, "-e", "setInterval(() => {}, 1000)"], + process.env, + cancel, + ); expect(result).toBe(128 + constants.signals.SIGHUP); }); @@ -842,17 +850,22 @@ function descendant(dir: string, rest: string, { leave = false } = {}) { } /** A systemd scope owner with a new unit, as the gate names one. */ -const scoped: Owner = (command, env, cancel) => scope(`quest-gate-${randomUUID()}.scope`)(command, env, cancel); +const scoped = exitStatus((command, env, cancel) => scope(`quest-gate-${randomUUID()}.scope`)(command, env, cancel)); + +/** Run through `owner` and resolve with the exit status alone. */ +function exitStatus(owner: Owner): (...args: Parameters) => Promise { + return (...args) => owner(...args).then(({ status }) => status); +} -const owners: [string, Owner][] = +const owners: [string, ReturnType][] = process.platform === "linux" ? [ ["a systemd scope", scoped], - ["a process group", group], + ["a process group", exitStatus(group)], ] : process.platform === "win32" ? [] - : [["a process group", group]]; + : [["a process group", exitStatus(group)]]; const bin = fileURLToPath(new URL("../src/cli/bin.ts", import.meta.url)); // Bun reads the checkout's config only from its working directory, which Just changes. @@ -991,7 +1004,7 @@ function outermostUnderJust(dir: string, command: string[]) { for (const name of ["SIGINT", "SIGTERM", "SIGHUP"]) process.on(name, () => writeFileSync(${JSON.stringify(dir)} + "/handled-" + ++handled, "")); writeFileSync(${JSON.stringify(pid)}, String(process.pid)); - const status = await group(${JSON.stringify(command)}, process.env, cancel); + const { status } = await group(${JSON.stringify(command)}, process.env, cancel); cancel.close(); process.exit(cancel.signal.aborted ? cancelled(cancel.signal) : status);`, ); @@ -1082,7 +1095,7 @@ describe.runIf(process.platform === "linux")("a systemd scope", () => { const done = join(tempDir(), "done"); expect(host.usage(unit)).toBeUndefined(); const spin = ["/bin/sh", "-c", `while [ ! -e ${quote(done)} ]; do :; done`]; - const result = scope(unit)(spin, process.env, cancellation().cancel); + const result = exitStatus(scope(unit))(spin, process.env, cancellation().cancel); await until(() => (host.usage(unit) ?? 0) > 0); writeFileSync(done, ""); expect(await result).toBe(0); @@ -1112,7 +1125,7 @@ async function insideScope(launch: (command: string[], env: NodeJS.ProcessEnv) = if (${run}) { const never = new AbortController().signal; const command = "cat /proc/self/cgroup > " + dir + "/cgroup; systemctl --user show --property=BindsTo --value " + unit + " > " + dir + "/binds; sleep 1000 & echo $! > " + dir + "/pid; exit 7"; - const status = await own(["/bin/sh", "-c", command], process.env, { signal: never, kill: never }); + const { status } = await own(["/bin/sh", "-c", command], process.env, { signal: never, kill: never }); const path = readFileSync(dir + "/cgroup", "utf8").trim().slice(3); const pid = readFileSync(dir + "/pid", "utf8").trim(); const ps = spawnSync("ps", ["-o", "stat=", "-p", pid], { encoding: "utf8" }); @@ -1135,7 +1148,7 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = // Bound to the enclosing scope, a gate's scope here, so that scope's owner stops it too. expect(report.binds).toMatch(/^quest-gate-[0-9a-f-]+\.scope$/); // A sibling, so only the gated command yields the CPU, not the rest of the enclosing scope. - expect(report.path).toMatch(/\/background\.slice\/quest-gate-[0-9a-f-]+\.scope$/); + expect(report.path).toMatch(/\/background\.slice\/quest-gate-[0-9a-f-]+\.scope\/command$/); }); test("is stopped when the enclosing scope's owner kills it", async () => { @@ -1185,7 +1198,7 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = stop.abort(new Error("cancelled by its caller")); }, 25); } - const status = await own(${JSON.stringify(command)}, process.env, cancel); + const { status } = await own(${JSON.stringify(command)}, process.env, cancel); signals.close(); process.exit(cancel.signal.aborted ? cancelled(cancel.signal) : status);`, ); @@ -1274,9 +1287,9 @@ describe("the Windows job stub", () => { const env: NodeJS.ProcessEnv = { ...process.env, EXPECTED: "keep" }; delete env.BUN_BE_BUN; const command = [process.execPath, "-e", script]; - expect(await job(command, { ...env, BUN_BE_BUN: "keep" }, cancellation().cancel)).toBe(7); + expect(await exitStatus(job)(command, { ...env, BUN_BE_BUN: "keep" }, cancellation().cancel)).toBe(7); const unset = "process.exit(process.env.BUN_BE_BUN === undefined ? 7 : 1)"; - expect(await job([process.execPath, "-e", unset], env, cancellation().cancel)).toBe(7); + expect(await exitStatus(job)([process.execPath, "-e", unset], env, cancellation().cancel)).toBe(7); }); test("passes the command's arguments as running it directly would", async () => { @@ -1286,7 +1299,7 @@ describe("the Windows job stub", () => { const script = "require('node:fs').writeFileSync(process.env.OUT, JSON.stringify(process.argv.slice(1)))"; const argv = async (owner: Owner, out: string) => { const env = { ...process.env, OUT: join(dir, out) }; - expect(await owner([process.execPath, "-e", script, ...args], env, cancellation().cancel)).toBe(0); + expect((await owner([process.execPath, "-e", script, ...args], env, cancellation().cancel)).status).toBe(0); return JSON.parse(readFileSync(env.OUT, "utf8")); }; const expected = await argv(direct, "direct"); @@ -1295,7 +1308,7 @@ describe("the Windows job stub", () => { }); test("reports a command that cannot start as a shell would", async () => { - expect(await job([join(tempDir(), "missing")], process.env, cancellation().cancel)).toBe(127); + expect(await exitStatus(job)([join(tempDir(), "missing")], process.env, cancellation().cancel)).toBe(127); }); test("hands the command the caller's environment, not the working directory's .env files", async () => { @@ -1310,7 +1323,7 @@ describe("the Windows job stub", () => { try { // The command is Bun too, so it must not load them either. const command = [process.execPath, "--no-env-file", "-e", script]; - expect(await job(command, env, cancellation().cancel)).toBe(0); + expect(await exitStatus(job)(command, env, cancellation().cancel)).toBe(0); } finally { process.chdir(cwd); } @@ -1344,7 +1357,7 @@ describe.runIf(process.platform === "win32")("a job object", () => { test("keeps the exact exit status and stops a detached descendant", async () => { const { command, pid } = detached(tempDir(), "process.exit(7);"); - expect(await job(command, process.env, cancellation().cancel)).toBe(7); + expect(await exitStatus(job)(command, process.env, cancellation().cancel)).toBe(7); await until(() => !running(pid())); }); @@ -1356,7 +1369,7 @@ describe.runIf(process.platform === "win32")("a job object", () => { `require('node:fs').writeFileSync(${JSON.stringify(ready)}, ''); setInterval(() => {}, 1000);`, ); const { kill, cancel } = cancellation(); - const result = job(command, process.env, cancel); + const result = exitStatus(job)(command, process.env, cancel); await created(ready); kill.abort(); expect(await result).not.toBe(0); @@ -1380,7 +1393,7 @@ describe.skipIf(process.platform === "win32")("a process group", () => { ); const { stop, cancel } = cancellation(); const env = { ...process.env, QUEST_GATE_ADMITTED: "1" }; - const result = group([process.execPath, bin, "gate", process.execPath, child], env, cancel); + const result = exitStatus(group)([process.execPath, bin, "gate", process.execPath, child], env, cancel); await created(ready); stop.abort("SIGINT"); expect(await result).toBe(128 + constants.signals.SIGINT); @@ -1389,7 +1402,7 @@ describe.skipIf(process.platform === "win32")("a process group", () => { test("leaves a descendant that started its own session", async () => { const { command, pid } = descendant(tempDir(), "exit 3", { leave: true }); - expect(await group(command, process.env, cancellation().cancel)).toBe(3); + expect(await exitStatus(group)(command, process.env, cancellation().cancel)).toBe(3); try { expect(stopped(pid())).toBe(false); } finally { @@ -1491,3 +1504,179 @@ test("gate rejects unknown options before the executable and forwards flags afte expect(gateBoundary(["gate", "--unknown", "ls"])).toEqual(["gate", "--unknown", "ls"]); expect(gateBoundary(["gate", "--", "-command"])).toEqual(["gate", "--", "-command"]); }); + +describe.skipIf(process.platform === "win32")("the run record", () => { + /** A Bun command that spins until it has used `ms` of CPU, reports when it ran and its CPU, and exits 3. */ + function spinner(dir: string, ms = 800) { + const out = join(dir, "spun.json"); + const script = + `const start = Date.now(); while (process.cpuUsage().user < ${ms * 1000}); ` + + `const { user, system } = process.cpuUsage(); ` + + `require("node:fs").writeFileSync(${JSON.stringify(out)}, JSON.stringify({ start, end: Date.now(), cpuMs: (user + system) / 1000 })); ` + + `process.exit(3);`; + return { + command: [process.execPath, "-e", script], + spun: (): { start: number; end: number; cpuMs: number } => JSON.parse(readFileSync(out, "utf8")), + }; + } + + /** A ledger of the test's own that admits at once, or never when `full`. */ + const ledger = (full = false): Host => ({ + ...fileHost(tempDir()), + load: () => 0, + memory: () => (full ? 0 : Infinity), + }); + + /** An outermost gate recording to `file`. */ + function outermost(file: string) { + vi.stubEnv("QUEST_GATE_ADMITTED", undefined); + vi.stubEnv("QUEST_GATE_PRIORITY", undefined); + vi.stubEnv("QUEST_GATE_RECORD", file); + } + + const lines = (file: string) => + readFileSync(file, "utf8") + .split("\n") + .filter(Boolean) + .map((line) => JSON.parse(line)); + + /** Cancel the newest gate with `name`, as its own signal handler would on receiving it. */ + function interrupt(name: NodeJS.Signals) { + (process.listeners(name).at(-1) as () => void)(); + } + + test("an outermost run appends one line with its wait, CPU, wall time, and exit", async () => { + const dir = tempDir(); + const file = join(dir, "record.jsonl"); + outermost(file); + const { command, spun } = spinner(dir); + const measured = owner(process.env).unit === undefined ? "children" : "cgroup"; + const before = Date.now(); + expect(await gate(command, { host: ledger() })).toBe(3); + const after = Date.now(); + const ran = spun(); + const [line, ...rest] = lines(file); + expect(rest).toEqual([]); + expect(line).toMatchObject({ + command, + directory: process.cwd(), + exit: 3, + cancelled: false, + error: null, + measured, + }); + const at = (field: string) => Date.parse(line[field]); + expect([before, at("started"), at("admitted"), ran.start]).toEqual( + [before, at("started"), at("admitted"), ran.start].sort((a, b) => a - b), + ); + expect([ran.end, at("ended"), after]).toEqual([ran.end, at("ended"), after].sort((a, b) => a - b)); + // The command ends before the gate's first CPU sample, and its whole use still counts. The + // command's own count leaves out only its exec and the owner's few processes around it. + expect(line.cpuMs).toBeGreaterThanOrEqual(Math.floor(ran.cpuMs)); + expect(line.cpuMs).toBeLessThan(ran.cpuMs + 1000); + expect(line.peakMemory).toBeGreaterThan(0); + }); + + test("a nested gate records nothing, and an empty variable is unset", async () => { + const dir = tempDir(); + const file = join(dir, "record.jsonl"); + vi.stubEnv("QUEST_GATE_RECORD", file); + vi.stubEnv("QUEST_GATE_ADMITTED", "1"); + expect(await gate([process.execPath, "-e", "process.exit(5)"])).toBe(5); + expect(existsSync(file)).toBe(false); + outermost(""); + expect(await gate([process.execPath, "-e", "process.exit(6)"], { host: ledger() })).toBe(6); + expect(readdirSync(dir)).toEqual([]); + }); + + test("a run cancelled while queued is recorded with no admission", async () => { + const dir = tempDir(); + const file = join(dir, "record.jsonl"); + const ran = join(dir, "ran"); + outermost(file); + const host = ledger(true); + const result = gate(["/bin/sh", "-c", `: > ${quote(ran)}`], { host }); + await until(() => host.ledger().admissions.length > 0); + interrupt("SIGHUP"); + expect(await result).toBe(128 + constants.signals.SIGHUP); + expect(existsSync(ran)).toBe(false); + expect(lines(file)).toEqual([ + expect.objectContaining({ + admitted: null, + cpuMs: null, + peakMemory: null, + measured: null, + exit: 128 + constants.signals.SIGHUP, + cancelled: true, + error: null, + }), + ]); + }); + + test("a record it cannot open refuses to run the command", async () => { + const dir = tempDir(); + const ran = join(dir, "ran"); + const command = ["/bin/sh", "-c", `: > ${quote(ran)}`]; + outermost(join(dir, "missing", "record.jsonl")); + await expect(gate(command, { host: ledger() })).rejects.toThrow(/cannot record its run in .*missing/); + outermost("record.jsonl"); + await expect(gate(command, { host: ledger() })).rejects.toThrow("must be an absolute path"); + expect(existsSync(ran)).toBe(false); + }); + + // /dev/full opens for appending, then fails every write. + test.runIf(process.platform === "linux")( + "a line it cannot write fails the gate, and a cancelled one keeps its status", + async () => { + outermost("/dev/full"); + await expect(gate(["true"], { host: ledger() })).rejects.toThrow("cannot record its run in /dev/full"); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + try { + const host = ledger(true); + const result = gate(["true"], { host }); + await until(() => host.ledger().admissions.length > 0); + interrupt("SIGINT"); + expect(await result).toBe(128 + constants.signals.SIGINT); + expect(stderr).toHaveBeenCalledWith(expect.stringContaining("cannot record its run in /dev/full")); + } finally { + stderr.mockRestore(); + } + }, + ); + + test("a process group measures its command's own usage", async () => { + const { command, spun } = spinner(tempDir()); + const exit = await group(command, process.env, cancellation().cancel); + expect(exit.status).toBe(3); + expect(exit.usage).toMatchObject({ measured: "children" }); + expect(exit.usage!.cpuMs).toBeGreaterThanOrEqual(Math.floor(spun().cpuMs)); + expect(exit.usage!.peakMemory).toBeGreaterThan(0); + }); + + test("concurrent gates leave only whole lines", async () => { + const dir = tempDir(); + const file = join(dir, "record.jsonl"); + const script = join(dir, "gates.ts"); + const gatePath = fileURLToPath(new URL("../src/cli/gate.ts", import.meta.url)); + // Long arguments make each line span many pages, so an interleaved write would show. + writeFileSync( + script, + `import { fileHost, gate } from ${JSON.stringify(gatePath)}; + const host = { ...fileHost(process.argv[2]), load: () => 0, memory: () => Infinity }; + process.exit(await gate(["true", process.argv[3].repeat(64 * 1024)], { host }));`, + ); + const env: NodeJS.ProcessEnv = { ...process.env, QUEST_GATE_RECORD: file }; + delete env.QUEST_GATE_ADMITTED; + const gates = "abcdefgh".split("").map((letter) => { + const child = spawn(process.execPath, [config, script, join(dir, `ledger-${letter}`), letter], { + env, + stdio: "inherit", + }); + return once(child, "exit"); + }); + expect((await Promise.all(gates)).map(([code]) => code)).toEqual(Array(8).fill(0)); + const recorded = lines(file).map((line) => line.command[1]); + expect(recorded.map((argument: string) => argument[0]).sort()).toEqual("abcdefgh".split("")); + for (const argument of recorded) expect(argument).toBe(argument[0].repeat(64 * 1024)); + }); +}); diff --git a/test/worktree.run.test.ts b/test/worktree.run.test.ts index 73816b695..15d94d182 100644 --- a/test/worktree.run.test.ts +++ b/test/worktree.run.test.ts @@ -188,7 +188,7 @@ describe.runIf(process.platform === "linux")("supervised runs under systemd", () const { stop, result } = gated(dir, home, leaf.script); await created(leaf.ready); stop.abort("SIGTERM"); - expect(await result).toBe(143); + expect((await result).status).toBe(143); expect(leaf.log()).toEqual(["SIGTERM"]); // Just's copy of the signal did not kill the run's scope before the leaf's cleanup finished. expect(existsSync(leaf.done)).toBe(true); @@ -202,7 +202,7 @@ describe.runIf(process.platform === "linux")("supervised runs under systemd", () stop.abort("SIGTERM"); await created(leaf.received); kill.abort(); - expect(await result).toBe(143); + expect((await result).status).toBe(143); // The gate killed the supervisor with everything else in its scope, and systemd the run's own. while (!stopped(leaf.pid())) await new Promise((resolve) => setTimeout(resolve, 25)); expect(await worktrees.stop(record.id)).toMatchObject({ phase: "released", runs: [] }); diff --git a/test/worktree.test.ts b/test/worktree.test.ts index 94cceb719..83b011823 100644 --- a/test/worktree.test.ts +++ b/test/worktree.test.ts @@ -41,7 +41,13 @@ const repository: (path: string) => string = template((dir) => { /** Scopes that are never live and whose runs exit at once, except where `overrides` says otherwise. */ function stubScopes(overrides: Partial): Scopes { - return { check() {}, live: () => false, run: () => Promise.resolve(0), stop: async () => {}, ...overrides }; + return { + check() {}, + live: () => false, + run: () => Promise.resolve({ status: 0, ended: 0 }), + stop: async () => {}, + ...overrides, + }; } /** Scopes whose runs finish when the test says so, and stay live while listed until stopped. */ function fakeScopes() { @@ -49,7 +55,7 @@ function fakeScopes() { const finish = new Map void>(); const scopes = stubScopes({ live: (unit) => live.has(unit), - run: (unit) => new Promise((resolve) => finish.set(unit, resolve)), + run: (unit) => new Promise((resolve) => finish.set(unit, (status) => resolve({ status, ended: 0 }))), stop: async (unit) => void live.delete(unit), }); return { live, finish, scopes }; From 95e4a6470065b4f8fbb0406c5e7df170520f554d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 20:47:21 -0800 Subject: [PATCH 03/13] fix(gate): drain the group token, count CPU in the command's cgroup, keep a retained run's usage Co-Authored-By: Claude Opus 5.5 --- README.md | 16 ++++++++++------ src/cli/gate.ts | 37 +++++++++++++++++++------------------ src/cli/supervise.ts | 36 ++++++++++++++++++++++++++---------- test/gate.test.ts | 15 ++++++++++++++- 4 files changed, 69 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index 15395ce28..5f7257e65 100644 --- a/README.md +++ b/README.md @@ -1302,20 +1302,24 @@ however the run ends: ``` - `started`, `admitted`, and `ended` give the wait and the wall time. `ended` is - when the command's own process exited, before the gate stopped what it left - behind. A run cancelled before admission has a null `admitted`. + when the gate learned that the command's own process exited, within + milliseconds, before it stopped what the command left behind. A run cancelled + before admission has a null `admitted`. - `cpuMs` and `peakMemory` (bytes) are null when the command never ran, and `peakMemory` also where the host cannot say. `measured` names what they count: - - `cgroup`: the command's systemd scope, so every process it started. Peak - memory is the scope's `memory.peak`, which includes page cache charged to - it and needs Linux 5.19. + - `cgroup`: the command's systemd scope. CPU covers every process the command + started. Peak memory is the scope's `memory.peak`, which needs Linux 5.19 + and also counts page cache charged to the scope and the gate's own shell + that holds it, about a megabyte. - `children`: on macOS, and Linux without a systemd user manager, the command's own resource usage, a lower bound: it covers the descendants the command waited for, not those the gate kills at the end, such as a leftover background server, and its peak memory is the largest single process's. - `exit` is the status the gate returns, `cancelled` whether it was cancelled, - and `error` the gate's own failure, such as a command that cannot start. + and `error` the gate's own failure, such as a missing `systemd-run` or a scope + it cannot confirm stopped. A command that cannot be found is not one: it + exits 127, as in a shell. Each line is one write to a file opened for appending, so concurrent gates leave only whole lines; give each agent its own file to tell them apart. Nested diff --git a/src/cli/gate.ts b/src/cli/gate.ts index c4ffac787..9407fb311 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -21,7 +21,7 @@ import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; import { setTimeout as sleep } from "node:timers/promises"; import { type Cancellation, cancelled, follow, forwards, GROUP, interrupts, received, requested } from "./interrupt"; import { stubbed } from "./job"; -import { type Exit, questScope, scopeGroup, startTime, systemdScopes } from "./supervise"; +import { type Exit, questScope, Retained, scopeGroup, startTime, systemdScopes } from "./supervise"; /** * The part of Bun's process API the group owner needs: Node's `child_process` reports no resource @@ -741,9 +741,16 @@ export const group: Owner = (command, env, cancel) => return reject(error); } const token = child.stdio[TOKEN]; - // Read only once a cancellation waits for the end; holders' writes are discarded. - let reader: ReadableStreamDefaultReader | undefined; - let released: Promise | undefined; + // Drained from the start, so a holder's writes never fill the pipe; only its end matters. + const reader = Bun.file(token).stream().getReader(); + const released = (async () => { + try { + while (!(await reader.read()).done); + } catch { + // A failed read ends the wait as the pipe's end would. + } + })(); + let signalled = false; const signal = (name: NodeJS.Signals) => { try { process.kill(-child.pid, name); @@ -762,17 +769,7 @@ export const group: Owner = (command, env, cancel) => } }; const stop = () => { - if (!reader) { - const stream = Bun.file(token).stream().getReader(); - reader = stream; - released = (async () => { - try { - while (!(await stream.read()).done); - } catch { - // A failed read ends the wait as the pipe's end would. - } - })(); - } + signalled = true; signal(requested(cancel.signal)); }; const terminate = () => signal("SIGKILL"); @@ -786,10 +783,10 @@ export const group: Owner = (command, env, cancel) => const status = child.exitCode ?? 128 + constants.signals[child.signalCode!]; const usage = child.resourceUsage(); // Every process in the group got the signal, so let each holder finish its own cleanup. - if (released) await Promise.race([released, killed]); + if (signalled) await Promise.race([released, killed]); cleanup(); try { - await reader?.cancel(); + await reader.cancel(); // Bun leaves a pipe it handed out open. closeSync(token); // The group outlives its leader while any member remains, so its ID is not reused yet. A @@ -885,6 +882,8 @@ export async function gate(command: string[], options: GateOptions = {}): Promis const outermost = process.env[ADMITTED] !== "1"; const record = outermost ? recordFile(process.env) : undefined; const started = Date.now(); + // Read now: the command may remove its own directory. + const directory = record ? process.cwd() : ""; const cancel = interrupts(); const observed: Observed = {}; try { @@ -896,13 +895,15 @@ export async function gate(command: string[], options: GateOptions = {}): Promis failure = error; // The status the CLI exits with for a gate's own failure. status = 1; + // A command that ran keeps its measurements even when its scope cannot be confirmed stopped. + if (error instanceof Retained) observed.exit ??= error.exit; } if (record) { const iso = (ms: number) => new Date(ms).toISOString(); const usage = observed.exit?.usage; const line = `${JSON.stringify({ command, - directory: process.cwd(), + directory, started: iso(started), admitted: observed.admitted === undefined ? null : iso(observed.admitted), ended: iso(observed.exit?.ended ?? Date.now()), diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index a8ea8476f..797f2594a 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -119,8 +119,13 @@ export interface Scopes { run(unit: string, cwd: string, command: string[], options?: RunOptions): Promise; } -/** The command may have started and its scope may still hold processes, so its run stays recorded. */ -export class Retained extends Error {} +/** + * The command may have started and its scope may still hold processes, so its run stays recorded. + * `exit` is how the command ended, where its holder reported that before the failure. + */ +export class Retained extends Error { + exit?: Exit; +} export const UNSUPPORTED = "quest worktree run needs Linux 5.14 or later with a systemd user manager (systemd 254 or later) and unified cgroup v2"; @@ -249,12 +254,14 @@ function signalGroup(group: string, signal: NodeJS.Signals): void { } /** - * The scope's CPU time and peak memory, which cover every process it held, as long as the holder - * keeps it from being removed. `memory.peak` needs Linux 5.19 and the memory controller. + * The CPU time of every process that ran in the command's cgroup `inner`, and the peak memory of + * its scope `group`, which also holds the holder shell, since no controller is enabled below the + * scope. Both last only while the holder keeps the scope from being removed. `memory.peak` needs + * Linux 5.19 and the memory controller. */ -function counters(group: string): Usage { - const usage = /^usage_usec (\d+)$/m.exec(readFileSync(join(group, "cpu.stat"), "utf8"))?.[1]; - if (usage === undefined) throw new Error(`cannot read the CPU use of ${group}`); +function counters(group: string, inner: string): Usage { + const usage = /^usage_usec (\d+)$/m.exec(readFileSync(join(inner, "cpu.stat"), "utf8"))?.[1]; + if (usage === undefined) throw new Error(`cannot read the CPU use of ${inner}`); let peakMemory: number | null = null; try { peakMemory = Number(readFileSync(join(group, "memory.peak"), "utf8")); @@ -480,7 +487,7 @@ function contain( }; } try { - return { usage: counters(group!) }; + return { usage: counters(group!, inner!) }; } catch (error) { return { error }; } @@ -553,12 +560,21 @@ function contain( await vacated(group); await describe.emptied(); } catch (error) { - throw new Retained( + const retained = new Retained( `cannot confirm that ${describe.name} stopped: ${error instanceof Error ? error.message : error}`, ); + if (exited) + retained.exit = { + ...exited, + usage: counted && "usage" in counted ? counted.usage : undefined, + }; + throw retained; } } - if (counted && "error" in counted) throw counted.error; + if (counted && "error" in counted) { + if (counted.error instanceof Retained && exited) counted.error.exit = exited; + throw counted.error; + } if (failure) throw failure; if (!admitted && !interrupted) throw new Error( diff --git a/test/gate.test.ts b/test/gate.test.ts index 651b53eb0..04b213e71 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -1574,7 +1574,15 @@ describe.skipIf(process.platform === "win32")("the run record", () => { // command's own count leaves out only its exec and the owner's few processes around it. expect(line.cpuMs).toBeGreaterThanOrEqual(Math.floor(ran.cpuMs)); expect(line.cpuMs).toBeLessThan(ran.cpuMs + 1000); - expect(line.peakMemory).toBeGreaterThan(0); + // A scope's peak needs Linux 5.19 and the memory controller in the slice gated scopes run in. + const uid = process.getuid!(); + const slice = `/sys/fs/cgroup/user.slice/user-${uid}.slice/user@${uid}.service/background.slice`; + const peak = + measured === "children" || + (existsSync(join(slice, "memory.peak")) && + readFileSync(join(slice, "cgroup.subtree_control"), "utf8").split(/\s+/).includes("memory")); + if (peak) expect(line.peakMemory).toBeGreaterThan(0); + else expect(line.peakMemory).toBeNull(); }); test("a nested gate records nothing, and an empty variable is unset", async () => { @@ -1644,6 +1652,11 @@ describe.skipIf(process.platform === "win32")("the run record", () => { }, ); + test("a process group drains what its command writes to the token", async () => { + const command = ["/bin/sh", "-c", "head -c 1048576 /dev/zero >&3; exit 4"]; + expect((await group(command, process.env, cancellation().cancel)).status).toBe(4); + }); + test("a process group measures its command's own usage", async () => { const { command, spun } = spinner(tempDir()); const exit = await group(command, process.env, cancellation().cancel); From 60bacf8ddce8defa0159ed562aea42a07dd6c767 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 21:12:15 -0800 Subject: [PATCH 04/13] fix(supervise): keep the command the spawned process, with the holder in a cgroup of its own The handshake shell forks the holder twice into a holder cgroup, moves itself into the command cgroup, enables the memory controller below the scope, and execs the command, so the supervisor's child is the command again (quest agent stop signals it) and its counters exclude the holder. Co-Authored-By: Claude Opus 5.5 --- README.md | 11 ++- src/cli/supervise.ts | 202 +++++++++++++++++++------------------------ test/gate.test.ts | 18 +++- 3 files changed, 111 insertions(+), 120 deletions(-) diff --git a/README.md b/README.md index 5f7257e65..d2223b32f 100644 --- a/README.md +++ b/README.md @@ -1308,17 +1308,16 @@ however the run ends: - `cpuMs` and `peakMemory` (bytes) are null when the command never ran, and `peakMemory` also where the host cannot say. `measured` names what they count: - - `cgroup`: the command's systemd scope. CPU covers every process the command - started. Peak memory is the scope's `memory.peak`, which needs Linux 5.19 - and also counts page cache charged to the scope and the gate's own shell - that holds it, about a megabyte. + - `cgroup`: a child cgroup of the command's systemd scope, so every process + the command started. Peak memory is its `memory.peak`, which needs Linux + 5.19 and also counts page cache charged to it. - `children`: on macOS, and Linux without a systemd user manager, the command's own resource usage, a lower bound: it covers the descendants the command waited for, not those the gate kills at the end, such as a leftover background server, and its peak memory is the largest single process's. - `exit` is the status the gate returns, `cancelled` whether it was cancelled, - and `error` the gate's own failure, such as a missing `systemd-run` or a scope - it cannot confirm stopped. A command that cannot be found is not one: it + and `error` the gate's own failure, such as a scope it cannot confirm stopped + or counters it cannot read. A command that cannot be found is not one: it exits 127, as in a shell. Each line is one write to a file opened for appending, so concurrent gates diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index 797f2594a..87ef740e4 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -1,9 +1,9 @@ // Linux process ownership for supervised checkout commands and gated ones. Each run gets its own // systemd user scope, named once and never reused, so its cgroup holds every descendant that does // not ask another service manager to start it. Other hosts fail explicitly; only the gate settles -// for a process group there (see `group` in gate.ts). The scope is delegated: a holder shell stays -// at its root while the command runs in a child cgroup, so the scope, and with it the counters that -// cover everything the command started, outlives the command until the owner has read them. +// for a process group there (see `group` in gate.ts). The scope is delegated: the command runs in a +// child cgroup whose counters cover everything it started, and a holder in another keeps the scope, +// and with it those counters, until the owner has read them. import { type ChildProcessWithoutNullStreams, execFileSync, spawn } from "node:child_process"; import { closeSync, existsSync, constants as fs, openSync, readdirSync, readFileSync, watch, writeSync } from "node:fs"; import { constants, hostname } from "node:os"; @@ -131,24 +131,29 @@ export const UNSUPPORTED = "quest worktree run needs Linux 5.14 or later with a systemd user manager (systemd 254 or later) and unified cgroup v2"; /** - * The shell proves it runs in the scope's own v2 cgroup, creates the child cgroup `command`, and - * waits for the supervisor before it starts the command there. A supervisor that died never - * acknowledges, so the command never starts. The shell stays at the scope's root as its holder, - * ignoring the signals forwarded to every process in the scope, which reach the command through the - * default handlers a subshell restores. It reports `started` once the command is in its cgroup and - * then the command's exit status, and keeps the scope populated until the supervisor, having stopped - * the command's leftovers and read the scope's counters, closes the acknowledgement pipe. + * The shell proves it runs in the scope's own v2 cgroup, creates the child cgroups `holder` and + * `command`, and waits for the supervisor before it executes the command in `command`. A + * supervisor that died never acknowledges, so the command never starts. The holder, a subshell + * that ignores the signals forwarded to every process in the scope, waits in `holder` for the + * acknowledgement pipe to close, which the supervisor does once it has stopped the command's + * leftovers and read its counters, so systemd cannot remove the scope first. With both moved out + * of the scope's root, the memory controller can be enabled below it, so `command` counts its own + * peak. The command itself stays the process the supervisor started, which `quest agent stop` signals. */ const handshake = "IFS=: read -r hierarchy controllers path &3; ' + - 'if ! read -r admitted <&4; then rmdir "$inner"; exit 125; fi; ' + - "trap : INT TERM HUP; " + - '(echo 0 >"$inner/cgroup.procs" || exit 125; echo started >&3; exec 3>&- 4<&-; exec "$@"); ' + - 'status=$?; printf "%s\\n" "$status" >&3; ' + - 'read -r released <&4; rmdir "$inner" 2>/dev/null; exit "$status"'; + "read -r admitted <&4 || exit 125; " + + // Forked twice, so the command never inherits the holder as a child it might wait for. + '(echo 0 >"$scope/holder/cgroup.procs" || exit 125; ' + + '(trap "" INT TERM HUP; exec /dev/null 2>&1 3>&-; read -r released <&4) &) || exit 125; ' + + 'echo 0 >"$scope/command/cgroup.procs" || exit 125; ' + + 'read -r available <"$scope/cgroup.controllers"; ' + + 'case " $available " in *" memory "*) echo +memory >"$scope/cgroup.subtree_control" || exit 125;; esac; ' + + "echo started >&3; " + + 'exec 3>&- 4<&-; exec "$@"'; function populated(group: string): boolean { try { @@ -235,7 +240,6 @@ export function kill(group: string): void { */ function signalGroup(group: string, signal: NodeJS.Signals): void { const signalled = new Set(); - let failure: unknown; for (let fresh = true; fresh;) { fresh = false; for (const pid of members(group)) { @@ -245,26 +249,23 @@ function signalGroup(group: string, signal: NodeJS.Signals): void { try { process.kill(pid, signal); } catch (error) { - // Such as EPERM from a process running as another user: still signal the rest. - if ((error as NodeJS.ErrnoException).code !== "ESRCH") failure ??= error; + if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; } } } - if (failure) throw failure; } /** - * The CPU time of every process that ran in the command's cgroup `inner`, and the peak memory of - * its scope `group`, which also holds the holder shell, since no controller is enabled below the - * scope. Both last only while the holder keeps the scope from being removed. `memory.peak` needs - * Linux 5.19 and the memory controller. + * The CPU time and peak memory of every process that ran in the command's cgroup `inner`, which + * last only while the holder keeps the scope from being removed. `memory.peak` needs Linux 5.19 + * and the memory controller. */ -function counters(group: string, inner: string): Usage { +function counters(inner: string): Usage { const usage = /^usage_usec (\d+)$/m.exec(readFileSync(join(inner, "cpu.stat"), "utf8"))?.[1]; if (usage === undefined) throw new Error(`cannot read the CPU use of ${inner}`); let peakMemory: number | null = null; try { - peakMemory = Number(readFileSync(join(group, "memory.peak"), "utf8")); + peakMemory = Number(readFileSync(join(inner, "memory.peak"), "utf8")); } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } @@ -404,8 +405,8 @@ interface Containment { /** * Start the handshake shell through `launch`, admit the command once the shell reports a cgroup - * that `owns` accepts, and once the command exits, kill whatever is left of it, read the scope's - * counters, release the holder, and wait until the cgroup is empty (see `Scopes.run`). + * that `owns` accepts, and once the command exits, kill whatever is left of it, read the counters, + * release the holder, and wait until the scope is empty (see `Scopes.run`). */ function contain( launch: string[], @@ -431,12 +432,10 @@ function contain( let group: string | undefined; let inner: string | undefined; let admitted = false; + // Set once the command runs in `inner`, beside the holder. let started = false; let interrupted = 0; let failure: Error | undefined; - // Set once the holder reports the command's exit; settles once its leftovers are stopped and counted. - let exited: { status: number; ended: number } | undefined; - let finishing: Promise<{ usage: Usage } | { error: unknown }> | undefined; // Set once every process in the cgroup was asked to stop; aborted once they no longer get time to. let signalled = false; const drain = new AbortController(); @@ -446,12 +445,17 @@ function contain( // The shell reads end-of-file and never starts the command. if (!admitted) return admission.destroy(); if (!forwards(cancel.signal)) return; + const name = requested(cancel.signal); signalled = true; try { // The holder ignores it; every other process runs its own handling. - signalGroup(group!, requested(cancel.signal)); + signalGroup(group!, name); } catch (error) { - process.stderr.write(`quest: cannot signal every process in ${describe.name}: ${error}\n`); + // Such as EPERM from a process running as another user: signal what this process can. + process.stderr.write( + `quest: cannot signal every process in ${describe.name}, only its command: ${error}\n`, + ); + child.kill(name); drain.abort(); } }; @@ -460,7 +464,7 @@ function contain( // The enclosing owner kills this cgroup at its own second signal (see `bound`). if (describe.bound && received(cancel.signal)) return; drain.abort(); - // Once the command runs, the holder outlives it to report its status and counters. + // Once the command runs, the holder outlives it, so the scope's counters can still be read. const target = started ? inner : group; if (target) kill(target); }; @@ -469,75 +473,38 @@ function contain( unfollow(); own?.close(); }; - - /** Stop what the command left in its cgroup, count the whole scope, then release the holder. */ - const finish = async (): Promise<{ usage: Usage } | { error: unknown }> => { - try { - try { - // Every process got the signal, so let each finish its own cleanup, even one that - // outlives the wrappers that exited first, until a second signal kills them. - if (signalled) await vacated(inner!, drain.signal); - kill(inner!); - await vacated(inner!); - } catch (error) { - return { - error: new Retained( - `cannot confirm that ${describe.name} stopped: ${error instanceof Error ? error.message : error}`, - ), - }; - } - try { - return { usage: counters(group!, inner!) }; - } catch (error) { - return { error }; - } - } finally { - admission.end(); - } - }; + const retained = (error: unknown) => + new Retained( + `cannot confirm that ${describe.name} stopped: ${error instanceof Error ? error.message : error}`, + ); admission.on("error", (error: NodeJS.ErrnoException) => { if (error.code !== "EPIPE") failure = error; }); let text = ""; - let line = 0; report.setEncoding("utf8"); + // The shell reports the scope's cgroup, then `started` once the command is in its own. report.on("data", (data: string) => { text += data; for (let end = text.indexOf("\n"); end >= 0; end = text.indexOf("\n")) { - const value = text.slice(0, end); + const line = text.slice(0, end); text = text.slice(end + 1); - reported(value, line++); + if (group === undefined) { + // A unique name can never name a later cgroup, so this path stays this run's. + if (!line.startsWith("/") || !owns(line)) { + failure = new Error(`${describe.unsupported}: the command ran outside its cgroup (${line})`); + admission.destroy(); + return; + } + group = join("/sys/fs/cgroup", line); + inner = join(group, "command"); + if (interrupted) return; + admitted = true; + admission.write("admitted\n"); + } else if (line === "started" && !started) started = true; + else failure ??= new Error(`${describe.launcher} reported an unexpected line: ${line}`); } }); - /** The holder reports the scope's cgroup, then `started`, then the command's exit status. */ - const reported = (value: string, index: number) => { - if (index === 0) { - // A unique name can never name a later cgroup, so this path stays this run's. - if (!value.startsWith("/") || !owns(value)) { - failure = new Error(`${describe.unsupported}: the command ran outside its cgroup (${value})`); - admission.destroy(); - return; - } - group = join("/sys/fs/cgroup", value); - inner = join(group, "command"); - if (interrupted) return; - admitted = true; - admission.write("admitted\n"); - } else if (index === 1 && value === "started") started = true; - else if (/^\d+$/.test(value) && exited === undefined) { - exited = { status: Number(value), ended: Date.now() }; - if (started) finishing = finish(); - else { - // The holder could not move the command into its cgroup, so it never ran. - if (!interrupted) - failure ??= new Error(`${describe.unsupported}: the command never entered its cgroup`); - admission.end(); - } - } else { - failure ??= new Error(`${describe.launcher} reported an unexpected line: ${value}`); - } - }; child.once("error", (error) => { cleanup(); @@ -548,43 +515,52 @@ function contain( ); }); child.once("exit", async (code, signal) => { + const exit: Exit = { + status: interrupted || (code ?? 128 + constants.signals[signal!]), + ended: Date.now(), + }; try { - const counted = await finishing; // Descendants that outlive the command are orphans of this run: stop them. if (group) { + let counted: unknown; try { - // A holder killed before it reported, as by the enclosing owner's kill, leaves - // the signalled processes their cleanup here. - if (signalled && !finishing) await vacated(group, drain.signal); + if (started) { + // Every process got the signal, so let each finish its own cleanup, even one + // that outlives the wrappers that exited first, until a second signal kills them. + if (signalled) await vacated(inner!, drain.signal); + kill(inner!); + await vacated(inner!); + try { + exit.usage = counters(inner!); + } catch (error) { + counted = error; + } + } + } catch (error) { + throw Object.assign(retained(error), { exit }); + } finally { + // The holder reads end-of-file and leaves, so the scope empties. + if (!admission.destroyed) admission.end(); + } + try { + // Before the command started, the signalled processes still get their cleanup. + if (signalled && !started) await vacated(group, drain.signal); kill(group); await vacated(group); await describe.emptied(); } catch (error) { - const retained = new Retained( - `cannot confirm that ${describe.name} stopped: ${error instanceof Error ? error.message : error}`, - ); - if (exited) - retained.exit = { - ...exited, - usage: counted && "usage" in counted ? counted.usage : undefined, - }; - throw retained; + throw Object.assign(retained(error), { exit }); } - } - if (counted && "error" in counted) { - if (counted.error instanceof Retained && exited) counted.error.exit = exited; - throw counted.error; + if (counted) throw counted; } if (failure) throw failure; if (!admitted && !interrupted) throw new Error( `${describe.unsupported}: ${describe.launcher} exited with ${code ?? signal} before the command started`, ); - resolve({ - status: interrupted || (exited?.status ?? code ?? 128 + constants.signals[signal!]), - ended: exited?.ended ?? Date.now(), - usage: counted && "usage" in counted ? counted.usage : undefined, - }); + if (admitted && !started && !interrupted) + throw new Error(`${describe.unsupported}: the command never entered its cgroup`); + resolve(exit); } catch (error) { reject(error); } finally { diff --git a/test/gate.test.ts b/test/gate.test.ts index 04b213e71..e6a92618b 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -43,7 +43,7 @@ import { } from "../src/cli/gate"; import { forwards, foregroundStat, statProcess } from "../src/cli/interrupt"; import { gateBoundary, main } from "../src/cli/main"; -import { alive as ownerAlive, questScope } from "../src/cli/supervise"; +import { alive as ownerAlive, questScope, systemdScopes } from "../src/cli/supervise"; import { created, onTerminal, run, signalled, tempDir } from "./fixture"; // Linux reports a /proc read that races a process's exit as ESRCH rather than ENOENT, so fail @@ -1089,6 +1089,22 @@ describe.runIf(process.platform === "linux")("a systemd scope", () => { await until(() => stopped(pid())); }); + test("runs the command as the process it spawned, which a signal to that PID reaches", async () => { + const dir = tempDir(); + const ready = join(dir, "ready"); + // What `quest agent stop` does: it signals the PID the supervisor spawned, not the scope. + let pid: number | undefined; + const result = systemdScopes.run( + `quest-gate-${randomUUID()}.scope`, + dir, + ["/bin/sh", "-c", `trap 'exit 3' TERM; : > ${quote(ready)}; while :; do sleep 0.1; done`], + { cancel: cancellation().cancel, within: null, spawned: (child) => (pid = child.pid) }, + ); + await created(ready); + process.kill(pid!, "SIGTERM"); + expect((await result).status).toBe(3); + }); + test("reports its CPU use to the gate until it ends", async () => { const host = fileHost(tempDir()); const unit = `quest-gate-${randomUUID()}.scope`; From 56aef8b115d018ac53e3d6a08023a34cb9e53ac5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 22:09:33 -0800 Subject: [PATCH 05/13] fix(supervise): wait for the start report before cleanup, keep signalling past EPERM; private record file Co-Authored-By: Claude Opus 5.5 --- README.md | 5 +++-- src/cli/gate.ts | 9 ++++++--- src/cli/supervise.ts | 30 ++++++++++++++++++++++-------- test/gate.test.ts | 6 ++++++ 4 files changed, 37 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index d2223b32f..ebc90a79d 100644 --- a/README.md +++ b/README.md @@ -1323,8 +1323,9 @@ however the run ends: Each line is one write to a file opened for appending, so concurrent gates leave only whole lines; give each agent its own file to tell them apart. Nested gates write nothing, since their outermost gate already counts their command. -An unset or empty variable writes nothing, a relative path is refused, and a -file the gate cannot open fails it before the command runs. A line it cannot +An unset or empty variable writes nothing, a relative path or a symlink is +refused, and a file the gate cannot open fails it before the command runs. A +new file is readable only by its user. A line it cannot write fails the gate too, though a cancelled run keeps its cancelled status. Windows refuses the variable for now. The agent's own processes write the file, so a record is only as trustworthy as the agent. diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 9407fb311..6e20b284f 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -5,6 +5,7 @@ import { spawn, spawnSync } from "node:child_process"; import { randomUUID } from "node:crypto"; import { closeSync, + constants as fs, linkSync, lstatSync, mkdirSync, @@ -21,7 +22,7 @@ import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; import { setTimeout as sleep } from "node:timers/promises"; import { type Cancellation, cancelled, follow, forwards, GROUP, interrupts, received, requested } from "./interrupt"; import { stubbed } from "./job"; -import { type Exit, questScope, Retained, scopeGroup, startTime, systemdScopes } from "./supervise"; +import { type Exit, questScope, scopeGroup, startTime, systemdScopes } from "./supervise"; /** * The part of Bun's process API the group owner needs: Node's `child_process` reports no resource @@ -848,7 +849,9 @@ function recordFile(env: NodeJS.ProcessEnv): { path: string; fd: number } | unde // The command's shells change directory, so a relative path would scatter one reader's lines. if (!isAbsolute(path)) throw new Error(`${RECORD} must be an absolute path: ${path}`); try { - return { path, fd: openSync(path, "a") }; + // Private to the user, and never through a symlink someone else planted at a shared path. + const flags = fs.O_WRONLY | fs.O_APPEND | fs.O_CREAT | fs.O_NOFOLLOW; + return { path, fd: openSync(path, flags, 0o600) }; } catch (error) { throw new Error(`gate cannot record its run in ${path}: ${message(error)}`); } @@ -896,7 +899,7 @@ export async function gate(command: string[], options: GateOptions = {}): Promis // The status the CLI exits with for a gate's own failure. status = 1; // A command that ran keeps its measurements even when its scope cannot be confirmed stopped. - if (error instanceof Retained) observed.exit ??= error.exit; + if (error instanceof Error && "exit" in error) observed.exit ??= error.exit as Exit; } if (record) { const iso = (ms: number) => new Date(ms).toISOString(); diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index 87ef740e4..896a39da7 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -240,6 +240,7 @@ export function kill(group: string): void { */ function signalGroup(group: string, signal: NodeJS.Signals): void { const signalled = new Set(); + let failure: unknown; for (let fresh = true; fresh;) { fresh = false; for (const pid of members(group)) { @@ -249,10 +250,12 @@ function signalGroup(group: string, signal: NodeJS.Signals): void { try { process.kill(pid, signal); } catch (error) { - if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; + // Such as EPERM from a process running as another user: still signal the rest. + if ((error as NodeJS.ErrnoException).code !== "ESRCH") failure ??= error; } } } + if (failure) throw failure; } /** @@ -451,11 +454,8 @@ function contain( // The holder ignores it; every other process runs its own handling. signalGroup(group!, name); } catch (error) { - // Such as EPERM from a process running as another user: signal what this process can. - process.stderr.write( - `quest: cannot signal every process in ${describe.name}, only its command: ${error}\n`, - ); - child.kill(name); + // The rest got the signal; one that did not is no reason to wait for the scope to empty. + process.stderr.write(`quest: cannot signal every process in ${describe.name}: ${error}\n`); drain.abort(); } }; @@ -482,6 +482,11 @@ function contain( if (error.code !== "EPIPE") failure = error; }); let text = ""; + // Every writer closes the report before the command runs, so it ends however the run goes. + const reported = new Promise((resolve) => { + report.once("close", resolve); + report.once("error", () => resolve()); + }); report.setEncoding("utf8"); // The shell reports the scope's cgroup, then `started` once the command is in its own. report.on("data", (data: string) => { @@ -520,6 +525,8 @@ function contain( ended: Date.now(), }; try { + // An exit can arrive before the report it follows, such as `started` from a short command. + await reported; // Descendants that outlive the command are orphans of this run: stop them. if (group) { let counted: unknown; @@ -551,14 +558,21 @@ function contain( } catch (error) { throw Object.assign(retained(error), { exit }); } - if (counted) throw counted; + if (counted) + throw Object.assign( + new Error( + `cannot read the counters of ${describe.name}: ${counted instanceof Error ? counted.message : counted}`, + ), + { exit }, + ); } if (failure) throw failure; if (!admitted && !interrupted) throw new Error( `${describe.unsupported}: ${describe.launcher} exited with ${code ?? signal} before the command started`, ); - if (admitted && !started && !interrupted) + // The shell's own failure; one killed before it ran the command reports that signal instead. + if (admitted && !started && !interrupted && code === 125) throw new Error(`${describe.unsupported}: the command never entered its cgroup`); resolve(exit); } catch (error) { diff --git a/test/gate.test.ts b/test/gate.test.ts index e6a92618b..a7f64e4cd 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -1573,6 +1573,7 @@ describe.skipIf(process.platform === "win32")("the run record", () => { const ran = spun(); const [line, ...rest] = lines(file); expect(rest).toEqual([]); + expect(statSync(file).mode & 0o777).toBe(0o600); expect(line).toMatchObject({ command, directory: process.cwd(), @@ -1645,6 +1646,11 @@ describe.skipIf(process.platform === "win32")("the run record", () => { await expect(gate(command, { host: ledger() })).rejects.toThrow(/cannot record its run in .*missing/); outermost("record.jsonl"); await expect(gate(command, { host: ledger() })).rejects.toThrow("must be an absolute path"); + const link = join(dir, "link.jsonl"); + symlinkSync(join(dir, "target.jsonl"), link); + outermost(link); + await expect(gate(command, { host: ledger() })).rejects.toThrow(/cannot record its run in .*link\.jsonl/); + expect(existsSync(join(dir, "target.jsonl"))).toBe(false); expect(existsSync(ran)).toBe(false); }); From 0a38eb72e0c2bc0adc1b1602f4860e8664000bc9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 23:04:39 -0800 Subject: [PATCH 06/13] fix(gate): a cancelled run exits as it would have when its record fails Co-Authored-By: Claude Opus 5.5 --- README.md | 10 ++++++---- src/cli/gate.ts | 4 +++- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index ebc90a79d..fa50e1b5d 100644 --- a/README.md +++ b/README.md @@ -1305,8 +1305,10 @@ however the run ends: when the gate learned that the command's own process exited, within milliseconds, before it stopped what the command left behind. A run cancelled before admission has a null `admitted`. -- `cpuMs` and `peakMemory` (bytes) are null when the command never ran, and - `peakMemory` also where the host cannot say. `measured` names what they +- `cpuMs` and `peakMemory` (bytes) are null when the gate never started the + command, as when cancelled before admission, and `peakMemory` also where the + host cannot say. A command that cannot be found still counts the little its + launch used. `measured` names what they count: - `cgroup`: a child cgroup of the command's systemd scope, so every process the command started. Peak memory is its `memory.peak`, which needs Linux @@ -1325,8 +1327,8 @@ leave only whole lines; give each agent its own file to tell them apart. Nested gates write nothing, since their outermost gate already counts their command. An unset or empty variable writes nothing, a relative path or a symlink is refused, and a file the gate cannot open fails it before the command runs. A -new file is readable only by its user. A line it cannot -write fails the gate too, though a cancelled run keeps its cancelled status. +new file is readable only by its user. A line it cannot write fails the gate +too, except that a cancelled run exits as it would have without the record. Windows refuses the variable for now. The agent's own processes write the file, so a record is only as trustworthy as the agent. diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 6e20b284f..659d1982e 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -923,8 +923,10 @@ export async function gate(command: string[], options: GateOptions = {}): Promis if (written !== bytes.length) throw new Error(`wrote ${written} of ${bytes.length} bytes`); } catch (error) { const unrecorded = `gate cannot record its run in ${record.path}: ${message(error)}`; - if (failure === undefined && cancel.signal.aborted) { + // A cancelled run exits as it would have, recorded or not. + if (cancel.signal.aborted) { process.stderr.write(`quest ${unrecorded}\n`); + if (failure !== undefined) throw failure; return status; } throw new Error(failure === undefined ? unrecorded : `${message(failure)}; ${unrecorded}`); From bfba5f4165e4e101d7e69a00fbe1c46dcd0c11ff Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 23:21:03 -0800 Subject: [PATCH 07/13] quest: plan the flaky logged-out review test Co-Authored-By: Claude Opus 5.5 --- quest/m0/review-login-flake.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 quest/m0/review-login-flake.md diff --git a/quest/m0/review-login-flake.md b/quest/m0/review-login-flake.md new file mode 100644 index 000000000..5f4d56788 --- /dev/null +++ b/quest/m0/review-login-flake.md @@ -0,0 +1,11 @@ +# [S] Fix the flaky logged-out review test + +## Goal + +`test/review.test.ts`'s "a login the API rejects fails as logged out" passes on every run, with the flake's root cause fixed rather than retried or skipped. + +## Plan + +- Known: during validation of [#685](https://github.com/moq-dev/quest/pull/685), a full `just test` run failed this test once; it then passed three reruns. #685 does not touch the code it covers. +- Reproduce it first, such as by running the test in a loop or under the full suite's load, then fix the cause in the test or the code it exercises. +- The maintainer chose: ✅ add a quest to reproduce the flake and fix it at the root cause (rejected: plan it only if it recurs). From b45f33218196a7a1678199e1ef2d9eef10229564 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 00:17:05 -0800 Subject: [PATCH 08/13] fix(gate): apply a cancellation that arrives during cleanup to the exit status Co-Authored-By: Claude Opus 5.5 --- src/cli/gate.ts | 7 ++++++- src/cli/supervise.ts | 8 +++++++- test/gate.test.ts | 15 +++++++++++++++ 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 97f0a63c3..0cf1fec42 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -799,7 +799,12 @@ export const group: Owner = (command, env, cancel) => return reject(error); } const cpuMs = Math.round(Number(usage.cpuTime.total) / 1000); - resolve({ status, ended, usage: { cpuMs, peakMemory: usage.maxRSS, measured: "children" } }); + // A cancellation that arrived while the holders drained still decides the status. + resolve({ + status: cancel.signal.aborted ? cancelled(cancel.signal) : status, + ended, + usage: { cpuMs, peakMemory: usage.maxRSS, measured: "children" }, + }); }); }); diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index 896a39da7..c628195f5 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -521,9 +521,13 @@ function contain( }); child.once("exit", async (code, signal) => { const exit: Exit = { - status: interrupted || (code ?? 128 + constants.signals[signal!]), + status: code ?? 128 + constants.signals[signal!], ended: Date.now(), }; + // A cancellation that arrives during the cleanup below still decides the status. + const settle = () => { + if (interrupted) exit.status = interrupted; + }; try { // An exit can arrive before the report it follows, such as `started` from a short command. await reported; @@ -574,8 +578,10 @@ function contain( // The shell's own failure; one killed before it ran the command reports that signal instead. if (admitted && !started && !interrupted && code === 125) throw new Error(`${describe.unsupported}: the command never entered its cgroup`); + settle(); resolve(exit); } catch (error) { + settle(); reject(error); } finally { cleanup(); diff --git a/test/gate.test.ts b/test/gate.test.ts index 8178b4fea..c86180e5a 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -994,6 +994,21 @@ describe.each(owners)("an owner using %s", (_name, run) => { }); }); +test.skipIf(process.platform !== "linux")( + "a cancellation during the scope's cleanup still decides the status", + async () => { + const { stop, cancel } = cancellation(); + const exit = await systemdScopes.run(`quest-gate-${randomUUID()}.scope`, tempDir(), ["true"], { + cancel, + within: null, + // Listeners run before the owner's exit handler resumes, so this aborts after the command's + // own status is known and before its scope is cleaned up. + spawned: (child) => child.once("exit", () => queueMicrotask(() => stop.abort("SIGINT"))), + }); + expect(exit.status).toBe(128 + constants.signals.SIGINT); + }, +); + /** * Just, as a recipe like `[script("quest", "gate", "--", "bash")]` has it, runs an outermost gate * that owns `command` through a process group, as on macOS or Linux without a systemd user From 76a5cef3bc67606fe06abf291eb13acbf761d78c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 08:43:37 -0800 Subject: [PATCH 09/13] refactor(gate): record the scope's unit and wait, not its usage Maintainer decision (2026-11-13): systemd already logs each scope's CPU time and peak memory in the user journal, so the record names the scope (`unit`) and the queue wait for a reader to join against it. Drops the delegated scope with its holder and command child cgroups, the Exit and Usage types, and the process group's rusage through Bun.spawn. With no measurement left to port, Windows records too (unit null), so the Windows record quest is deleted. Also corrects `quest worktree run`'s help, which called its scope delegated. Co-Authored-By: Claude Opus 5.5 --- README.md | 49 +++----- quest/m1/gate-record-windows.md | 33 ------ src/cli/gate.ts | 149 +++++++++-------------- src/cli/main.ts | 9 +- src/cli/supervise.ts | 201 ++++++-------------------------- src/cli/worktree.ts | 2 +- test/agent.test.ts | 2 +- test/gate.test.ts | 163 ++++++-------------------- test/worktree.run.test.ts | 4 +- test/worktree.test.ts | 10 +- 10 files changed, 156 insertions(+), 466 deletions(-) delete mode 100644 quest/m1/gate-record-windows.md diff --git a/README.md b/README.md index 05bc67c27..899ee1a95 100644 --- a/README.md +++ b/README.md @@ -1249,8 +1249,7 @@ gate exits with 128 plus the signal's number. How far ownership reaches depends on the host: - Linux with a systemd 254+ user manager, unified cgroup v2, and Linux 5.14+ - runs the command in a child cgroup of its own delegated systemd user scope - in `background.slice`, whose + runs the command in its own systemd user scope in `background.slice`, whose low CPU weight keeps the desktop session ahead of gated work under contention, and owns every descendant, including ones that start their own process group or session. Only a process that asks another service manager to start it @@ -1302,46 +1301,36 @@ command waits its turn for as long as the host stays busy. ### Record each run A process that starts an agent, such as a job runner or a CPU audit, can read -back what the agent's gated commands waited for and cost by setting -`QUEST_GATE_RECORD` in the agent's environment to an absolute path. Every -outermost `quest gate` run on Linux or macOS, including those the agent's -parallel shells and subagents start, then appends one JSON line to that file, -however the run ends: +back what the agent's gated commands waited for by setting `QUEST_GATE_RECORD` +in the agent's environment to an absolute path. Every outermost `quest gate` +run, including those the agent's parallel shells and subagents start, then +appends one JSON line to that file, however the run ends: ```json -{"command":["cargo","test"],"directory":"/home/me/src/app","started":"2026-11-12T19:40:01.120Z","admitted":"2026-11-12T19:40:31.452Z","ended":"2026-11-12T19:42:10.008Z","cpuMs":512340,"peakMemory":3221225472,"measured":"cgroup","exit":0,"cancelled":false,"error":null} +{"command":["cargo","test"],"directory":"/home/me/src/app","started":"2026-11-12T19:40:01.120Z","admitted":"2026-11-12T19:40:31.452Z","unit":"quest-gate-0d6c3f9e-5a51-4c1b-9d0e-2f4b8a7c6e10.scope","exit":0,"cancelled":false,"error":null} ``` -- `started`, `admitted`, and `ended` give the wait and the wall time. `ended` is - when the gate learned that the command's own process exited, within - milliseconds, before it stopped what the command left behind. A run cancelled - before admission has a null `admitted`. -- `cpuMs` and `peakMemory` (bytes) are null when the gate never started the - command, as when cancelled before admission, and `peakMemory` also where the - host cannot say. A command that cannot be found still counts the little its - launch used. `measured` names what they - count: - - `cgroup`: a child cgroup of the command's systemd scope, so every process - the command started. Peak memory is its `memory.peak`, which needs Linux - 5.19 and also counts page cache charged to it. - - `children`: on macOS, and Linux without a systemd user manager, the - command's own resource usage, a lower bound: it covers the descendants the - command waited for, not those the gate kills at the end, such as a leftover - background server, and its peak memory is the largest single process's. +- `started` and `admitted` give the wait. A run cancelled before admission has + a null `admitted`. +- `unit` is the command's systemd user scope, null where the gate owns it + another way, such as on macOS or Windows. Once the scope ends, systemd logs + `: Consumed 1.386s CPU time over 1.357s wall clock time, 8.1M memory + peak.` to the user journal (`journalctl --user -u `), but only when the + scope used a notable amount, such as a second of CPU; a run with no such line + used less. - `exit` is the status the gate returns, `cancelled` whether it was cancelled, - and `error` the gate's own failure, such as a scope it cannot confirm stopped - or counters it cannot read. A command that cannot be found is not one: it - exits 127, as in a shell. + and `error` the gate's own failure, such as a scope it cannot confirm + stopped. Each line is one write to a file opened for appending, so concurrent gates leave only whole lines; give each agent its own file to tell them apart. Nested -gates write nothing, since their outermost gate already counts their command. +gates write nothing, since their command runs inside their outermost gate's. An unset or empty variable writes nothing, a relative path or a symlink is refused, and a file the gate cannot open fails it before the command runs. A new file is readable only by its user. A line it cannot write fails the gate too, except that a cancelled run exits as it would have without the record. -Windows refuses the variable for now. The agent's own processes write the -file, so a record is only as trustworthy as the agent. +The agent's own processes write the file, so a record is only as trustworthy +as the agent. ## Route work by size diff --git a/quest/m1/gate-record-windows.md b/quest/m1/gate-record-windows.md deleted file mode 100644 index 8eec780d1..000000000 --- a/quest/m1/gate-record-windows.md +++ /dev/null @@ -1,33 +0,0 @@ -# [M] The gate's run record on Windows - -## Goal - -On Windows, `QUEST_GATE_RECORD` works as it does on Linux and macOS: each -outermost `quest gate` run appends one line, with CPU time and peak -memory taken from the Job Object that owns the command and everything it -starts. - -## Plan - -Decided 2026-11-12 (maintainer, `/quest-plan`): split from the gate run -record, since m0 leaves native Windows and its suite to m1. The README's -"Record each run" section gives the variable and the line format; until -this lands, `quest gate` on Windows refuses `QUEST_GATE_RECORD`. - -The Job Object lives in the Bun stub that `src/cli/job.ts` runs, and it -ends with the stub, so the gate never holds it. The stub reads the job's -basic accounting (total user and kernel time) and its peak job memory -once the command exits, before it closes the handle, and passes them to -the gate over a channel of its own, such as an extra pipe, keeping the -command's stdio untouched. Its line's `measured` is `job`, since the -job covers everything the command started. Without a job, which the stub -only warns about today, the line carries `children` from the stub's own -child, as on a process group. - -Tests, in the native Windows suite: a stub command's line carries its -CPU and peak memory, a grandchild's CPU counts toward it, and a run -without a job is labeled `children`. - -## Hosts - -- windows - the Job Object accounting diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 0cf1fec42..22ffe9074 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -19,29 +19,11 @@ import { } from "node:fs"; import { availableParallelism, constants, cpus, hostname, loadavg, totalmem, userInfo } from "node:os"; import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; +import type { Readable } from "node:stream"; import { setTimeout as sleep } from "node:timers/promises"; import { type Cancellation, cancelled, follow, forwards, GROUP, interrupts, received, requested } from "./interrupt"; import { stubbed } from "./job"; -import { type Exit, questScope, scopeGroup, startTime, systemdScopes } from "./supervise"; - -/** - * The part of Bun's process API the group owner needs: Node's `child_process` reports no resource - * usage of a child, and Quest always runs on Bun. - */ -declare const Bun: { - spawn( - command: string[], - options: { env: NodeJS.ProcessEnv; stdio: ["inherit", "inherit", "inherit", "pipe"]; detached: true }, - ): { - pid: number; - stdio: [null, null, null, number]; - exited: Promise; - exitCode: number | null; - signalCode: NodeJS.Signals | null; - resourceUsage(): { cpuTime: { total: number | bigint }; maxRSS: number } | undefined; - }; - file(fd: number): { stream(): ReadableStream }; -}; +import { questScope, scopeGroup, startTime, systemdScopes } from "./supervise"; /** * One admitted command: a unique id, the wall-clock time it was admitted, and the process ID of @@ -638,13 +620,12 @@ const ADMITTED = "QUEST_GATE_ADMITTED"; export const PRIORITY = "QUEST_GATE_PRIORITY"; /** - * Runs an admitted command with inherited streams and resolves with its exact exit status, when it - * exited, and what it used where the owner measures that. + * Runs an admitted command with inherited streams and resolves with its exact exit status. * Asking to cancel delivers the requested signal to every process the owner holds, unless they * already have it (see `forwards`), so each one's own handling of it runs; killing stops them * outright. */ -export type Owner = (command: string[], env: NodeJS.ProcessEnv, cancel: Cancellation) => Promise; +export type Owner = (command: string[], env: NodeJS.ProcessEnv, cancel: Cancellation) => Promise; /** * Run the command as a plain child. `held` says whether an enclosing owner that signals everything @@ -670,7 +651,7 @@ function plain(held: boolean): Owner { }); child.once("close", (code, killed) => { cleanup(); - resolve({ status: code ?? (killed ? 128 + constants.signals[killed] : 1), ended: Date.now() }); + resolve(code ?? (killed ? 128 + constants.signals[killed] : 1)); }); }); } @@ -725,36 +706,24 @@ export const job: Owner = (command, env, cancel) => direct(...stubbed(command, e * it or a parent closes it, as Bun and Node do for what they start. Its read end ends once every * holder has exited, so each one, such as a leaf that outlives the wrappers around it, finishes its * own cleanup before the rest of the group is killed, unless a second signal kills it first. - * - * Its usage is the command's own as `wait4` reports it, which covers the descendants it waited for - * but not those the group kills at the end, and as peak memory the largest single process's. */ export const group: Owner = (command, env, cancel) => new Promise((resolve, reject) => { - let child: ReturnType; - try { - child = Bun.spawn(["nice", "-n", "10", ...command], { - env: { ...env, [GROUP]: "1" }, - stdio: ["inherit", "inherit", "inherit", "pipe"], - detached: true, - }); - } catch (error) { - return reject(error); - } - const token = child.stdio[TOKEN]; - // Drained from the start, so a holder's writes never fill the pipe; only its end matters. - const reader = Bun.file(token).stream().getReader(); - const released = (async () => { - try { - while (!(await reader.read()).done); - } catch { - // A failed read ends the wait as the pipe's end would. - } - })(); - let signalled = false; + const child = spawn("nice", ["-n", "10", ...command], { + env: { ...env, [GROUP]: "1" }, + stdio: ["inherit", "inherit", "inherit", "pipe"], + detached: true, + }); + const token = child.stdio[TOKEN] as Readable; + const released = new Promise((resolve) => { + token.once("end", resolve); + token.once("close", resolve); + token.once("error", () => resolve()); + }); + token.resume(); // Holders' writes are discarded; only the end matters. const signal = (name: NodeJS.Signals) => { try { - process.kill(-child.pid, name); + process.kill(-child.pid!, name); } catch (error) { const code = (error as NodeJS.ErrnoException).code; if (code === "ESRCH") return; @@ -769,42 +738,33 @@ export const group: Owner = (command, env, cancel) => if (live) throw error; } }; + let signalled = false; const stop = () => { signalled = true; signal(requested(cancel.signal)); }; const terminate = () => signal("SIGKILL"); + child.once("error", reject); + if (child.pid === undefined) return; // It never started. const cleanup = follow(cancel, stop, terminate); const killed = new Promise((resolve) => { if (cancel.kill.aborted) resolve(); else cancel.kill.addEventListener("abort", () => resolve(), { once: true }); }); - void child.exited.then(async () => { - const ended = Date.now(); - const status = child.exitCode ?? 128 + constants.signals[child.signalCode!]; - const usage = child.resourceUsage(); + child.once("exit", async (code, name) => { // Every process in the group got the signal, so let each holder finish its own cleanup. if (signalled) await Promise.race([released, killed]); cleanup(); + token.destroy(); try { - await reader.cancel(); - // Bun leaves a pipe it handed out open. - closeSync(token); // The group outlives its leader while any member remains, so its ID is not reused yet. A // group that emptied during the drain could only be mistaken once the host cycled through // every process ID. terminate(); - if (!usage) throw new Error("gate cannot read its command's resource usage"); } catch (error) { return reject(error); } - const cpuMs = Math.round(Number(usage.cpuTime.total) / 1000); - // A cancellation that arrived while the holders drained still decides the status. - resolve({ - status: cancel.signal.aborted ? cancelled(cancel.signal) : status, - ended, - usage: { cpuMs, peakMemory: usage.maxRSS, measured: "children" }, - }); + resolve(code ?? 128 + constants.signals[name!]); }); }); @@ -842,7 +802,7 @@ const notices: Record = { /** * Names a file to which every outermost gate appends one JSON line describing its run, however it - * ends (see `gate`). A nested gate runs inside a command its outermost gate already measures. + * ends (see `gate`). A nested gate runs inside a command its outermost gate already covers. */ export const RECORD = "QUEST_GATE_RECORD"; @@ -850,7 +810,6 @@ export const RECORD = "QUEST_GATE_RECORD"; function recordFile(env: NodeJS.ProcessEnv): { path: string; fd: number } | undefined { const path = env[RECORD]; if (!path) return undefined; - if (process.platform === "win32") throw new Error(`gate cannot record its run on Windows yet (${RECORD})`); // The command's shells change directory, so a relative path would scatter one reader's lines. if (!isAbsolute(path)) throw new Error(`${RECORD} must be an absolute path: ${path}`); try { @@ -876,7 +835,8 @@ export interface GateOptions { interface Observed { /** Wall-clock milliseconds when the command was admitted. */ admitted?: number; - exit?: Exit; + /** The systemd scope the command ran in, whose journal lines hold its CPU time and peak memory. */ + unit?: string; } /** @@ -903,41 +863,38 @@ export async function gate(command: string[], options: GateOptions = {}): Promis failure = error; // The status the CLI exits with for a gate's own failure. status = 1; - // A command that ran keeps its measurements even when its scope cannot be confirmed stopped. - if (error instanceof Error && "exit" in error) observed.exit ??= error.exit as Exit; } - if (record) { - const iso = (ms: number) => new Date(ms).toISOString(); - const usage = observed.exit?.usage; - const line = `${JSON.stringify({ + if (!record) { + if (failure !== undefined) throw failure; + return status; + } + const iso = (ms: number) => new Date(ms).toISOString(); + const line = Buffer.from( + `${JSON.stringify({ command, directory, started: iso(started), admitted: observed.admitted === undefined ? null : iso(observed.admitted), - ended: iso(observed.exit?.ended ?? Date.now()), - cpuMs: usage?.cpuMs ?? null, - peakMemory: usage?.peakMemory ?? null, - measured: usage?.measured ?? null, + unit: observed.unit ?? null, exit: status, cancelled: cancel.signal.aborted, error: failure === undefined ? null : message(failure), - })}\n`; - try { - const bytes = Buffer.from(line); - const written = writeSync(record.fd, bytes); - if (written !== bytes.length) throw new Error(`wrote ${written} of ${bytes.length} bytes`); - } catch (error) { - const unrecorded = `gate cannot record its run in ${record.path}: ${message(error)}`; - // A cancelled run exits as it would have, recorded or not. - if (cancel.signal.aborted) { - process.stderr.write(`quest ${unrecorded}\n`); - if (failure !== undefined) throw failure; - return status; - } - throw new Error(failure === undefined ? unrecorded : `${message(failure)}; ${unrecorded}`); - } finally { - closeSync(record.fd); + })}\n`, + ); + try { + const written = writeSync(record.fd, line); + if (written !== line.length) throw new Error(`wrote ${written} of ${line.length} bytes`); + } catch (error) { + const unrecorded = `gate cannot record its run in ${record.path}: ${message(error)}`; + // A cancelled run exits as it would have, recorded or not. + if (cancel.signal.aborted) { + process.stderr.write(`quest ${unrecorded}\n`); + if (failure !== undefined) throw failure; + return status; } + throw new Error(failure === undefined ? unrecorded : `${message(failure)}; ${unrecorded}`); + } finally { + closeSync(record.fd); } if (failure !== undefined) throw failure; return status; @@ -989,13 +946,15 @@ async function gated( const env: NodeJS.ProcessEnv = { ...process.env, [ADMITTED]: "1" }; // mbx otherwise takes its configured priority, which agents' hosts may lower. if (process.env[PRIORITY] === "1") env.MBX_SCHEDULER_PRIORITY ??= "normal"; + observed.unit = unit; + let result: number; try { - observed.exit = await run(command, env, cancel); + result = await run(command, env, cancel); } finally { stop.abort(); } const failed = await measuring; if (cancel.signal.aborted) return cancelled(cancel.signal); if (failed) throw failed.error; - return observed.exit.status; + return result; } diff --git a/src/cli/main.ts b/src/cli/main.ts index 7939b4e6e..94ee3adac 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -149,9 +149,10 @@ export function main(argv: string[], out: Output): number | Promise { "descendant that leaves the group or session escapes. Windows owns every descendant through a " + "kill-on-close Job Object. Exits with the command's status, or 128 plus the signal's number once " + "cancelled.\n\n" + - "With QUEST_GATE_RECORD naming an absolute path, an outermost gate on Linux or macOS appends one JSON " + - "line per run to that file, however the run ends: its command, directory, when it started, was " + - "admitted, and ended, its CPU time, peak memory, and exit status. A file it cannot write fails the gate.", + "With QUEST_GATE_RECORD naming an absolute path, an outermost gate appends one JSON line per run to " + + "that file, however the run ends: its command, directory, when it started and was admitted, its " + + "systemd scope, whose journal holds its CPU time and peak memory, and its exit status. A file it " + + "cannot write fails the gate.", ) .option("--quick", `Start a cheap command after at most ${QUICK_MS / 1000} seconds of waiting`) .argument("", "Executable to run") @@ -906,7 +907,7 @@ export function main(argv: string[], out: Output): number | Promise { .description( "Run a command in the checkout, restoring it first when released, without the repository-location " + "variables, such as GIT_DIR, that a Git hook exports. The command and its descendants " + - "run in a new, delegated systemd user scope; once the command exits, descendants still in the scope " + + "run in a new systemd user scope; once the command exits, descendants still in the scope " + "are killed. A `quest gate` inside it runs its command in a scope of its own, bound to this one. " + "When no other supervised run remains, the scratch declared at creation is pruned. Put the whole job, " + "such as an agent session or a script that installs, builds, and tests, in one run.\n\n" + diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index c628195f5..7bde5a572 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -1,9 +1,7 @@ // Linux process ownership for supervised checkout commands and gated ones. Each run gets its own // systemd user scope, named once and never reused, so its cgroup holds every descendant that does // not ask another service manager to start it. Other hosts fail explicitly; only the gate settles -// for a process group there (see `group` in gate.ts). The scope is delegated: the command runs in a -// child cgroup whose counters cover everything it started, and a holder in another keeps the scope, -// and with it those counters, until the owner has read them. +// for a process group there (see `group` in gate.ts). import { type ChildProcessWithoutNullStreams, execFileSync, spawn } from "node:child_process"; import { closeSync, existsSync, constants as fs, openSync, readdirSync, readFileSync, watch, writeSync } from "node:fs"; import { constants, hostname } from "node:os"; @@ -53,30 +51,6 @@ export function alive(owner: Owner): boolean { } } -/** What an owner measured of its command, including what that command started. */ -export interface Usage { - /** CPU time, in milliseconds. */ - cpuMs: number; - /** Peak memory in bytes, or null where the host cannot say. */ - peakMemory: number | null; - /** - * `cgroup`: a scope's counters, which cover every process in it. `children`: the command's own - * resource usage, which covers only the descendants it waited for, and as peak memory the largest - * single process's. - */ - measured: "cgroup" | "children"; -} - -/** How an owned command ended. */ -export interface Exit { - /** Its exit status, 128 plus the signal's number when signalled, or the cancelled status. */ - status: number; - /** Wall-clock milliseconds when the command's own process exited, before its leftovers were stopped. */ - ended: number; - /** Undefined where the owner measures nothing, or lost the count to an enclosing owner's kill. */ - usage?: Usage; -} - /** How a scope runs its command. */ export interface RunOptions { /** The command's environment; Quest's own when omitted. */ @@ -112,47 +86,27 @@ export interface Scopes { * Run a command in a new scope, then stop its remaining descendants: at once, or, when a * cancellation signalled the whole scope, once the scope empties or the run is killed, so each * process can finish its own cleanup. Resolves with the command's exit status, or the cancelled - * status once asked to stop, only once the scope is empty, with the scope's counters. Rejects with - * `Retained` when that cannot be confirmed, and otherwise only when the command never started or - * its counters cannot be read. + * status once asked to stop, only once the scope is empty. Rejects with `Retained` when that + * cannot be confirmed, and otherwise only when the command never started. */ - run(unit: string, cwd: string, command: string[], options?: RunOptions): Promise; + run(unit: string, cwd: string, command: string[], options?: RunOptions): Promise; } -/** - * The command may have started and its scope may still hold processes, so its run stays recorded. - * `exit` is how the command ended, where its holder reported that before the failure. - */ -export class Retained extends Error { - exit?: Exit; -} +/** The command may have started and its scope may still hold processes, so its run stays recorded. */ +export class Retained extends Error {} export const UNSUPPORTED = "quest worktree run needs Linux 5.14 or later with a systemd user manager (systemd 254 or later) and unified cgroup v2"; /** - * The shell proves it runs in the scope's own v2 cgroup, creates the child cgroups `holder` and - * `command`, and waits for the supervisor before it executes the command in `command`. A - * supervisor that died never acknowledges, so the command never starts. The holder, a subshell - * that ignores the signals forwarded to every process in the scope, waits in `holder` for the - * acknowledgement pipe to close, which the supervisor does once it has stopped the command's - * leftovers and read its counters, so systemd cannot remove the scope first. With both moved out - * of the scope's root, the memory controller can be enabled below it, so `command` counts its own - * peak. The command itself stays the process the supervisor started, which `quest agent stop` signals. + * The shell proves it runs in the scope's own v2 cgroup, then waits for the supervisor before it + * executes the command. A supervisor that died never acknowledges, so the command never starts. */ const handshake = "IFS=: read -r hierarchy controllers path &3; ' + "read -r admitted <&4 || exit 125; " + - // Forked twice, so the command never inherits the holder as a child it might wait for. - '(echo 0 >"$scope/holder/cgroup.procs" || exit 125; ' + - '(trap "" INT TERM HUP; exec /dev/null 2>&1 3>&-; read -r released <&4) &) || exit 125; ' + - 'echo 0 >"$scope/command/cgroup.procs" || exit 125; ' + - 'read -r available <"$scope/cgroup.controllers"; ' + - 'case " $available " in *" memory "*) echo +memory >"$scope/cgroup.subtree_control" || exit 125;; esac; ' + - "echo started >&3; " + 'exec 3>&- 4<&-; exec "$@"'; function populated(group: string): boolean { @@ -240,7 +194,6 @@ export function kill(group: string): void { */ function signalGroup(group: string, signal: NodeJS.Signals): void { const signalled = new Set(); - let failure: unknown; for (let fresh = true; fresh;) { fresh = false; for (const pid of members(group)) { @@ -250,33 +203,14 @@ function signalGroup(group: string, signal: NodeJS.Signals): void { try { process.kill(pid, signal); } catch (error) { - // Such as EPERM from a process running as another user: still signal the rest. - if ((error as NodeJS.ErrnoException).code !== "ESRCH") failure ??= error; + if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; } } } - if (failure) throw failure; -} - -/** - * The CPU time and peak memory of every process that ran in the command's cgroup `inner`, which - * last only while the holder keeps the scope from being removed. `memory.peak` needs Linux 5.19 - * and the memory controller. - */ -function counters(inner: string): Usage { - const usage = /^usage_usec (\d+)$/m.exec(readFileSync(join(inner, "cpu.stat"), "utf8"))?.[1]; - if (usage === undefined) throw new Error(`cannot read the CPU use of ${inner}`); - let peakMemory: number | null = null; - try { - peakMemory = Number(readFileSync(join(inner, "memory.peak"), "utf8")); - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; - } - return { cpuMs: Math.round(Number(usage) / 1000), peakMemory, measured: "cgroup" }; } /** The processes in the cgroup and every cgroup below it; none once it is removed. */ -function members(group: string): number[] { +export function members(group: string): number[] { try { const own = readFileSync(join(group, "cgroup.procs"), "utf8").split("\n").filter(Boolean).map(Number); const children = readdirSync(group, { withFileTypes: true }).filter((entry) => entry.isDirectory()); @@ -362,8 +296,6 @@ export const systemdScopes: Scopes = { "--scope", "--quiet", "--collect", - // The holder creates the command's child cgroup (see `handshake`). - "--property=Delegate=yes", `--unit=${unit}`, ...bound, ...(options.slice ? [`--slice=${options.slice}`] : []), @@ -408,8 +340,8 @@ interface Containment { /** * Start the handshake shell through `launch`, admit the command once the shell reports a cgroup - * that `owns` accepts, and once the command exits, kill whatever is left of it, read the counters, - * release the holder, and wait until the scope is empty (see `Scopes.run`). + * that `owns` accepts, and once the command exits, kill whatever is left in that cgroup and wait + * until it is empty (see `Scopes.run`). */ function contain( launch: string[], @@ -418,7 +350,7 @@ function contain( cwd: string, command: string[], options: RunOptions, -): Promise { +): Promise { return new Promise((resolve, reject) => { const own = options.cancel ? undefined : interrupts(); const cancel = options.cancel ?? own!; @@ -431,12 +363,8 @@ function contain( options.spawned?.(child as ChildProcessWithoutNullStreams); const report = child.stdio[3] as Readable; const admission = (child.stdio as unknown as (Writable | null)[])[4]!; - // The scope's cgroup, and the command's child cgroup in it. let group: string | undefined; - let inner: string | undefined; let admitted = false; - // Set once the command runs in `inner`, beside the holder. - let started = false; let interrupted = 0; let failure: Error | undefined; // Set once every process in the cgroup was asked to stop; aborted once they no longer get time to. @@ -451,11 +379,13 @@ function contain( const name = requested(cancel.signal); signalled = true; try { - // The holder ignores it; every other process runs its own handling. signalGroup(group!, name); } catch (error) { - // The rest got the signal; one that did not is no reason to wait for the scope to empty. - process.stderr.write(`quest: cannot signal every process in ${describe.name}: ${error}\n`); + // Such as EPERM from a process running as another user: signal what this process can. + process.stderr.write( + `quest: cannot signal every process in ${describe.name}, only its command: ${error}\n`, + ); + child.kill(name); drain.abort(); } }; @@ -464,51 +394,33 @@ function contain( // The enclosing owner kills this cgroup at its own second signal (see `bound`). if (describe.bound && received(cancel.signal)) return; drain.abort(); - // Once the command runs, the holder outlives it, so the scope's counters can still be read. - const target = started ? inner : group; - if (target) kill(target); + if (group) kill(group); }; const unfollow = follow(cancel, stop, terminate); const cleanup = () => { unfollow(); own?.close(); }; - const retained = (error: unknown) => - new Retained( - `cannot confirm that ${describe.name} stopped: ${error instanceof Error ? error.message : error}`, - ); admission.on("error", (error: NodeJS.ErrnoException) => { if (error.code !== "EPIPE") failure = error; }); let text = ""; - // Every writer closes the report before the command runs, so it ends however the run goes. - const reported = new Promise((resolve) => { - report.once("close", resolve); - report.once("error", () => resolve()); - }); report.setEncoding("utf8"); - // The shell reports the scope's cgroup, then `started` once the command is in its own. report.on("data", (data: string) => { text += data; - for (let end = text.indexOf("\n"); end >= 0; end = text.indexOf("\n")) { - const line = text.slice(0, end); - text = text.slice(end + 1); - if (group === undefined) { - // A unique name can never name a later cgroup, so this path stays this run's. - if (!line.startsWith("/") || !owns(line)) { - failure = new Error(`${describe.unsupported}: the command ran outside its cgroup (${line})`); - admission.destroy(); - return; - } - group = join("/sys/fs/cgroup", line); - inner = join(group, "command"); - if (interrupted) return; - admitted = true; - admission.write("admitted\n"); - } else if (line === "started" && !started) started = true; - else failure ??= new Error(`${describe.launcher} reported an unexpected line: ${line}`); + if (group !== undefined || !text.endsWith("\n")) return; + const path = text.slice(0, -1); + // A unique name can never name a later cgroup, so this path stays this run's. + if (!path.startsWith("/") || path.includes("\n") || !owns(path)) { + failure = new Error(`${describe.unsupported}: the command ran outside its cgroup (${path})`); + admission.destroy(); + return; } + group = join("/sys/fs/cgroup", path); + if (interrupted) return; + admitted = true; + admission.end("admitted\n"); }); child.once("error", (error) => { @@ -520,68 +432,29 @@ function contain( ); }); child.once("exit", async (code, signal) => { - const exit: Exit = { - status: code ?? 128 + constants.signals[signal!], - ended: Date.now(), - }; - // A cancellation that arrives during the cleanup below still decides the status. - const settle = () => { - if (interrupted) exit.status = interrupted; - }; try { - // An exit can arrive before the report it follows, such as `started` from a short command. - await reported; // Descendants that outlive the command are orphans of this run: stop them. if (group) { - let counted: unknown; - try { - if (started) { - // Every process got the signal, so let each finish its own cleanup, even one - // that outlives the wrappers that exited first, until a second signal kills them. - if (signalled) await vacated(inner!, drain.signal); - kill(inner!); - await vacated(inner!); - try { - exit.usage = counters(inner!); - } catch (error) { - counted = error; - } - } - } catch (error) { - throw Object.assign(retained(error), { exit }); - } finally { - // The holder reads end-of-file and leaves, so the scope empties. - if (!admission.destroyed) admission.end(); - } try { - // Before the command started, the signalled processes still get their cleanup. - if (signalled && !started) await vacated(group, drain.signal); + // Every process got the signal, so let each finish its own cleanup, even one that + // outlives the wrappers that exited first, until a second signal kills them. + if (signalled) await vacated(group, drain.signal); kill(group); await vacated(group); await describe.emptied(); } catch (error) { - throw Object.assign(retained(error), { exit }); - } - if (counted) - throw Object.assign( - new Error( - `cannot read the counters of ${describe.name}: ${counted instanceof Error ? counted.message : counted}`, - ), - { exit }, + throw new Retained( + `cannot confirm that ${describe.name} stopped: ${error instanceof Error ? error.message : error}`, ); + } } if (failure) throw failure; if (!admitted && !interrupted) throw new Error( `${describe.unsupported}: ${describe.launcher} exited with ${code ?? signal} before the command started`, ); - // The shell's own failure; one killed before it ran the command reports that signal instead. - if (admitted && !started && !interrupted && code === 125) - throw new Error(`${describe.unsupported}: the command never entered its cgroup`); - settle(); - resolve(exit); + resolve(interrupted || (code ?? 128 + constants.signals[signal!])); } catch (error) { - settle(); reject(error); } finally { cleanup(); diff --git a/src/cli/worktree.ts b/src/cli/worktree.ts index dfc1b1517..3bd7b4454 100644 --- a/src/cli/worktree.ts +++ b/src/cli/worktree.ts @@ -935,7 +935,7 @@ export class Worktrees { // The command runs in this checkout, where a location inherited from a Git hook names the wrong repository. const env = discoveryEnv(options?.env); const result = await this.scopes.run(run.unit, record.path, command, { ...options, env }).then( - ({ status }) => ({ code: status }), + (code) => ({ code }), (error: unknown) => ({ error }), ); // Waits out a busy lock rather than failing: the run's record keeps its scratch until this lands. diff --git a/test/agent.test.ts b/test/agent.test.ts index 04269b55b..c175554ab 100644 --- a/test/agent.test.ts +++ b/test/agent.test.ts @@ -193,7 +193,7 @@ async function managed() { }); options!.spawned!(child); child.once("error", reject); - child.once("close", (code) => resolve({ status: code ?? 1, ended: Date.now() })); + child.once("close", (code) => resolve(code ?? 1)); }), ); writeFileSync(join(s.worktree, ".gitignore"), "cache/\n.worktrees/\n"); diff --git a/test/gate.test.ts b/test/gate.test.ts index c86180e5a..e67a4d94f 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -43,7 +43,7 @@ import { } from "../src/cli/gate"; import { forwards, foregroundStat, statProcess } from "../src/cli/interrupt"; import { gateBoundary, main } from "../src/cli/main"; -import { alive as ownerAlive, questScope, systemdScopes } from "../src/cli/supervise"; +import { alive as ownerAlive, questScope } from "../src/cli/supervise"; import { created, onTerminal, run, signalled, tempDir } from "./fixture"; // Linux reports a /proc read that races a process's exit as ESRCH rather than ENOENT, so fail @@ -739,7 +739,7 @@ function cancellation() { test("a command runs in place with the given environment and its exact exit status", async () => { const script = "process.exit(process.env.QUEST_GATE_ADMITTED === '1' ? 7 : 1)"; const env = { ...process.env, QUEST_GATE_ADMITTED: "1" }; - expect(await exitStatus(direct)([process.execPath, "-e", script], env, cancellation().cancel)).toBe(7); + expect(await direct([process.execPath, "-e", script], env, cancellation().cancel)).toBe(7); }); test("a nested gate runs at once and keeps the exact exit status", async () => { @@ -782,11 +782,7 @@ test.runIf(process.platform === "linux")("a gated command's piped output blocks" test("a command cancelled before listener registration still stops", async () => { const { stop, cancel } = cancellation(); stop.abort(); - const result = await exitStatus(direct)( - [process.execPath, "-e", "setInterval(() => {}, 1000)"], - process.env, - cancel, - ); + const result = await direct([process.execPath, "-e", "setInterval(() => {}, 1000)"], process.env, cancel); expect(result).not.toBe(0); }); @@ -794,11 +790,7 @@ test("a command cancelled before listener registration still stops", async () => test.skipIf(process.platform === "win32")("cancellation forwards the received signal to the command", async () => { const { stop, cancel } = cancellation(); stop.abort("SIGHUP"); - const result = await exitStatus(direct)( - [process.execPath, "-e", "setInterval(() => {}, 1000)"], - process.env, - cancel, - ); + const result = await direct([process.execPath, "-e", "setInterval(() => {}, 1000)"], process.env, cancel); expect(result).toBe(128 + constants.signals.SIGHUP); }); @@ -862,22 +854,17 @@ function descendant(dir: string, rest: string, { leave = false } = {}) { } /** A systemd scope owner with a new unit, as the gate names one. */ -const scoped = exitStatus((command, env, cancel) => scope(`quest-gate-${randomUUID()}.scope`)(command, env, cancel)); - -/** Run through `owner` and resolve with the exit status alone. */ -function exitStatus(owner: Owner): (...args: Parameters) => Promise { - return (...args) => owner(...args).then(({ status }) => status); -} +const scoped: Owner = (command, env, cancel) => scope(`quest-gate-${randomUUID()}.scope`)(command, env, cancel); -const owners: [string, ReturnType][] = +const owners: [string, Owner][] = process.platform === "linux" ? [ ["a systemd scope", scoped], - ["a process group", exitStatus(group)], + ["a process group", group], ] : process.platform === "win32" ? [] - : [["a process group", exitStatus(group)]]; + : [["a process group", group]]; const bin = fileURLToPath(new URL("../src/cli/bin.ts", import.meta.url)); // Bun reads the checkout's config only from its working directory, which Just changes. @@ -994,21 +981,6 @@ describe.each(owners)("an owner using %s", (_name, run) => { }); }); -test.skipIf(process.platform !== "linux")( - "a cancellation during the scope's cleanup still decides the status", - async () => { - const { stop, cancel } = cancellation(); - const exit = await systemdScopes.run(`quest-gate-${randomUUID()}.scope`, tempDir(), ["true"], { - cancel, - within: null, - // Listeners run before the owner's exit handler resumes, so this aborts after the command's - // own status is known and before its scope is cleaned up. - spawned: (child) => child.once("exit", () => queueMicrotask(() => stop.abort("SIGINT"))), - }); - expect(exit.status).toBe(128 + constants.signals.SIGINT); - }, -); - /** * Just, as a recipe like `[script("quest", "gate", "--", "bash")]` has it, runs an outermost gate * that owns `command` through a process group, as on macOS or Linux without a systemd user @@ -1031,7 +1003,7 @@ function outermostUnderJust(dir: string, command: string[]) { for (const name of ["SIGINT", "SIGTERM", "SIGHUP"]) process.on(name, () => writeFileSync(${JSON.stringify(dir)} + "/handled-" + ++handled, "")); writeFileSync(${JSON.stringify(pid)}, String(process.pid)); - const { status } = await group(${JSON.stringify(command)}, process.env, cancel); + const status = await group(${JSON.stringify(command)}, process.env, cancel); cancel.close(); process.exit(cancel.signal.aborted ? cancelled(cancel.signal) : status);`, ); @@ -1116,29 +1088,13 @@ describe.runIf(process.platform === "linux")("a systemd scope", () => { await until(() => stopped(pid())); }); - test("runs the command as the process it spawned, which a signal to that PID reaches", async () => { - const dir = tempDir(); - const ready = join(dir, "ready"); - // What `quest agent stop` does: it signals the PID the supervisor spawned, not the scope. - let pid: number | undefined; - const result = systemdScopes.run( - `quest-gate-${randomUUID()}.scope`, - dir, - ["/bin/sh", "-c", `trap 'exit 3' TERM; : > ${quote(ready)}; while :; do sleep 0.1; done`], - { cancel: cancellation().cancel, within: null, spawned: (child) => (pid = child.pid) }, - ); - await created(ready); - process.kill(pid!, "SIGTERM"); - expect((await result).status).toBe(3); - }); - test("reports its CPU use to the gate until it ends", async () => { const host = fileHost(tempDir()); const unit = `quest-gate-${randomUUID()}.scope`; const done = join(tempDir(), "done"); expect(host.usage(unit)).toBeUndefined(); const spin = ["/bin/sh", "-c", `while [ ! -e ${quote(done)} ]; do :; done`]; - const result = exitStatus(scope(unit))(spin, process.env, cancellation().cancel); + const result = scope(unit)(spin, process.env, cancellation().cancel); await until(() => (host.usage(unit) ?? 0) > 0); writeFileSync(done, ""); expect(await result).toBe(0); @@ -1168,7 +1124,7 @@ async function insideScope(launch: (command: string[], env: NodeJS.ProcessEnv) = if (${run}) { const never = new AbortController().signal; const command = "cat /proc/self/cgroup > " + dir + "/cgroup; systemctl --user show --property=BindsTo --value " + unit + " > " + dir + "/binds; sleep 1000 & echo $! > " + dir + "/pid; exit 7"; - const { status } = await own(["/bin/sh", "-c", command], process.env, { signal: never, kill: never }); + const status = await own(["/bin/sh", "-c", command], process.env, { signal: never, kill: never }); const path = readFileSync(dir + "/cgroup", "utf8").trim().slice(3); const pid = readFileSync(dir + "/pid", "utf8").trim(); const ps = spawnSync("ps", ["-o", "stat=", "-p", pid], { encoding: "utf8" }); @@ -1191,7 +1147,7 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = // Bound to the enclosing scope, a gate's scope here, so that scope's owner stops it too. expect(report.binds).toMatch(/^quest-gate-[0-9a-f-]+\.scope$/); // A sibling, so only the gated command yields the CPU, not the rest of the enclosing scope. - expect(report.path).toMatch(/\/background\.slice\/quest-gate-[0-9a-f-]+\.scope\/command$/); + expect(report.path).toMatch(/\/background\.slice\/quest-gate-[0-9a-f-]+\.scope$/); }); test("is stopped when the enclosing scope's owner kills it", async () => { @@ -1241,7 +1197,7 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = stop.abort(new Error("cancelled by its caller")); }, 25); } - const { status } = await own(${JSON.stringify(command)}, process.env, cancel); + const status = await own(${JSON.stringify(command)}, process.env, cancel); signals.close(); process.exit(cancel.signal.aborted ? cancelled(cancel.signal) : status);`, ); @@ -1330,9 +1286,9 @@ describe("the Windows job stub", () => { const env: NodeJS.ProcessEnv = { ...process.env, EXPECTED: "keep" }; delete env.BUN_BE_BUN; const command = [process.execPath, "-e", script]; - expect(await exitStatus(job)(command, { ...env, BUN_BE_BUN: "keep" }, cancellation().cancel)).toBe(7); + expect(await job(command, { ...env, BUN_BE_BUN: "keep" }, cancellation().cancel)).toBe(7); const unset = "process.exit(process.env.BUN_BE_BUN === undefined ? 7 : 1)"; - expect(await exitStatus(job)([process.execPath, "-e", unset], env, cancellation().cancel)).toBe(7); + expect(await job([process.execPath, "-e", unset], env, cancellation().cancel)).toBe(7); }); test("passes the command's arguments as running it directly would", async () => { @@ -1342,7 +1298,7 @@ describe("the Windows job stub", () => { const script = "require('node:fs').writeFileSync(process.env.OUT, JSON.stringify(process.argv.slice(1)))"; const argv = async (owner: Owner, out: string) => { const env = { ...process.env, OUT: join(dir, out) }; - expect((await owner([process.execPath, "-e", script, ...args], env, cancellation().cancel)).status).toBe(0); + expect(await owner([process.execPath, "-e", script, ...args], env, cancellation().cancel)).toBe(0); return JSON.parse(readFileSync(env.OUT, "utf8")); }; const expected = await argv(direct, "direct"); @@ -1351,7 +1307,7 @@ describe("the Windows job stub", () => { }); test("reports a command that cannot start as a shell would", async () => { - expect(await exitStatus(job)([join(tempDir(), "missing")], process.env, cancellation().cancel)).toBe(127); + expect(await job([join(tempDir(), "missing")], process.env, cancellation().cancel)).toBe(127); }); test("hands the command the caller's environment, not the working directory's .env files", async () => { @@ -1366,7 +1322,7 @@ describe("the Windows job stub", () => { try { // The command is Bun too, so it must not load them either. const command = [process.execPath, "--no-env-file", "-e", script]; - expect(await exitStatus(job)(command, env, cancellation().cancel)).toBe(0); + expect(await job(command, env, cancellation().cancel)).toBe(0); } finally { process.chdir(cwd); } @@ -1400,7 +1356,7 @@ describe.runIf(process.platform === "win32")("a job object", () => { test("keeps the exact exit status and stops a detached descendant", async () => { const { command, pid } = detached(tempDir(), "process.exit(7);"); - expect(await exitStatus(job)(command, process.env, cancellation().cancel)).toBe(7); + expect(await job(command, process.env, cancellation().cancel)).toBe(7); await until(() => !running(pid())); }); @@ -1412,7 +1368,7 @@ describe.runIf(process.platform === "win32")("a job object", () => { `require('node:fs').writeFileSync(${JSON.stringify(ready)}, ''); setInterval(() => {}, 1000);`, ); const { kill, cancel } = cancellation(); - const result = exitStatus(job)(command, process.env, cancel); + const result = job(command, process.env, cancel); await created(ready); kill.abort(); expect(await result).not.toBe(0); @@ -1436,7 +1392,7 @@ describe.skipIf(process.platform === "win32")("a process group", () => { ); const { stop, cancel } = cancellation(); const env = { ...process.env, QUEST_GATE_ADMITTED: "1" }; - const result = exitStatus(group)([process.execPath, bin, "gate", process.execPath, child], env, cancel); + const result = group([process.execPath, bin, "gate", process.execPath, child], env, cancel); await created(ready); stop.abort("SIGINT"); expect(await result).toBe(128 + constants.signals.SIGINT); @@ -1445,7 +1401,7 @@ describe.skipIf(process.platform === "win32")("a process group", () => { test("leaves a descendant that started its own session", async () => { const { command, pid } = descendant(tempDir(), "exit 3", { leave: true }); - expect(await exitStatus(group)(command, process.env, cancellation().cancel)).toBe(3); + expect(await group(command, process.env, cancellation().cancel)).toBe(3); try { expect(stopped(pid())).toBe(false); } finally { @@ -1549,20 +1505,6 @@ test("gate rejects unknown options before the executable and forwards flags afte }); describe.skipIf(process.platform === "win32")("the run record", () => { - /** A Bun command that spins until it has used `ms` of CPU, reports when it ran and its CPU, and exits 3. */ - function spinner(dir: string, ms = 800) { - const out = join(dir, "spun.json"); - const script = - `const start = Date.now(); while (process.cpuUsage().user < ${ms * 1000}); ` + - `const { user, system } = process.cpuUsage(); ` + - `require("node:fs").writeFileSync(${JSON.stringify(out)}, JSON.stringify({ start, end: Date.now(), cpuMs: (user + system) / 1000 })); ` + - `process.exit(3);`; - return { - command: [process.execPath, "-e", script], - spun: (): { start: number; end: number; cpuMs: number } => JSON.parse(readFileSync(out, "utf8")), - }; - } - /** A ledger of the test's own that admits at once, or never when `full`. */ const ledger = (full = false): Host => ({ ...fileHost(tempDir()), @@ -1588,45 +1530,26 @@ describe.skipIf(process.platform === "win32")("the run record", () => { (process.listeners(name).at(-1) as () => void)(); } - test("an outermost run appends one line with its wait, CPU, wall time, and exit", async () => { + test("an outermost run appends one line with its wait, scope, and exit", async () => { const dir = tempDir(); const file = join(dir, "record.jsonl"); + const cgroup = join(dir, "cgroup"); outermost(file); - const { command, spun } = spinner(dir); - const measured = owner(process.env).unit === undefined ? "children" : "cgroup"; + const scoped = owner(process.env).unit !== undefined; + const command = ["/bin/sh", "-c", `cat /proc/self/cgroup > ${quote(cgroup)} 2>/dev/null; exit 3`]; const before = Date.now(); expect(await gate(command, { host: ledger() })).toBe(3); const after = Date.now(); - const ran = spun(); const [line, ...rest] = lines(file); expect(rest).toEqual([]); expect(statSync(file).mode & 0o777).toBe(0o600); - expect(line).toMatchObject({ - command, - directory: process.cwd(), - exit: 3, - cancelled: false, - error: null, - measured, - }); - const at = (field: string) => Date.parse(line[field]); - expect([before, at("started"), at("admitted"), ran.start]).toEqual( - [before, at("started"), at("admitted"), ran.start].sort((a, b) => a - b), - ); - expect([ran.end, at("ended"), after]).toEqual([ran.end, at("ended"), after].sort((a, b) => a - b)); - // The command ends before the gate's first CPU sample, and its whole use still counts. The - // command's own count leaves out only its exec and the owner's few processes around it. - expect(line.cpuMs).toBeGreaterThanOrEqual(Math.floor(ran.cpuMs)); - expect(line.cpuMs).toBeLessThan(ran.cpuMs + 1000); - // A scope's peak needs Linux 5.19 and the memory controller in the slice gated scopes run in. - const uid = process.getuid!(); - const slice = `/sys/fs/cgroup/user.slice/user-${uid}.slice/user@${uid}.service/background.slice`; - const peak = - measured === "children" || - (existsSync(join(slice, "memory.peak")) && - readFileSync(join(slice, "cgroup.subtree_control"), "utf8").split(/\s+/).includes("memory")); - if (peak) expect(line.peakMemory).toBeGreaterThan(0); - else expect(line.peakMemory).toBeNull(); + expect(line).toMatchObject({ command, directory: process.cwd(), exit: 3, cancelled: false, error: null }); + const times = [before, Date.parse(line.started), Date.parse(line.admitted), after]; + expect(times).toEqual([...times].sort((a, b) => a - b)); + // The unit names the scope the command actually ran in, so a reader can find its journal lines. + if (scoped) + expect(readFileSync(cgroup, "utf8").trim()).toMatch(new RegExp(`/${line.unit.replaceAll(".", "\\.")}$`)); + else expect(line.unit).toBeNull(); }); test("a nested gate records nothing, and an empty variable is unset", async () => { @@ -1641,7 +1564,7 @@ describe.skipIf(process.platform === "win32")("the run record", () => { expect(readdirSync(dir)).toEqual([]); }); - test("a run cancelled while queued is recorded with no admission", async () => { + test("a run cancelled while queued is recorded with no admission or scope", async () => { const dir = tempDir(); const file = join(dir, "record.jsonl"); const ran = join(dir, "ran"); @@ -1655,9 +1578,7 @@ describe.skipIf(process.platform === "win32")("the run record", () => { expect(lines(file)).toEqual([ expect.objectContaining({ admitted: null, - cpuMs: null, - peakMemory: null, - measured: null, + unit: null, exit: 128 + constants.signals.SIGHUP, cancelled: true, error: null, @@ -1701,20 +1622,6 @@ describe.skipIf(process.platform === "win32")("the run record", () => { }, ); - test("a process group drains what its command writes to the token", async () => { - const command = ["/bin/sh", "-c", "head -c 1048576 /dev/zero >&3; exit 4"]; - expect((await group(command, process.env, cancellation().cancel)).status).toBe(4); - }); - - test("a process group measures its command's own usage", async () => { - const { command, spun } = spinner(tempDir()); - const exit = await group(command, process.env, cancellation().cancel); - expect(exit.status).toBe(3); - expect(exit.usage).toMatchObject({ measured: "children" }); - expect(exit.usage!.cpuMs).toBeGreaterThanOrEqual(Math.floor(spun().cpuMs)); - expect(exit.usage!.peakMemory).toBeGreaterThan(0); - }); - test("concurrent gates leave only whole lines", async () => { const dir = tempDir(); const file = join(dir, "record.jsonl"); diff --git a/test/worktree.run.test.ts b/test/worktree.run.test.ts index 15d94d182..73816b695 100644 --- a/test/worktree.run.test.ts +++ b/test/worktree.run.test.ts @@ -188,7 +188,7 @@ describe.runIf(process.platform === "linux")("supervised runs under systemd", () const { stop, result } = gated(dir, home, leaf.script); await created(leaf.ready); stop.abort("SIGTERM"); - expect((await result).status).toBe(143); + expect(await result).toBe(143); expect(leaf.log()).toEqual(["SIGTERM"]); // Just's copy of the signal did not kill the run's scope before the leaf's cleanup finished. expect(existsSync(leaf.done)).toBe(true); @@ -202,7 +202,7 @@ describe.runIf(process.platform === "linux")("supervised runs under systemd", () stop.abort("SIGTERM"); await created(leaf.received); kill.abort(); - expect((await result).status).toBe(143); + expect(await result).toBe(143); // The gate killed the supervisor with everything else in its scope, and systemd the run's own. while (!stopped(leaf.pid())) await new Promise((resolve) => setTimeout(resolve, 25)); expect(await worktrees.stop(record.id)).toMatchObject({ phase: "released", runs: [] }); diff --git a/test/worktree.test.ts b/test/worktree.test.ts index 83b011823..94cceb719 100644 --- a/test/worktree.test.ts +++ b/test/worktree.test.ts @@ -41,13 +41,7 @@ const repository: (path: string) => string = template((dir) => { /** Scopes that are never live and whose runs exit at once, except where `overrides` says otherwise. */ function stubScopes(overrides: Partial): Scopes { - return { - check() {}, - live: () => false, - run: () => Promise.resolve({ status: 0, ended: 0 }), - stop: async () => {}, - ...overrides, - }; + return { check() {}, live: () => false, run: () => Promise.resolve(0), stop: async () => {}, ...overrides }; } /** Scopes whose runs finish when the test says so, and stay live while listed until stopped. */ function fakeScopes() { @@ -55,7 +49,7 @@ function fakeScopes() { const finish = new Map void>(); const scopes = stubScopes({ live: (unit) => live.has(unit), - run: (unit) => new Promise((resolve) => finish.set(unit, (status) => resolve({ status, ended: 0 }))), + run: (unit) => new Promise((resolve) => finish.set(unit, resolve)), stop: async (unit) => void live.delete(unit), }); return { live, finish, scopes }; From 5cacb5d6ae9c21dc6d533e34960c92d40f14b97e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 08:43:49 -0800 Subject: [PATCH 10/13] refactor(gate): leave supervise.ts as the base has it Co-Authored-By: Claude Opus 5.5 --- src/cli/supervise.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index 7bde5a572..feac04a22 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -210,7 +210,7 @@ function signalGroup(group: string, signal: NodeJS.Signals): void { } /** The processes in the cgroup and every cgroup below it; none once it is removed. */ -export function members(group: string): number[] { +function members(group: string): number[] { try { const own = readFileSync(join(group, "cgroup.procs"), "utf8").split("\n").filter(Boolean).map(Number); const children = readdirSync(group, { withFileTypes: true }).filter((entry) => entry.isDirectory()); From f67fca1ef02f0aadd562016bf534f04eed9fce94 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 08:43:57 -0800 Subject: [PATCH 11/13] refactor(gate): keep RECORD private Co-Authored-By: Claude Opus 5.5 --- src/cli/gate.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 22ffe9074..76b62bdde 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -804,7 +804,7 @@ const notices: Record = { * Names a file to which every outermost gate appends one JSON line describing its run, however it * ends (see `gate`). A nested gate runs inside a command its outermost gate already covers. */ -export const RECORD = "QUEST_GATE_RECORD"; +const RECORD = "QUEST_GATE_RECORD"; /** Open the file `RECORD` names for appending, before anything runs; undefined while it is unset or empty. */ function recordFile(env: NodeJS.ProcessEnv): { path: string; fd: number } | undefined { From 1a2d4654fba9a1afac1f59b214edf055bd18a096 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 08:47:23 -0800 Subject: [PATCH 12/13] fix(gate): keep refusing the record on Windows, which cannot refuse a symlink Co-Authored-By: Claude Opus 5.5 --- README.md | 7 ++++--- src/cli/gate.ts | 2 ++ src/cli/main.ts | 8 ++++---- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 899ee1a95..e0cc42a43 100644 --- a/README.md +++ b/README.md @@ -1303,8 +1303,8 @@ command waits its turn for as long as the host stays busy. A process that starts an agent, such as a job runner or a CPU audit, can read back what the agent's gated commands waited for by setting `QUEST_GATE_RECORD` in the agent's environment to an absolute path. Every outermost `quest gate` -run, including those the agent's parallel shells and subagents start, then -appends one JSON line to that file, however the run ends: +run on Linux or macOS, including those the agent's parallel shells and +subagents start, then appends one JSON line to that file, however the run ends: ```json {"command":["cargo","test"],"directory":"/home/me/src/app","started":"2026-11-12T19:40:01.120Z","admitted":"2026-11-12T19:40:31.452Z","unit":"quest-gate-0d6c3f9e-5a51-4c1b-9d0e-2f4b8a7c6e10.scope","exit":0,"cancelled":false,"error":null} @@ -1313,7 +1313,7 @@ appends one JSON line to that file, however the run ends: - `started` and `admitted` give the wait. A run cancelled before admission has a null `admitted`. - `unit` is the command's systemd user scope, null where the gate owns it - another way, such as on macOS or Windows. Once the scope ends, systemd logs + another way, such as on macOS. Once the scope ends, systemd logs `: Consumed 1.386s CPU time over 1.357s wall clock time, 8.1M memory peak.` to the user journal (`journalctl --user -u `), but only when the scope used a notable amount, such as a second of CPU; a run with no such line @@ -1329,6 +1329,7 @@ An unset or empty variable writes nothing, a relative path or a symlink is refused, and a file the gate cannot open fails it before the command runs. A new file is readable only by its user. A line it cannot write fails the gate too, except that a cancelled run exits as it would have without the record. +Windows refuses the variable, since it cannot refuse a symlink atomically. The agent's own processes write the file, so a record is only as trustworthy as the agent. diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 76b62bdde..4beb31da7 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -810,6 +810,8 @@ const RECORD = "QUEST_GATE_RECORD"; function recordFile(env: NodeJS.ProcessEnv): { path: string; fd: number } | undefined { const path = env[RECORD]; if (!path) return undefined; + // Windows has no O_NOFOLLOW, and no reader there yet. + if (process.platform === "win32") throw new Error(`gate cannot record its run on Windows (${RECORD})`); // The command's shells change directory, so a relative path would scatter one reader's lines. if (!isAbsolute(path)) throw new Error(`${RECORD} must be an absolute path: ${path}`); try { diff --git a/src/cli/main.ts b/src/cli/main.ts index 94ee3adac..cb67cf6ff 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -149,10 +149,10 @@ export function main(argv: string[], out: Output): number | Promise { "descendant that leaves the group or session escapes. Windows owns every descendant through a " + "kill-on-close Job Object. Exits with the command's status, or 128 plus the signal's number once " + "cancelled.\n\n" + - "With QUEST_GATE_RECORD naming an absolute path, an outermost gate appends one JSON line per run to " + - "that file, however the run ends: its command, directory, when it started and was admitted, its " + - "systemd scope, whose journal holds its CPU time and peak memory, and its exit status. A file it " + - "cannot write fails the gate.", + "With QUEST_GATE_RECORD naming an absolute path, an outermost gate on Linux or macOS appends one JSON " + + "line per run to that file, however the run ends: its command, directory, when it started and was " + + "admitted, its systemd scope, whose journal holds its CPU time and peak memory, and its exit status. " + + "A file it cannot write fails the gate.", ) .option("--quick", `Start a cheap command after at most ${QUICK_MS / 1000} seconds of waiting`) .argument("", "Executable to run") From 12ed6c7ac2c602dbb50bf8367eb77531b44221c1 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 09:04:02 -0800 Subject: [PATCH 13/13] docs(gate): qualify the journal's usage line by log level and version Co-Authored-By: Claude Opus 5.5 --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e0cc42a43..8f683a924 100644 --- a/README.md +++ b/README.md @@ -1315,9 +1315,9 @@ subagents start, then appends one JSON line to that file, however the run ends: - `unit` is the command's systemd user scope, null where the gate owns it another way, such as on macOS. Once the scope ends, systemd logs `: Consumed 1.386s CPU time over 1.357s wall clock time, 8.1M memory - peak.` to the user journal (`journalctl --user -u `), but only when the - scope used a notable amount, such as a second of CPU; a run with no such line - used less. + peak.` to the user journal (`journalctl --user -u `) at info level when + the scope used a notable amount, such as a second of CPU, and at debug level + otherwise. The memory peak needs systemd 255 and Linux 5.19. - `exit` is the status the gate returns, `cancelled` whether it was cancelled, and `error` the gate's own failure, such as a scope it cannot confirm stopped.