Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 6 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1133,18 +1133,13 @@ it, as in arbitrary-UID containers, uses
only its UID can write it. `quest agent` keeps its admissions beside them, in
`agents-v2/<hostname>`.

Two kinds of command go ahead of the rest. `quest gate --quick <command>` waits
at most a minute, then starts the command whatever the host's load, memory, and
place in the queue;
pass it only for cheap commands, such as a formatter or `quest check`, that cost
less to run than to hold back. A check that lands PRs goes first: with
`QUEST_GATE_PRIORITY=1` in its environment, every gate inside the check waits
A check that lands PRs goes ahead of the rest: with `QUEST_GATE_PRIORITY=1` in
its environment, every gate inside the check waits
for memory but never for load or the queue, whether or not other gated commands
run, and its command gets `MBX_SCHEDULER_PRIORITY=normal` unless it sets its
own, so [mbx](https://github.com/jdx/mr-boxington), where installed, schedules
its compiles ahead of lower-priority builds. Both kinds of command
record their admissions, so the other callers count them against the host's
headroom. Set the variable where the check starts, such as a merge queue
its compiles ahead of lower-priority builds. Its admissions are recorded, so
the other callers count them against the host's headroom. Set the variable where the check starts, such as a merge queue
build's workflow, rather than relying on a slot in
the host's ledger: a container's gates keep a ledger of their own, yet wait on
the host's load. `quest agent` drops the variable, as it drops an admission.
Expand Down Expand Up @@ -1217,8 +1212,8 @@ first one as Just's copy, not as a second signal: a SIGTERM to Just's process
group reaches the gate both directly and through Just. A repeated SIGTERM still
kills, though when sent to Just or the gate alone, or when the two copies
arrive together, it takes a third; a repeated Ctrl-C, which Just does not
forward, kills at the second. There is no timeout except `--quick`'s: any other
command waits its turn for as long as the host stays busy.
forward, kills at the second. There is no timeout: a command waits its turn for as long
as the host stays busy.

### Record each run

Expand Down
2 changes: 0 additions & 2 deletions quest/m0/benchmark/gating.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,5 +81,3 @@ Baseline, 2026-11-01 to 11-15:

- [Mistake audit](/quest/m0/mistake-audit.md) - reads these records as a
class each week
- [Remove the gate's quick flag](/quest/m0/gate-quick.md) - the same rule
change leaves the flag no use
24 changes: 0 additions & 24 deletions quest/m0/gate-quick.md

This file was deleted.

2 changes: 1 addition & 1 deletion scripts/lifecycle-check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ mkdir "$work/dotenv"
for file in .env .env.development .env.local; do
printf 'QUEST_DOTENV_LEAK=%s\n' "$file" >"$work/dotenv/$file"
done
seen=$(cd "$work/dotenv" && QUEST_DOTENV_CALLER=kept "$binary" gate --quick env)
seen=$(cd "$work/dotenv" && QUEST_DOTENV_CALLER=kept "$binary" gate env)
if [[ $seen == *QUEST_DOTENV_LEAK=* || $seen != *QUEST_DOTENV_CALLER=kept* ]]; then
echo "quest gate passed its working directory's .env files to its command, or dropped its caller's environment:" >&2
grep '^QUEST_DOTENV_' <<<"$seen" >&2
Expand Down
24 changes: 3 additions & 21 deletions src/cli/gate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,26 +93,16 @@ export const SETTLE_MS = 5000;
*/
export const RECENT_MS = 60_000;

/**
* The longest a quick command waits for room: the time the load average takes
* to reflect new work, so it waits out at most one burst's lag.
*/
export const QUICK_MS = RECENT_MS;

/**
* How a caller is admitted ahead of the rest.
*
* - `priority`: wait only for memory, never for load, whether or not another
* gated command runs. Its admissions still count against every other caller's
* headroom, so it goes first while the host is loaded. For work whose own
* runner bounds it, such as a check that lands PRs.
* - `quick`: wait at most QUICK_MS, then start whatever the load and memory.
* For a command too cheap to be worth holding back, such as a formatter or a
* linter. Once started, it holds a slot like any admission.
*/
export interface Precedence {
priority?: boolean;
quick?: boolean;
}

/** The memory one gated build or test stack is assumed to need. */
Expand Down Expand Up @@ -517,13 +507,11 @@ export function room(limits: Limits, host: Host, reserved: number): Waiting | nu
* lasts until its caller is admitted, cancelled, or its gate ends; one whose
* gate's start time is unknown, as on Windows, could pass for a reused process
* ID, so its caller refreshes it, and it frees a minute after the last refresh.
* A priority caller goes ahead of the queue, and a quick one starts at its
* deadline whatever its place (see `Precedence`).
* A priority caller goes ahead of the queue (see `Precedence`).
*
* Each caller decides against the newest ledger and publishes atomically, so
* concurrent callers never exceed the rule and one always wins while it has
* room. No admission timeout except a quick caller's. Resolves with the
* admission's id.
* room. There is no admission timeout. Resolves with the admission's id.
*/
export async function admit(
limits: Limits,
Expand All @@ -537,7 +525,6 @@ export async function admit(
const place = randomUUID();
// Every admission names its gate's start, so a reused process ID holds no slot.
const self = { pid: process.pid, start: processStart(process.pid) };
const deadline = precedence.quick ? host.now() + QUICK_MS : Infinity;

/** One decision against the newest ledger: true once admitted. */
const turn = async (): Promise<boolean> => {
Expand Down Expand Up @@ -583,8 +570,6 @@ export async function admit(
// An admission takes the place of this caller's entry in the queue.
const admitted = (at: number) =>
host.publish(ledger, [...kept.filter((admission) => admission.id !== place), { id, at, ...self }]);
// Recorded like any admission, so it still counts against the others' headroom.
if (now >= deadline) return admitted(now);
// Priority ignores load, whether or not a gated command runs; memory still applies.
const { used, load, memory } = standing(limits, host, [...running], precedence.priority);
let wait = SETTLE_MS;
Expand All @@ -607,7 +592,7 @@ export async function admit(
}
}
}
await host.sleep(Math.min(wait, deadline - now), signal);
await host.sleep(wait, signal);
return false;
};

Expand Down Expand Up @@ -898,8 +883,6 @@ const message = (error: unknown) => (error instanceof Error ? error.message : St

/** How `gate` admits its command. */
export interface GateOptions {
/** Wait at most QUICK_MS (see `Precedence`). */
quick?: boolean;
/** The admission ledger and host measures; this user's on this host when omitted. */
host?: Host;
}
Expand Down Expand Up @@ -1023,7 +1006,6 @@ async function gated(
};
const precedence = {
priority: process.env[PRIORITY] === "1",
quick: options.quick,
};
const host = options.host ?? fileHost(claimDirectory());
admitted = { host, id: await admit(hostLimits(), host, cancel.signal, notice, precedence) };
Expand Down
18 changes: 8 additions & 10 deletions src/cli/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ import {
} from "./agent";
import { held, openDrafts, reach, target, upstream, upstreamRefs } from "./claims";
import { PENDING_MS } from "./admission";
import { gate, QUICK_MS } from "./gate";
import { gate } from "./gate";
import { gitHubDependencies, GitHubInputError, pullRequest, readRemote, remoteRepository } from "./github";
import { mergedHolds } from "../github/client";
import { type Hold, finish, land, release, take } from "./hold";
Expand Down Expand Up @@ -171,8 +171,7 @@ export function main(argv: string[], out: Output): number | Promise<number> {
"quest review, long-running servers, and quick tools such as formatters out of the gate. Nested gate " +
"calls run immediately. Limits come from the host; there is " +
"no configuration.\n\n" +
`--quick waits at most ${QUICK_MS / 1000} seconds, then starts the command anyway; pass it only for ` +
"cheap commands such as formatters. With QUEST_GATE_PRIORITY=1 in the environment, as a check " +
"With QUEST_GATE_PRIORITY=1 in the environment, as a check " +
"that lands PRs should set, the command waits for memory but never for load, and gets " +
"MBX_SCHEDULER_PRIORITY=normal unless it sets its own.\n\n" +
"Once the command exits, or is cancelled and exits, its remaining descendants are killed; a second " +
Expand All @@ -190,11 +189,10 @@ export function main(argv: string[], out: Output): number | Promise<number> {
"its time queued and running under $XDG_STATE_HOME/quest/waits for the session's record; a log it cannot " +
"write fails the gate.",
)
.option("--quick", `Start a cheap command after at most ${QUICK_MS / 1000} seconds of waiting`)
.argument("<command>", "Executable to run")
.argument("[args...]", "Arguments forwarded verbatim")
.action((command: string, args: string[], options: { quick?: boolean }) => {
pending = gate([command, ...args], options).catch((error: unknown) => {
.action((command: string, args: string[]) => {
pending = gate([command, ...args]).catch((error: unknown) => {
warn(error instanceof Error ? error.message : String(error));
return 1;
});
Expand Down Expand Up @@ -2005,19 +2003,19 @@ function positiveSeconds(value: string): number {

/**
* Stop option parsing before a gated executable, keeping its own flags intact.
* Quest's global options still apply on either side of `gate`, and its own `--quick` after it.
* Quest's global options still apply on either side of `gate`.
*/
export function gateBoundary(argv: string[]): string[] {
const skip = (index: number, own: string[] = []) => {
const skip = (index: number) => {
for (;;) {
if (argv[index] === "--root") index += 2;
else if (argv[index]?.startsWith("--root=") || own.includes(argv[index])) index += 1;
else if (argv[index]?.startsWith("--root=")) index += 1;
else return index;
}
};
const index = skip(0);
if (argv[index] !== "gate") return argv;
const command = skip(index + 1, ["--quick"]);
const command = skip(index + 1);
if (!argv[command] || argv[command].startsWith("-")) return argv;
return [...argv.slice(0, command), "--", ...argv.slice(command)];
}
Expand Down
74 changes: 11 additions & 63 deletions test/gate.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ import {
type Owner,
owner,
processStart,
QUICK_MS,
RECENT_MS,
SETTLE_MS,
job,
Expand Down Expand Up @@ -331,32 +330,6 @@ test("a priority caller waits for memory, never for load, even while a gated com
expect(ordinary).toEqual([`sleep ${SETTLE_MS}`]);
});

test("a quick caller starts after at most QUICK_MS, and its admission is recorded", async () => {
const directory = tempDir();
const state = { now: 0, load: 99, memory: 0 };
const events: string[] = [];
const host = testHost(directory, state, events);
// A build admitted long ago still runs: the load is the gate's own, and its slot survives.
const build = { id: "build", at: -10 * RECENT_MS, ...me };
expect(host.publish(host.ledger(), [build])).toBe(true);
await admit({ load: 4, memory: GiB }, host, signal, undefined, { quick: true });
const sleeps = Array.from({ length: QUICK_MS / SETTLE_MS }, () => `sleep ${SETTLE_MS}`);
expect(events).toEqual(sleeps);
expect(host.ledger().admissions).toEqual([build, { id: expect.any(String), at: QUICK_MS, ...me }]);
// A sleep that overran shortens the last one, which ends at the deadline.
const late = { now: 0, load: 99, memory: 0 };
const overran: string[] = [];
const slow = testHost(tempDir(), late, overran);
const sleep = slow.sleep;
slow.sleep = async (ms, signal) => {
await sleep(ms, signal);
if (overran.length === 1) late.now += 3000;
};
await admit({ load: 4, memory: GiB }, slow, signal, undefined, { quick: true });
expect(overran.at(-1)).toBe(`sleep ${SETTLE_MS - 3000}`);
expect(slow.ledger().admissions).toEqual([{ id: expect.any(String), at: QUICK_MS, ...me }]);
});

test("waiting callers start in arrival order, a later one first only while the host has room for both", async () => {
const limits = { load: 4, memory: GiB };
// A build still runs, and an earlier caller waits for its turn.
Expand Down Expand Up @@ -429,7 +402,7 @@ test("a place frees once its gate ends or its caller cancels, or a minute after
expect(queued.ledger().admissions).toEqual([build, fresh]);
});

test("a priority caller goes ahead of the queue, and a quick one starts at its deadline whatever its place", async () => {
test("a priority caller goes ahead of the queue", async () => {
const limits = { load: 4, memory: GiB };
const build = { id: "build", at: -10 * RECENT_MS, ...me };
const earlier = { id: "earlier", at: 0, ...me, queued: true };
Expand All @@ -440,28 +413,19 @@ test("a priority caller goes ahead of the queue, and a quick one starts at its d
const events: string[] = [];
await admit(limits, testHost(directory, state, events), signal, undefined, { priority: true });
expect(events).toEqual([]);
await admit(limits, testHost(directory, state, events), signal, undefined, { quick: true });
expect(events).toEqual(Array.from({ length: QUICK_MS / SETTLE_MS }, () => `sleep ${SETTLE_MS}`));
expect(host.ledger().admissions).toEqual([
build,
earlier,
{ id: expect.any(String), at: 0, ...me },
{ id: expect.any(String), at: QUICK_MS, ...me },
]);
expect(host.ledger().admissions).toEqual([build, earlier, { id: expect.any(String), at: 0, ...me }]);
});

test("a lone quick or priority caller starts at once on load from outside the gate", async () => {
// No gated command runs, so the load is work the gate cannot wait out, whatever the precedence.
test("a lone priority caller starts at once on load from outside the gate", async () => {
// No gated command runs, so the load is work the gate cannot wait out.
const state = { now: 0, load: 12, memory: Infinity };
const limits = { load: 3, memory: GiB };
for (const precedence of [{ quick: true }, { priority: true }]) {
const host = { ...testHost(tempDir(), state), running: () => false };
host.sleep = async () => {
throw new Error("a lone command waited for load it cannot wait out");
};
await admit(limits, host, signal, undefined, precedence);
expect(host.ledger().admissions).toEqual([{ id: expect.any(String), at: 0, ...me }]);
}
const host = { ...testHost(tempDir(), state), running: () => false };
host.sleep = async () => {
throw new Error("a lone command waited for load it cannot wait out");
};
await admit(limits, host, signal, undefined, { priority: true });
expect(host.ledger().admissions).toEqual([{ id: expect.any(String), at: 0, ...me }]);
});

test("a burst that all read the ledger before any records admits only what the rule allows", async () => {
Expand Down Expand Up @@ -1493,24 +1457,8 @@ test("the gated command keeps its own flags, even ones Quest also parses", () =>
expect(gateBoundary(["gate", "--root", "r", "ls"])).toEqual(["gate", "--root", "r", "--", "ls"]);
expect(gateBoundary(["gate", "--root=r", "ls"])).toEqual(["gate", "--root=r", "--", "ls"]);
expect(gateBoundary(["gate", "--version"])).toEqual(["gate", "--version"]);
// Gate's own option stays Quest's; the same flag after the executable is the command's.
expect(gateBoundary(["gate", "--quick", "ls", "--quick"])).toEqual(["gate", "--quick", "--", "ls", "--quick"]);
// A flag after the executable is the command's.
expect(gateBoundary(["gate", "ls", "--host"])).toEqual(["gate", "--", "ls", "--host"]);
expect(gateBoundary(["gate", "--root", "r", "--quick", "ls"])).toEqual([
"gate",
"--root",
"r",
"--quick",
"--",
"ls",
]);
expect(gateBoundary(["--quick", "gate", "ls"])).toEqual(["--quick", "gate", "ls"]);
});

test("gate parses --quick before its command", async () => {
vi.stubEnv("QUEST_GATE_ADMITTED", "1");
const out = { stdout: () => {}, stderr: () => {} };
expect(await main(["gate", "--quick", process.execPath, "-e", "process.exit(4)"], out)).toBe(4);
});

test("global options after gate keep working", async () => {
Expand Down
Loading