Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1210,11 +1210,16 @@ they kept every CPU busy. macOS, Windows, and Linux without a systemd user
manager have no scope to measure, so the load average alone decides there.

A user's admissions on one host are recorded under
`~/.cache/quest/admissions/<hostname>`, whatever the caller's environment or
login session. The home directory comes from the user database, not `$HOME`;
an account missing from it, as in arbitrary-UID containers, uses
`/tmp/quest-<uid>/admissions/<hostname>`, and refuses that directory unless only
its UID can write it.
`~/.cache/quest/admissions-v2/<hostname>`, whatever the caller's environment or
login session. A gate judges whether another still runs by its process ID, so
on Linux each PID namespace, such as a sandbox's, keeps a ledger of its own,
named `<hostname>-pid<inode>`, and its gates do not count the host's. A gate
that finds its running admission dropped records it again and warns. The home
directory comes from the user database, not `$HOME`; an account missing from
it, as in arbitrary-UID containers, uses
`/tmp/quest-<uid>/admissions-v2/<hostname>`, and refuses that directory unless
only its UID can write it. `quest agent` keeps its admissions beside them, in
`agents-v2/<hostname>`.

Two kinds of command go ahead of the rest. `quest gate --quick <command>` waits
at most a minute, then starts the command whatever the host's load, memory, and
Expand Down
36 changes: 0 additions & 36 deletions quest/m0/gate-ledger-compat.md

This file was deleted.

3 changes: 2 additions & 1 deletion src/cli/admission.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import {
type Admission,
type Host,
type Limits,
LEDGER_VERSION,
SETTLE_MS,
type Waiting,
claimDirectory,
Expand Down Expand Up @@ -82,7 +83,7 @@ export async function admit(deps: AdmissionDependencies, signal: AbortSignal, he
}

/** The agents' ledger of this host, beside the gate's. */
const agentsDirectory = (gate: string) => join(dirname(dirname(gate)), "agents", basename(gate));
const agentsDirectory = (gate: string) => join(dirname(dirname(gate)), `agents-v${LEDGER_VERSION}`, basename(gate));

/** The admission `quest agent` runs with: the host's own limits and ledgers. */
export function processAdmission(notice: (text: string) => void): AdmissionDependencies {
Expand Down
68 changes: 54 additions & 14 deletions src/cli/gate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
lstatSync,
mkdirSync,
openSync,
readlinkSync,
readdirSync,
readFileSync,
rmSync,
Expand Down Expand Up @@ -227,19 +228,42 @@ export function accountHome(): string | null {
}

/**
* Every caller of one user on one host shares this directory, whatever its
* environment or login sessions. It is named for the host, since a home
* directory may be shared between machines. An account missing from the user
* database, as in arbitrary-UID containers, has no home of its own, so it
* shares one in `tmp` keyed by its UID.
* The Linux PID namespace this process runs in, such as a sandbox's own, or undefined elsewhere.
* A gate only sees the process IDs of its namespace, so it cannot tell whether a gate in another
* one still runs, and would drop that gate's admission a minute after it was admitted.
*/
export function pidNamespace(): string | undefined {
if (process.platform !== "linux") return undefined;
try {
return /^pid:\[(\d+)\]$/.exec(readlinkSync("/proc/self/ns/pid"))?.[1];
} catch {
return undefined;
}
}

/**
* The format version of the gate and agents ledgers, part of their paths, so builds whose ledgers disagree never share one.
* Bump it with any change to what an admission records or how a gate keeps one.
*/
export const LEDGER_VERSION = 2;
Comment thread
kixelated marked this conversation as resolved.
Resolved

/**
* Every caller of one user on one host and PID namespace shares this directory, whatever its
* environment or login sessions. It is named for the host, since a home directory may be shared
* between machines, and for the namespace, since a gate judges another's admission by its
* process ID (see `pidNamespace`). An account missing from the user database, as in arbitrary-UID
* containers, has no home of its own, so it shares one in `tmp` keyed by its UID.
*/
export function claimDirectory(
home = accountHome(),
host = hostname(),
uid = process.getuid?.(),
tmp = "/tmp",
namespace: string | null = pidNamespace() ?? null,
): string {
if (home !== null) return join(home, ".cache", "quest", "admissions", host);
const name = namespace === null ? host : `${host}-pid${namespace}`;
const ledger = `admissions-v${LEDGER_VERSION}`;
if (home !== null) return join(home, ".cache", "quest", ledger, name);
// Other users can create this name first, so it must be a directory only this UID can write.
const root = join(tmp, `quest-${uid}`);
try {
Expand All @@ -250,7 +274,7 @@ export function claimDirectory(
const stat = lstatSync(root);
if (!stat.isDirectory() || stat.uid !== uid || (stat.mode & 0o022) !== 0)
throw new Error(`gate cannot trust ${root}: it must be a directory that only UID ${uid} can write`);
return join(root, "admissions", host);
return join(root, ledger, name);
}

/** A version cut short, as by power loss, is read as empty: its commands ended with the host. */
Expand Down Expand Up @@ -611,11 +635,17 @@ export async function admit(
* SETTLE_MS, and keep its admission's `cores` at the most CPUs it kept busy over one sample in the
* last RECENT_MS, rounded, so other callers count it at its busiest rather than at a lull.
*/
export async function measure(host: Host, admission: { id: string; unit: string }, signal: AbortSignal) {
export async function measure(
host: Host,
admission: { id: string; unit: string },
signal: AbortSignal,
lost?: (id: string) => void,
) {
const samples: { at: number; cores: number }[] = [];
// The command's scope starts with no CPU use, so a burst before the first sample still counts.
let previous = { at: host.now(), usage: 0 };
let recorded = 0;
// The admission as last read, to record again should another gate drop it.
let known: Admission = { id: admission.id, at: host.now(), pid: process.pid, start: processStart(process.pid) };
for (;;) {
try {
await host.sleep(SETTLE_MS, signal);
Expand All @@ -632,11 +662,19 @@ export async function measure(host: Host, admission: { id: string; unit: string
}
while (samples.length > 0 && samples[0].at <= at - RECENT_MS) samples.shift();
const cores = Math.round(Math.max(0, ...samples.map((sample) => sample.cores)));
for (let ledger = host.ledger(); cores !== recorded; ledger = host.ledger()) {
// Checked every sample, so an admission another gate dropped counts again within one.
for (let ledger = host.ledger(); ; ledger = host.ledger()) {
const mine = ledger.admissions.find(({ id }) => id === admission.id);
if (mine === undefined) throw new Error(`gate lost its admission ${admission.id} from the ledger`);
const updated = ledger.admissions.map((entry) => (entry === mine ? { ...mine, cores } : entry));
if (host.publish(ledger, updated)) recorded = cores;
if (mine !== undefined) known = mine;
if (mine !== undefined && (mine.cores ?? 0) === cores) break;
const updated =
mine === undefined
? [...ledger.admissions, { ...known, cores }]
: ledger.admissions.map((other) => (other === mine ? { ...mine, cores } : other));
if (host.publish(ledger, updated)) {
if (mine === undefined) lost?.(admission.id);
break;
}
}
}
}
Expand Down Expand Up @@ -972,7 +1010,9 @@ async function gated(
// Held as a value until the command ends, so a failure never goes unhandled meanwhile.
const measuring =
admitted !== undefined && unit !== undefined
? measure(admitted.host, { id: admitted.id, unit }, stop.signal).then(
? measure(admitted.host, { id: admitted.id, unit }, stop.signal, (id) =>
process.stderr.write(`quest gate: recorded admission ${id} again after another gate dropped it\n`),
).then(
() => undefined,
(error: unknown) => ({ error }),
)
Expand Down
35 changes: 32 additions & 3 deletions test/gate.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import {
type Admission,
type Host,
hostLimits,
LEDGER_VERSION,
linuxMemory,
measure,
type Owner,
Expand Down Expand Up @@ -281,6 +282,31 @@ test("a gate records its command's burst before the first sample", async () => {
expect(published).toEqual([`4 at ${SETTLE_MS}`, `0 at ${SETTLE_MS + RECENT_MS}`]);
});

test("a gate records its admission again once another gate drops it, even while its cores hold", async () => {
const state = { now: 0, load: 0, memory: Infinity };
const host = testHost(tempDir(), state);
host.publish(host.ledger(), [{ id: "build", at: 0, ...me }]);
host.usage = () => 0;
const stop = new AbortController();
let samples = 3;
host.sleep = async () => {
if (samples-- === 0) {
stop.abort();
throw stop.signal.reason;
}
state.now += SETTLE_MS;
// Before the second sample, a gate in another PID namespace drops it as gone.
if (samples === 1) host.publish(host.ledger(), [{ id: "other", at: state.now, ...me }]);
};
const lost: string[] = [];
await measure(host, { id: "build", unit: "quest-gate-build.scope" }, stop.signal, (id) => lost.push(id));
expect(lost).toEqual(["build"]);
expect(host.ledger().admissions).toEqual([
{ id: "other", at: 2 * SETTLE_MS, ...me },
{ id: "build", at: 0, ...me, cores: 0 },
]);
});

test("a priority caller waits for memory, never for load, even while a gated command runs", async () => {
const directory = tempDir();
const state = { now: 0, load: 99, memory: 0 };
Expand Down Expand Up @@ -680,8 +706,11 @@ test("Linux memory reads the container cgroup mounted as the root", () => {

test("every caller on this host shares one ledger, whatever its environment or sessions", () => {
// A home directory shared between machines keeps each host's admissions apart.
expect(claimDirectory("/home/u", "a")).toBe(join("/home/u", ".cache", "quest", "admissions", "a"));
const ledger = `admissions-v${LEDGER_VERSION}`;
expect(claimDirectory("/home/u", "a", 1, "/tmp", null)).toBe(join("/home/u", ".cache", "quest", ledger, "a"));
expect(claimDirectory("/home/u", "b")).not.toBe(claimDirectory("/home/u", "a"));
// So does a PID namespace, such as a sandbox's, whose gates cannot see the others' processes.
expect(claimDirectory("/home/u", "a", 1, "/tmp", "7")).toBe(join("/home/u", ".cache", "quest", ledger, "a-pid7"));
// A login session's runtime directory comes and goes; an overridden HOME is still this
// account. Runtimes may read both once at startup, so each caller is a new process.
const gate = fileURLToPath(new URL("../src/cli/gate.ts", import.meta.url));
Expand All @@ -702,9 +731,9 @@ test.skipIf(process.platform === "win32")("an account without a home shares a pr
const uid = process.getuid!();
const tmp = tempDir();
const root = join(tmp, `quest-${uid}`);
expect(claimDirectory(null, "a", uid, tmp)).toBe(join(root, "admissions", "a"));
expect(claimDirectory(null, "a", uid, tmp, null)).toBe(join(root, `admissions-v${LEDGER_VERSION}`, "a"));
expect(statSync(root).mode & 0o777).toBe(0o700);
expect(claimDirectory(null, "a", uid, tmp)).toBe(join(root, "admissions", "a"));
expect(claimDirectory(null, "a", uid, tmp, null)).toBe(join(root, `admissions-v${LEDGER_VERSION}`, "a"));
// Another user may create the name first, writable by them or owned by them.
chmodSync(root, 0o777);
expect(() => claimDirectory(null, "a", uid, tmp)).toThrow("only UID");
Expand Down
Loading