Repository navigation
Conversation
Quest-Claim: 768cd5ac4daed53a
…es while the host has room Checkpoint: scope ceilings, ledger accounting, and tests; docs and the quest file follow. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… an out-of-memory kill from systemd Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…/gate-memory.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s no room A limit that starts small keeps Node and the JVM small, since they size their heap from it at startup. Count the limits of the slice the scopes run in as well. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Design change: a gate scope's memory limit no longer starts small and grows. Review round 1 found that a runtime sizing its heap from its cgroup's limit at startup would stay small under a limit that only grows later. Measured in a 512 MiB scope: Node 22 took a 259 MiB heap limit (4144 MiB outside) and the JVM a 128 MiB Now each running command has a share in the ledger, starting at 512 MiB and moving with what it holds, and the shares still fit the host together. The scope's kernel limit starts at the host's memory less one command's budget and stays there while raises fit. Only once no raise fits does the gate lower the limit to the share, where the kernel slows and then kills the command. The cost: between two one-second samples a command can pass its share into the budget kept free for sessions and services, so the kernel limits fit the host together only once the host is out of room, not at every moment. Rejected: starting every limit at one command's budget (4 GiB), which holds the shares exactly but lets only as many commands run as the host has spare 4 GiB blocks, and still shrinks default heaps (JVM 1 GiB, Node 2 GiB); keeping the 512 MiB start. (Written by Fable 5.2) |
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Closing after the 2026-11-13 complexity audit: host systemd-oomd's swap kill already kills a runaway scope as a whole, which was this quest's goal, and a per-scope ceiling loop four times a second adds about 630 lines while shrinking Node's and the JVM's default heaps. If the gate needs anything, it is OOMPolicy=kill and reporting an oom-kill result. (Written by Opus 5.5) |
Maintainer decision (2026-11-13): keep the host half, which reuses the gate's room and pending-count logic, and drop the Codex account-usage half. It parsed Codex session logs and could not see the Claude quota behind the "credentials cooling down" errors. `quest agent admit` loses --harness and exit 3, start and resume no longer print `outcome limited` on admission, and the agents' ledger no longer tags entries by account. Deletes the follow-up quests agent-usage.md and agent-admission-measure.md. Also deletes quest/m0/gate-memory.md, whose PR #693 the maintainer closed: the host's systemd-oomd swap kill owns memory, and the gate keeps at most OOMPolicy=kill plus reporting an oom-kill result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Finishes quest/m0/gate-memory.md and deletes it.
Waiting
Maintainer: decide how hard the memory partition must be; review round 3 keeps one high finding open on it, at its checkpoint. A kernel limit small enough to partition the host is also the limit Node and the JVM size their heaps from when they start (measured in a 512 MiB scope: Node 22 took a 259 MiB heap limit instead of 4144 MiB, the JVM a 128 MiB
MaxHeapSizeinstead of 15 GiB). So a per-command limit cannot both partition the host at every moment and leave those runtimes alone. Options:MemoryMax=32Gnow set by hand onbackground.sliceis a fixed version of this, and this PR already counts it.Done when the answer is recorded under
## Decisionshere and the finding is settled in a new review round.Summary
Gate scopes started with no memory limit, so a gated command that overran took the host with it: the kernel's OOM killer, or earlyoom, picked victims elsewhere. Where a systemd scope owns the command and the user manager may limit memory, each gated command now has a share of the host's memory, and its scope is held to it once the host runs out.
src/cli/gate.ts): each admission records its share (memory) and what its command holds (held) in the shared ledger. A share starts at an eighth of one command's budget (512 MiB on most hosts). Four times a second its gate (ration) records what the command holds, without file cache, and:src/cli/supervise.ts): a gate scope starts withMemoryHighandMemoryMaxat the host's memory less one command's budget,MemorySwapMax=0, andOOMPolicy=kill. Once no raise fits, the gate lowers both to the share and says so on stderr: the kernel slows the command over the last eighth below it and kills the whole scope if it fills the rest. The run then rejects withExhausted("... ran out of memory"), read from systemd'sResult=oom-kill, and the gate exits 1. Such a scope runs without--collect, so the failed unit survives until the run has asked, then is reset. The limits lift once the command wants no more or a raise fits.background.slice, where gate scopes run, and on the cgroups holding it. The gate itself runs outside that slice, so before this a raise could pass what the slice holds, and the slice's kill would pick the victim. With theMemoryMax=32Gset by hand on this host's slice, scopes start at 28 GiB.linuxMemoryno longer counts an enclosing gate scope's own limit against such a command.quest gate --helpdescribe the shares.Choices
Made without asking, since the quest left them to the implementer:
MemorySwapMax=0: with swap, a command held to its share would thrash instead of stopping.--quickcommand past its deadline still starts whatever the memory, as before.The design changed after review round 1; see the
Design change:comment.Validation
On a 32-core, 60 GiB Fedora host (systemd 262) under load from other sessions, on the head:
just checkandjust testpass (1305 passed, 5 skipped), run inside a gate scope with these limits.quest gate sh -c '...; exit 7'printedmax=28672 MiB high=28672 MiB swap=0 group=1, Node reported a 4144 MiB heap limit and the JVM a 7 GiBMaxHeapSize(both as outside the gate, or a quarter of the scope's limit), and the gate exited 7.highevent. Under the first design, which started the limit at 512 MiB, the same run took 3.6 s with 3088highevents.Exhausted; the wrapper and its child die with it, and a neighbour scope holding 64 MiB keeps running (test/gate.test.ts). Repeated six times under load.limit, and a limited scope keeps a nonzero exit status (test/gate.test.ts).ration's tests cover it on a mocked host, and the scope tests above cover the limits and the kill on the real kernel.Public API
No library API. CLI behavior on Linux with a systemd scope and a delegated memory controller: a gated command may be slowed, or killed with exit 1 and
ran out of memory, once the host cannot give it more memory, and it runs without swap. The admission ledger's entries gain optionalmemoryandheld; older builds sharing the ledger keep them.Caused by
No single PR: gate scopes have had no memory limit since they were introduced.
Follow-ups
None added. Option 2 above would be a new quest if picked. moq-dev/quest#685 restructures the scope's cgroup in the same files; whichever lands second merges the other.
Outside this PR's reach: a command run directly in an agent's scope, not through the gate, has no share, and a host whose dev shell pins an older quest gets none until its pin moves.
Models
Size: S
Implementer: Claude Code, claude-fable-5-2, unknown effort
Review
Rewritten by
quest reviewafter each round.Head: 9a0b59c, named in full once no reviewer reports a finding
Base: origin/main at 58bfa33
Reviewer: Codex, gpt-6.1-sol, medium effort
Session:
01a1d553-5688-7b43-8f42-cb02b6becc5e, round 3Score: 5/10,
The shorter sampling interval narrows the risk, but concurrent commands can still exhaust host memory and kill unrelated processes, leaving the aggregate protection issue unresolved.Rounds: 3; findings 3 (high 3): fixed 1, refuted 1, open 1
Findings, open until the next round settles them:
src/cli/gate.ts:988:Preserve aggregate memory protection between samplesThe last round's findings, at d6d1e83:
src/cli/gate.ts:984:Preserve aggregate memory protection between samples;CEILING_MS now equals 250 and the README explicitly acknowledges that concurrent commands can still exhaust the host between samples; aggregate limits remain unenforced, and the response records no maintainer decision accepting that tradeoff.The maintainer waived opencode for this PR while it is out of quota ("Codex alone"), so Codex reviewed alone with
--harness codex.