Skip to content

feat(gate): hold each gated command to a memory ceiling - #693

Closed
kixelated wants to merge 6 commits into
mainfrom
quest/m0/gate-memory
Closed

kixelated wants to merge 6 commits into
mainfrom
quest/m0/gate-memory

Conversation

@kixelated

@kixelated kixelated commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m0/gate-memory.md and deletes it.

Waiting

  • Maintainer: decide how hard the memory partition must be; review round 3 keeps one high finding open on it, at its checkpoint. A kernel limit small enough to partition the host is also the limit Node and the JVM size their heaps from when they start (measured in a 512 MiB scope: Node 22 took a 259 MiB heap limit instead of 4144 MiB, the JVM a 128 MiB MaxHeapSize instead of 15 GiB). So a per-command limit cannot both partition the host at every moment and leave those runtimes alone. Options:

    1. (recommended) Land this as is. Each command has a share in the ledger and the shares fit the host; its scope's kernel limit stays large until the host has no room, then drops to the share. Gap: commands that together take more than the 4 GiB kept free within 250 ms, between two samples, can still exhaust the host.
    2. Also bound all gated commands together with a limit on a slice the gate owns, resized as other memory use moves. It closes the gap for sessions and services; its kill picks the largest gated command, not the one that overran. More work, best as its own quest after this lands. The MemoryMax=32G now set by hand on background.slice is a fixed version of this, and this PR already counts it.
    3. Hard per-command limits from the start, growing from 512 MiB (this PR's first design). No gap, but Node and the JVM take small default heaps and can run out of heap on a host with memory to spare, unless every command sets its own heap size.
    4. Hard limits starting at one command's budget, 4 GiB. No gap; default heaps shrink less (JVM 1 GiB, Node 2 GiB); only as many commands run as the host has spare 4 GiB blocks, about two at a time on the host this was measured on.

    Done when the answer is recorded under ## Decisions here and the finding is settled in a new review round.

Summary

Gate scopes started with no memory limit, so a gated command that overran took the host with it: the kernel's OOM killer, or earlyoom, picked victims elsewhere. Where a systemd scope owns the command and the user manager may limit memory, each gated command now has a share of the host's memory, and its scope is held to it once the host runs out.

  • Shares fit the host together (src/cli/gate.ts): each admission records its share (memory) and what its command holds (held) in the shared ledger. A share starts at an eighth of one command's budget (512 MiB on most hosts). Four times a second its gate (ration) records what the command holds, without file cache, and:
    • raises the share by half (at least one starting share) once the command holds three quarters of it. The raise comes out of the host's available memory, less what other running commands may still take of their shares, and less one command's budget kept free for sessions and services;
    • lowers it to one starting share above the command once the command holds less than half.
  • Admission: what running commands may still take of their shares counts as taken, however long they have run, so a new command starts only when its share fits.
  • Scope limits (src/cli/supervise.ts): a gate scope starts with MemoryHigh and MemoryMax at the host's memory less one command's budget, MemorySwapMax=0, and OOMPolicy=kill. Once no raise fits, the gate lowers both to the share and says so on stderr: the kernel slows the command over the last eighth below it and kills the whole scope if it fills the rest. The run then rejects with Exhausted ("... ran out of memory"), read from systemd's Result=oom-kill, and the gate exits 1. Such a scope runs without --collect, so the failed unit survives until the run has asked, then is reset. The limits lift once the command wants no more or a raise fits.
  • The slice's limits count: available memory is now also bounded by any memory limit on background.slice, where gate scopes run, and on the cgroups holding it. The gate itself runs outside that slice, so before this a raise could pass what the slice holds, and the slice's kill would pick the victim. With the MemoryMax=32G set by hand on this host's slice, scopes start at 28 GiB.
  • Nested gates: an admitted command's nested gates stay in its cgroup, under its limits. A gate inside another Quest scope gets a sibling scope with its own admission and share, taken from the same host memory, rather than drawing on an enclosing command's share: every scope the gate starts has an admission of its own, so none is left unlimited. linuxMemory no longer counts an enclosing gate scope's own limit against such a command.
  • README and quest gate --help describe the shares.

Choices

Made without asking, since the quest left them to the implementer:

  • A share that follows the command, not one learned from earlier runs. The quest suggested a per-command peak history; moq-dev/quest#673 dropped the matching CPU history after measuring it, and a live share needs no key for "the same command" and no state between runs.
  • The reserve kept free for everything outside the gate is one command's budget (4 GiB, or a quarter of RAM on small hosts), the same amount admission already requires.
  • MemorySwapMax=0: with swap, a command held to its share would thrash instead of stopping.
  • A --quick command past its deadline still starts whatever the memory, as before.
  • Hosts where the memory controller is not delegated to the user manager, and hosts without a systemd scope, get no shares, as before.

The design changed after review round 1; see the Design change: comment.

Validation

On a 32-core, 60 GiB Fedora host (systemd 262) under load from other sessions, on the head:

  • just check and just test pass (1305 passed, 5 skipped), run inside a gate scope with these limits.
  • Built binary: quest gate sh -c '...; exit 7' printed max=28672 MiB high=28672 MiB swap=0 group=1, Node reported a 4144 MiB heap limit and the JVM a 7 GiB MaxHeapSize (both as outside the gate, or a quarter of the scope's limit), and the gate exited 7.
  • Built binary, fast growth: four processes each taking 400 MiB at once all exited 0 in 1.6 s, 1.5 s of it their own wait, with no high event. Under the first design, which started the limit at 512 MiB, the same run took 3.6 s with 3088 high events.
  • Kill: a real scope with a 256 MiB limit running a wrapper, a sleeping child, and a process taking 1 GiB rejects with Exhausted; the wrapper and its child die with it, and a neighbour scope holding 64 MiB keeps running (test/gate.test.ts). Repeated six times under load.
  • A real scope's limits read back as set, move with limit, and a limited scope keeps a nonzero exit status (test/gate.test.ts).
  • Not run: the gate CLI lowering a limit on a host out of room, which needs several GiB taken from a host already short of memory. ration's tests cover it on a mocked host, and the scope tests above cover the limits and the kill on the real kernel.

Public API

No library API. CLI behavior on Linux with a systemd scope and a delegated memory controller: a gated command may be slowed, or killed with exit 1 and ran out of memory, once the host cannot give it more memory, and it runs without swap. The admission ledger's entries gain optional memory and held; older builds sharing the ledger keep them.

Caused by

No single PR: gate scopes have had no memory limit since they were introduced.

Follow-ups

None added. Option 2 above would be a new quest if picked. moq-dev/quest#685 restructures the scope's cgroup in the same files; whichever lands second merges the other.

Outside this PR's reach: a command run directly in an agent's scope, not through the gate, has no share, and a host whose dev shell pins an older quest gets none until its pin moves.

Models

Size: S
Implementer: Claude Code, claude-fable-5-2, unknown effort

Review

Rewritten by quest review after each round.

  • Head: 9a0b59c, named in full once no reviewer reports a finding

  • Base: origin/main at 58bfa33

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1d553-5688-7b43-8f42-cb02b6becc5e, round 3

  • Score: 5/10, The shorter sampling interval narrows the risk, but concurrent commands can still exhaust host memory and kill unrelated processes, leaving the aggregate protection issue unresolved.

  • Rounds: 3; findings 3 (high 3): fixed 1, refuted 1, open 1

  • Findings, open until the next round settles them:

    • High, src/cli/gate.ts:988: Preserve aggregate memory protection between samples
  • The last round's findings, at d6d1e83:

    • Open: high, src/cli/gate.ts:984: Preserve aggregate memory protection between samples; CEILING_MS now equals 250 and the README explicitly acknowledges that concurrent commands can still exhaust the host between samples; aggregate limits remain unenforced, and the response records no maintainer decision accepting that tradeoff.

The maintainer waived opencode for this PR while it is out of quota ("Codex alone"), so Codex reviewed alone with --harness codex.

kixelated and others added 4 commits November 12, 2026 22:25
Quest-Claim: 768cd5ac4daed53a
…es while the host has room

Checkpoint: scope ceilings, ledger accounting, and tests; docs and the quest file follow.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… an out-of-memory kill from systemd

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…/gate-memory.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kixelated kixelated changed the title A gated command cannot exhaust the host's memory feat(gate): hold each gated command to a memory ceiling Nov 13, 2026
…s no room

A limit that starts small keeps Node and the JVM small, since they size their heap from it at startup. Count the limits of the slice the scopes run in as well.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Design change: a gate scope's memory limit no longer starts small and grows.

Review round 1 found that a runtime sizing its heap from its cgroup's limit at startup would stay small under a limit that only grows later. Measured in a 512 MiB scope: Node 22 took a 259 MiB heap limit (4144 MiB outside) and the JVM a 128 MiB MaxHeapSize (15 GiB outside), so either could run out of heap before reaching the threshold that raised the limit.

Now each running command has a share in the ledger, starting at 512 MiB and moving with what it holds, and the shares still fit the host together. The scope's kernel limit starts at the host's memory less one command's budget and stays there while raises fit. Only once no raise fits does the gate lower the limit to the share, where the kernel slows and then kills the command.

The cost: between two one-second samples a command can pass its share into the budget kept free for sessions and services, so the kernel limits fit the host together only once the host is out of room, not at every moment.

Rejected: starting every limit at one command's budget (4 GiB), which holds the shares exactly but lets only as many commands run as the host has spare 4 GiB blocks, and still shrinks default heaps (JVM 1 GiB, Node 2 GiB); keeping the 512 MiB start.

(Written by Fable 5.2)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Closing after the 2026-11-13 complexity audit: host systemd-oomd's swap kill already kills a runaway scope as a whole, which was this quest's goal, and a per-scope ceiling loop four times a second adds about 630 lines while shrinking Node's and the JVM's default heaps. If the gate needs anything, it is OOMPolicy=kill and reporting an oom-kill result.

(Written by Opus 5.5)

@kixelated kixelated closed this Nov 13, 2026
kixelated added a commit that referenced this pull request Nov 13, 2026
Maintainer decision (2026-11-13): keep the host half, which reuses the
gate's room and pending-count logic, and drop the Codex account-usage
half. It parsed Codex session logs and could not see the Claude quota
behind the "credentials cooling down" errors. `quest agent admit` loses
--harness and exit 3, start and resume no longer print `outcome limited`
on admission, and the agents' ledger no longer tags entries by account.
Deletes the follow-up quests agent-usage.md and agent-admission-measure.md.

Also deletes quest/m0/gate-memory.md, whose PR #693 the maintainer
closed: the host's systemd-oomd swap kill owns memory, and the gate keeps
at most OOMPolicy=kill plus reporting an oom-kill result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant