From 4425779fb501b5044d0a6ee8ae9ce3bb0c57be19 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 1 Nov 2026 03:57:52 -0800 Subject: [PATCH 1/5] chore(quest): claim quest/m0/ci/queue-host-disk.md Quest-Claim: 0c760c1a7feda0f8 From 265cda2fb557098bfd210ed1dbcf8f435d4c07cd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 1 Nov 2026 04:02:56 -0800 Subject: [PATCH 2/5] docs(quest): checkpoint queue host disk ownership and isolation Co-Authored-By: GPT-6.1 Sol --- quest/m0/ci/queue-host-disk.md | 137 ++++++++++++++++++++++++++++----- 1 file changed, 119 insertions(+), 18 deletions(-) diff --git a/quest/m0/ci/queue-host-disk.md b/quest/m0/ci/queue-host-disk.md index 99436bf2f05..9dc93158dcd 100644 --- a/quest/m0/ci/queue-host-disk.md +++ b/quest/m0/ci/queue-host-disk.md @@ -8,24 +8,125 @@ crash comes back on its own. ## Plan -On 2026-10-30 the host filled twice (about 03:00 and 07:30 PDT). Each time -non-CI usage grew by roughly 480 GB and was later freed. CI's XFS quota is -whatever the host leaves free past a 32 GiB reserve, so the runner hit ENOSPC -mid-check: quest land ejected moq-dev/moq#6985 and moq-dev/moq#6999 as -`failed_checks`, and at 07:49 the runner aborted and stayed down for 36 -minutes. - -Done so far, in the host's configuration: -an `agent-storage-gc` user timer reclaims idle worktree build outputs and -leaked test scratch every 10 minutes and logs free space, and the runner unit -restarts after a crash (that change is in the PR but not yet deployed). - -Remaining: - -- Find what consumed the 480 GB, using the timer's free-space log the next - time it happens, and fix that source. -- Guarantee CI a floor: put agent storage (`/home`, `/tmp`) in its own XFS - project with a hard cap, so an agent gets ENOSPC instead of the queue. +### Incident and attribution + +On 2026-10-30 the host filled twice (about 03:00 and 07:30 PDT). The original +investigation recorded roughly 480 GB of non-CI growth, later freed. Queue +checks ejected [moq-dev/moq#6985](https://github.com/moq-dev/moq/pull/6985) +and [moq-dev/moq#6999](https://github.com/moq-dev/moq/pull/6999), and at 07:49 +the runner aborted and stayed down for 36 minutes. + +Read-only investigation on 2026-11-01 confirmed the capacity failure from +`journalctl -u ci-storage-refresh.service --since 2026-10-30`: +at 03:00:50 PDT, CI used 249371398144 bytes against a proposed +249002205184-byte limit. At 08:05:10 it used 145833361408 bytes against a +proposed 119020105728-byte limit. These are failed refresh calculations, +not evidence that those limits were successfully installed. + +The live `ci-storage.py` computes the allowance as current project usage +plus physical free space minus 32 GiB. It has no reserved CI floor; a +periodic refresh cannot prevent other writers consuming space between +refreshes. A fixed CI quota alone would still not reserve physical blocks. + +The consumer of the original 480 GB remains unknown. The first retained +`agent-storage-gc` journal entry is 2026-10-30 08:21:49 PDT, after both +incidents. That run recovered 8 GiB. Its logs contain global free space and +selected removals, not per-writer allocation history. Neither those logs nor +today's directory sizes establish which process allocated and freed the +incident data. Do not attribute the bursts to worktrees, MBX, or test scratch +without allocation evidence. + +### Configuration ownership and live state + +The home host is `nixos`, configured in `kixelated/nixos`, not this +repository's `infra/`. Existing work is owned by the maintainer's host PRs: + +- [kixelated/nixos#15](https://github.com/kixelated/nixos/pull/15), open at + `f4e822e4ce7332be253bfc6ce9a0b8e5c60ba4e0`, owns idle-output GC and runner + restart recovery. GC is deployed and its timer enabled. The real container + unit, `github-runner-runner.service`, was active with `Restart=no` on + 2026-11-01; the restart commit is still not active there. Do not recreate + either fix in MoQ. +- [kixelated/nixos#16](https://github.com/kixelated/nixos/pull/16), open at + `eb007f8bc296b9782f0a479d53c5bd7d10565a20`, owns the newer host landing + daemons and ephemeral check/fixer containers. The live + `quest-land-moq.service` is active with `Restart=on-failure`. Storage work + must cover this gate as well as the retained Actions runner. A deployment + branch combines host changes; `main` alone does not describe the live host. + +Read-only `findmnt`, `xfs_quota ... state/report`, and `xfs_io -r -c stat` +on 2026-11-01 found: + +- `/home`, `/tmp`, and `/var/tmp` share the root XFS filesystem with project + enforcement on. Their directory project IDs are 0 without project + inheritance. Only projects 0 and 42000 appear in the quota report. +- CI project 42000 has `limitGiB=null`, a 32 GiB reserve, and eight million + inodes. Its configured trees include `/nix`, native container state, build + scratch, and the newer gate's caches and staging trees. The sampled + `/var/cache/quest-land-moq-check` has project 42000 and inheritance enabled. +- Agent storage extends beyond worktree outputs: directory scans found + approximately 41 GiB in the user's MBX cache, 39 GiB in a media build + cache, and 64 GiB in rootless container storage. `/var/tmp` included an + 8.6 GiB user-owned benchmark directory. These are current allocated-file + counts, not incident attribution or unique physical usage with reflinks. +- `/var/lib/quest-land` has project ID 0. The gate and its launcher use + `PrivateTmp=true`; the host has their private directories under both + `/tmp` and `/var/tmp`. Capping either shared temporary root indiscriminately + can therefore make the gate hit the agent quota too. + +### Proposed storage separation, pending maintainer decisions + +Keep implementation in the host repository, coordinated with the owners of +the two PRs above. Reuse its XFS policy and VM tests. Recommended direction: + +- Give agent-writable trees a separate enforced project with block and inode + budgets, covering home caches, rootless container storage, and agent + temporary/benchmark outputs. Put CI private temporary files and daemon + state in CI-owned trees before tagging the shared temporary roots. Check + existing files and inheritance, including rename/hard-link boundaries; + tagging only a directory does not account for its existing descendants. +- Establish an explicit CI allowance and a bounded budget for every other + writer on the filesystem, including host services and metadata headroom. + Their worst-case allocations must fit together with the existing host + reserve. Never derive the promised floor from a free-space sample or + silently shrink it when agents grow. Refuse an infeasible plan before + enabling writers; preserve existing data during any migration. +- Resolve the shared Nix writer first. Agent `nix develop` and CI use the + host daemon and `/nix` in project 42000. A home/tmp cap cannot stop agents + exhausting that shared allowance. Preserving the Goal's full isolation + requires separating agent and CI writable Nix storage and temporary builds, + not merely tagging their checkouts. Existing shared-store cache reuse is a + deliberate host design; changing it needs the maintainer's decision. +- Add bounded allocation evidence to the host's existing logging: physical + free space, per-project blocks/inodes, and sizes of the identified mutable + storage roots. Record timestamps and failures. Avoid recursive whole-disk + scans every refresh and avoid claiming reflink totals are physical usage. + +The [XFS project-quota documentation](https://man7.org/linux/man-pages/man8/xfs_quota.8.html) +describes recursive initialization, inheritance, and project boundaries. +Project limits bound allocations; they do not provision a separate volume. + +Decisions to return to the foreground before implementation: + +1. **Owner:** extend the existing host-owner work (recommended), or explicitly + assign a separate host PR based on the current gate configuration. Neither + host PR has been changed by this investigation. +2. **Isolation:** retain the full Goal and separate writable Nix storage + (recommended), or explicitly narrow the guarantee to direct agent writes + outside the shared Nix pool. A dedicated CI filesystem is an alternative + with a clearer capacity boundary, but needs storage/migration approval. +3. **Capacity:** approve the CI floor, agent block/inode caps, and residual host + budget together after measuring representative gate peaks. No values are + chosen. The historical 300 GiB CI cap is not a safe default: the current + quota snapshot already accounts for about 435 GiB in project 42000. + +Remaining work is attribution of the original growth where evidence permits, +the approved host source change, and verification in the host's XFS VM tests: +agent block/inode exhaustion must leave CI writable; cover private temporary +files, shared-store policy, existing-file migration, concurrent writers, +disabled enforcement, and restart recovery for the services still in use. +Live migration, deployment, cleanup, and quota changes remain separate +maintainer actions. This checkpoint changes none of them. Decided 2026-10-30 (maintainer, /quest-plan of the 2026-10-30 /quest-complete follow-ups): ✅ new m0 quest / note in the host configuration only / drop. From af0812143afba9b618acdff4d185db6e678bdb15 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 13:07:34 -0800 Subject: [PATCH 3/5] chore(quest): claim quest/m0/ci/queue-host-disk.md Quest-Claim: ee1a1a9050cf0bac From 5f6194c23bdde0e053e7e51fe3f66620ae94faa4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 17:01:41 -0800 Subject: [PATCH 4/5] docs(quest): record the shared Nix daemon blocking queue host disk separation Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/queue-host-disk.md | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/quest/m0/ci/queue-host-disk.md b/quest/m0/ci/queue-host-disk.md index 990a65663df..35e09e6d54b 100644 --- a/quest/m0/ci/queue-host-disk.md +++ b/quest/m0/ci/queue-host-disk.md @@ -185,6 +185,43 @@ CI's resident use stays near its 380-476 GiB post-purge range. Its current head adds no such cleanup, and the 2026-10-31 purge was a one-off. Confirm the cleanup holds across the gate's run history. +### Open: which side leaves the shared Nix daemon + +A read-only check on 2026-11-03 found the topology described above still in +place: NixOS containers `moq-ci`, `moq-pro`, `quest-ci`, `rind-ci`, and +`rust2-ci`, plus the `quest-land-*` watchers and check/fixer sockets. The +root filesystem had 228 GiB free of 953 GiB. The host deploys from +`ci/public-merge-queues-cutover` +([kixelated/nixos#19](https://github.com/kixelated/nixos/pull/19)), which +merges #16. Other sessions pushed both on 2026-11-03. + +The host runs a single `nix-daemon` (Nix 2.34.9). Agents, the runner +containers, and the gate's check and fixer instances all use it. It runs in +the `ci-builders` network namespace, with CI's egress policy and CPU and +memory caps. Its `build-dir` is `/var/lib/ci-nix-builds`, which is in +project 42000, and `kixelated` is one of its trusted users. Giving agents +and CI separate writable stores (decision 2) therefore means running a +second daemon. The Plan does not yet say which side moves to it: + +1. **CI moves to a new store (recommended).** A second daemon stays in the + `ci-builders` namespace and keeps CI's caps. Its store root is in project + 42000, and it is mounted at `/nix/store` inside the runner containers and + the gate's instance roots. Each container's system closure is copied into + that store when the container starts. The host's `/nix`, its build + directory, and its daemon (which leaves the CI network namespace) move + into the agent project. Agent workflows and their cache stay unchanged. +2. **Agents move to a new store.** A second daemon serves a store root inside + the agent project. Agent `nix develop` and `nix run` sessions run in a + mount namespace that puts that store at `/nix/store`. Unless the store is + layered over the host store with the experimental `local-overlay-store`, + that hides the host system's closure. CI stays unchanged, but every agent + session on the host changes. + +Decision 1 sends the host change to the existing owners' work, but other +sessions hold #16 and #19. The change therefore goes in a new +`kixelated/nixos` PR built on `ci/public-merge-queues-cutover`, which merges +that branch as it moves. + Decided 2026-11-01 (maintainer, /quest-complete iterations of [moq-dev/moq#7548](https://github.com/moq-dev/moq/pull/7548)): From 2b796e6bc89e88c0acc5aa735d8b1205d9e13260 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 17:33:57 -0800 Subject: [PATCH 5/5] quest(ci): drop the queue-host disk quest The maintainer dropped it on 2026-11-03: the disk has not filled since 2026-10-30, nixos has 215-246 GB free, and an agent-storage GC now runs. Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/README.md | 1 - quest/m0/ci/queue-host-disk.md | 254 --------------------------------- 2 files changed, 255 deletions(-) delete mode 100644 quest/m0/ci/queue-host-disk.md diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 32c131c796e..4db7e1c5eeb 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -162,7 +162,6 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Unify dependency features](/quest/m0/ci/hakari.md) - align the FFI builds' features and record the hosted platform jobs' hits once the workspace-hack lands - [Drop main from the publish Environment](/quest/m0/ci/publish-release-only.md) - the maintainer narrows `publish` to `release` and release tags once no job naming it runs as `main` - [Binary release publishers run on their tag](/quest/m0/ci/release-tag-publishers.md) - brew, Go and Swift releases stop running through `workflow_run` on main -- [The queue host keeps the queue's disk](/quest/m0/ci/queue-host-disk.md) - agent sessions can't starve the queue runner of disk, and a crashed runner restarts - [The bench comparison builds moq-gst's catalog bench on its base](/quest/m0/ci/bench-base-build.md) - A paired bench comparison that selects `rust/gst/moq-gst/catalog` against `origin/main` builds the base side and reports a comparison, instead of marking it unavailable with "base build failed" - [The workspace-hack refuses trybuild edges and spares non-media builds](/quest/m0/ci/hakari-followups.md) - `hakari.py` refuses trybuild dev-dependency edges, and non-media Linux test builds stop needing libpipewire and libclang - [The flake lane skips hosts that can't map user namespaces](/quest/m0/ci/userns-uid-map.md) - `check flake` skips instead of failing where `/proc/self/uid_map` is unwritable diff --git a/quest/m0/ci/queue-host-disk.md b/quest/m0/ci/queue-host-disk.md deleted file mode 100644 index 35e09e6d54b..00000000000 --- a/quest/m0/ci/queue-host-disk.md +++ /dev/null @@ -1,254 +0,0 @@ -# [M] The queue host keeps the queue's disk - -## Goal - -Agent sessions sharing the self-hosted queue host can never fill its disk to -the point where the `moq-queue` runner fails a check, and a runner that does -crash comes back on its own. - -## Plan - -### Incident and attribution - -On 2026-10-30 the host filled twice (about 03:00 and 07:30 PDT). The original -investigation recorded roughly 480 GB of non-CI growth, later freed. Queue -checks ejected [moq-dev/moq#6985](https://github.com/moq-dev/moq/pull/6985) -and [moq-dev/moq#6999](https://github.com/moq-dev/moq/pull/6999), and at 07:49 -the runner aborted and stayed down for 36 minutes. - -Read-only investigation on 2026-11-01 confirmed the capacity failure from -`journalctl -u ci-storage-refresh.service --since 2026-10-30`: -at 03:00:50 PDT, CI used 249371398144 bytes against a proposed -249002205184-byte limit. At 08:05:10 it used 145833361408 bytes against a -proposed 119020105728-byte limit. These are failed refresh calculations, -not evidence that those limits were successfully installed. - -The live `ci-storage.py` computes the allowance as current project usage -plus physical free space minus 32 GiB. CI therefore fails once free space -falls under that 32 GiB reserve, whoever consumed it. Nothing bounds the -other writers, so today an agent can take the space CI needs. - -The consumer of the original 480 GB remains unknown. The first retained -`agent-storage-gc` journal entry is 2026-10-30 08:21:49 PDT, after both -incidents. That run recovered 8 GiB. Its logs contain global free space and -selected removals, not per-writer allocation history. Neither those logs nor -today's directory sizes establish which process allocated and freed the -incident data. Do not attribute the bursts to worktrees, MBX, or test scratch -without allocation evidence. - -### Configuration ownership and live state - -The home host is `nixos`, configured in `kixelated/nixos`, not this -repository's `infra/`. Existing work is owned by the maintainer's host PRs: - -- [kixelated/nixos#15](https://github.com/kixelated/nixos/pull/15), open at - `f4e822e4ce7332be253bfc6ce9a0b8e5c60ba4e0`, owns idle-output GC and runner - restart recovery. GC is deployed and its timer enabled. The real container - unit, `github-runner-runner.service`, was active with `Restart=no` on - 2026-11-01; the restart commit is still not active there. Do not recreate - either fix in MoQ. -- [kixelated/nixos#16](https://github.com/kixelated/nixos/pull/16), open at - `27700296eb7357001a1c44257cb6c140236b5925`, owns the newer host landing - daemons and ephemeral check/fixer containers. The live - `quest-land-moq.service` is active with `Restart=on-failure`. Storage work - must cover this gate as well as the retained Actions runner. A deployment - branch combines host changes; `main` alone does not describe the live host. - -Read-only `findmnt`, `xfs_quota ... state/report`, and `xfs_io -r -c stat` -on 2026-11-01 found: - -- `/home`, `/tmp`, and `/var/tmp` share the root XFS filesystem with project - enforcement on. Their directory project IDs are 0 without project - inheritance. Only projects 0 and 42000 appear in the quota report. -- CI project 42000 has `limitGiB=null`, a 32 GiB reserve, and eight million - inodes. Its configured trees include `/nix`, native container state, build - scratch, and the newer gate's caches and staging trees. The sampled - `/var/cache/quest-land-moq-check` has project 42000 and inheritance enabled. -- Agent storage extends beyond worktree outputs: home caches (MBX, a media - build cache), rootless container storage, and user-owned `/var/tmp` - benchmark output. Sizes are under "Measured peaks" below; they are `du` - allocated block usage, not inode counts, incident attribution, or unique - physical usage with reflinks. -- `/var/lib/quest-land` has project ID 0. The gate and its launcher use - `PrivateTmp=true`; the host has their private directories under both - `/tmp` and `/var/tmp`. Capping either shared temporary root indiscriminately - can therefore make the gate hit the agent quota too. - -### Measured peaks (2026-11-01, read-only) - -Sources: the per-minute `ci-storage-refresh` journal since 2026-10-26 -(7687 samples; free space is `limit - current + 32 GiB`), the user -`agent-storage-gc` journal, systemd's per-unit `written to disk` totals for -621 gate check and fixer runs since 2026-10-30, `xfs_quota report -p`, and -`du -x` of the mutable roots. Nothing was changed or built for this. - -- Filesystem: 952.4 GiB XFS root, 615.8 GiB used, 336.6 GiB free, 2% of - inodes used. Project 0 is charged 265.2 GiB and 3.27M inodes; project - 42000 is charged 426.7 GiB and 5.81M of its 8M inodes. The charged sum - exceeds physical use by 76 GiB, consistent with XFS charging shared - (reflinked) extents to every owning file. Limits expressed in charged - bytes are therefore conservative against physical space. -- CI charged usage: 137-247 GiB from 2026-10-26 to 2026-10-29. From - 2026-10-30 19:00 PDT it grew steadily to 842 GiB at 2026-10-31 22:04, - leaving 103 GiB free, because finished check and fixer instances kept - their per-instance caches. A one-off root `systemd-run` purge of those - caches at 22:24 dropped it from 834 to 384 GiB within three minutes. - Since then it has stayed between 380 and 476 GiB. Completed instance - directories now measure 0 MiB. That leak is CI's own growth, which the - gate's cleanup must prevent (below); no storage cap absorbs it. -- Gate runs since the purge (286 runs): `quest-land-moq-check` wrote a - p50 7.0, p90 10.1, and maximum 21.4 GiB per run; rust2 9.6, moq-pro 1.2, - rind 2.0, quest 0.6, and the fixer up to 25.1 GiB. Over the whole window, - moq-check peaked at 29.1 and moq-pro at 20.1 GiB. At most five runs - overlapped, and the largest concurrent total was 45.9 GiB across three. - The active moq-check instance measured 11.6 GiB with `du`. -- CI resident trees: `/nix` 102.1 GiB, `/var/cache/quest-land-caches` - 222.0 GiB (moq 136.9, moq-pro 66.2, rind 18.7), `/var/cache/ci-mbx` - 51.9 GiB, and `/var/lib/nixos-containers` 45.3 GiB. -- Agent trees: `/home/kixelated` 205.5 GiB (caches 91.8, including MBX - 40.7 and a media build cache 39.1; worktrees 57.2; `.local` 27.9), - `/tmp` 8.1 GiB, and `/var/tmp` 8.6 GiB, about 222 GiB together. The rest - of project 0, about 43 GiB, is other host state. -- Non-CI incidents: on 2026-10-30, free space fell to 6 GiB while CI was - charged under 250 GiB, then rose 578 GiB at 08:00 PDT. That non-CI writer - remains unattributed. - -### Storage separation - -Implementation stays in the host repository, extending the existing owners' -work above. Reuse its XFS policy and VM tests. Everything stays on the one -shared XFS filesystem; the only new limit is a cap on agents. - -- Give agent-writable trees one enforced project with a 340 GiB block limit - in charged bytes and a 10M inode limit. It covers home caches, rootless - container storage, agent temporary/benchmark outputs, and the agent Nix - store below. Put CI private temporary files and daemon state in CI-owned - trees before tagging the shared temporary roots. Check existing files and - inheritance, including rename/hard-link boundaries; tagging only a - directory does not account for its existing descendants. Preserve - existing data during any migration. -- Separate the writable Nix storage first. Agent `nix develop` and CI use - the host daemon and `/nix` in project 42000, so a home/tmp cap cannot stop - agents exhausting the space CI uses. Give agents and CI separate writable - Nix stores and temporary build trees, rather than tagging only their - checkouts. The agent store and its build trees are inside the agent - project and count against its 340 GiB. Agents lose today's shared-store - cache reuse with CI. -- CI and the rest of the host share whatever the agent cap leaves, with no - CI floor and no separate host budget. `ci-storage.py` keeps its existing - free-space allowance and 32 GiB reserve. -- Add bounded allocation evidence to the host's existing logging: physical - free space, per-project blocks/inodes, and sizes of the identified mutable - storage roots. Record timestamps and failures. Avoid recursive whole-disk - scans every refresh and avoid claiming reflink totals are physical usage. - -The [XFS project-quota documentation](https://man7.org/linux/man-pages/man8/xfs_quota.8.html) -describes recursive initialization, inheritance, and project boundaries. -Project limits bound allocations; they do not provision a separate volume. - -The agent limit is sized from the measured peaks: 222 GiB of agent trees -plus at most 102 GiB for an agent store sized like today's shared `/nix`, -324 GiB together, leaving agents 16 GiB of headroom. The agent store's real -size can only be measured after the separation deploys; resize the limit -from that measurement. The inode limit is sized the same way. All of -project 0 holds 3.27M inodes today. `/nix`'s own count was not measured, -but it is part of project 42000's 5.81M, so an agent store like it adds at -most 5.81M, about 9.1M together. The 10M limit covers that bound. Before -deployment, count `/nix` inodes read-only, and after the separation resize -the limit from the agent project's measured inode use. - -The cap keeps the Goal with a 56 GiB margin. CI fails once free space -falls under its 32 GiB reserve, so writers can use 920.4 GiB before CI -fails. Agents at their cap hold 340 GiB plus uncharged inode metadata, -under 5 GiB at 10M inodes of 512 bytes. That leaves about 575 GiB for CI and -other host state. Their measured peak together is 519 GiB (CI 476, other -host state 43), so 56 GiB remains, more than the 45.9 GiB largest -concurrent gate write. Charged bytes overstate physical use under reflinks, -so the margin is conservative. - -The cap covers only agent writes inside the agent project. By design it -does not stop CI or other host state growing past 575 GiB on its own: the -842 GiB cache leak plus 43 GiB of host state would exceed that even with -agents idle. The unattributed 2026-10-30 burst left 6 GiB free with CI -charged under 250 GiB. CI's charged bytes bound its physical use from -above, so the extents non-CI files referenced then covered at least -696 GiB of the 946 GiB used, and their charged bytes were at least that. -If agents wrote it in covered trees, the cap would have stopped them at -340 GiB. If a host -service or an agent path outside the project wrote it, the cap would not. -Attribute it where evidence permits, and after migration verify that no -agent-owned data remains outside the agent project. - -The gate's owner, [kixelated/nixos#16](https://github.com/kixelated/nixos/pull/16), -must make every finished check and fixer instance release its caches, so -CI's resident use stays near its 380-476 GiB post-purge range. Its current -head adds no such cleanup, and the 2026-10-31 purge was a one-off. Confirm -the cleanup holds across the gate's run history. - -### Open: which side leaves the shared Nix daemon - -A read-only check on 2026-11-03 found the topology described above still in -place: NixOS containers `moq-ci`, `moq-pro`, `quest-ci`, `rind-ci`, and -`rust2-ci`, plus the `quest-land-*` watchers and check/fixer sockets. The -root filesystem had 228 GiB free of 953 GiB. The host deploys from -`ci/public-merge-queues-cutover` -([kixelated/nixos#19](https://github.com/kixelated/nixos/pull/19)), which -merges #16. Other sessions pushed both on 2026-11-03. - -The host runs a single `nix-daemon` (Nix 2.34.9). Agents, the runner -containers, and the gate's check and fixer instances all use it. It runs in -the `ci-builders` network namespace, with CI's egress policy and CPU and -memory caps. Its `build-dir` is `/var/lib/ci-nix-builds`, which is in -project 42000, and `kixelated` is one of its trusted users. Giving agents -and CI separate writable stores (decision 2) therefore means running a -second daemon. The Plan does not yet say which side moves to it: - -1. **CI moves to a new store (recommended).** A second daemon stays in the - `ci-builders` namespace and keeps CI's caps. Its store root is in project - 42000, and it is mounted at `/nix/store` inside the runner containers and - the gate's instance roots. Each container's system closure is copied into - that store when the container starts. The host's `/nix`, its build - directory, and its daemon (which leaves the CI network namespace) move - into the agent project. Agent workflows and their cache stay unchanged. -2. **Agents move to a new store.** A second daemon serves a store root inside - the agent project. Agent `nix develop` and `nix run` sessions run in a - mount namespace that puts that store at `/nix/store`. Unless the store is - layered over the host store with the experimental `local-overlay-store`, - that hides the host system's closure. CI stays unchanged, but every agent - session on the host changes. - -Decision 1 sends the host change to the existing owners' work, but other -sessions hold #16 and #19. The change therefore goes in a new -`kixelated/nixos` PR built on `ci/public-merge-queues-cutover`, which merges -that branch as it moves. - -Decided 2026-11-01 (maintainer, /quest-complete iterations of -[moq-dev/moq#7548](https://github.com/moq-dev/moq/pull/7548)): - -1. **Owner:** ✅ extend the existing host owner's work / a separate host PR. -2. **Isolation:** ✅ keep the full Goal, with separate writable Nix storage - for agents and for CI / narrow the guarantee / a dedicated CI filesystem. -3. **Capacity measurement:** ✅ measure representative gate peaks before - approving any values / pick values without measuring. -4. **Capacity shape:** ✅ cap agents only: one agent quota of about 340 GiB - plus an inode limit, sized from the measured agent peak and the agent - Nix store; CI and the rest of the host share what is left, with no CI - floor or host budget, and gate cleanup handles CI's cache leak / A: CI - floor 520 GiB, agents 340 GiB, host budget 60 GiB / B: CI floor - 560 GiB, agents 300 GiB, host budget 60 GiB. The maintainer first asked - why CI and the host could not share the disk; it already is one shared - filesystem, and the limits are only caps on it. This drops the CI floor - and host budget that decision 3 had planned to approve. - -Remaining work is attribution of the original growth where evidence permits, -the host source change, the gate's cache cleanup, and verification in the -host's XFS VM tests: agent block/inode exhaustion must leave CI writable; -cover private temporary files, the separate Nix stores, existing-file -migration, concurrent writers, disabled enforcement, and restart recovery -for the services still in use. Live migration, deployment, cleanup, and -quota changes remain separate maintainer actions. This checkpoint changes -none of them. - -Decided 2026-10-30 (maintainer, /quest-plan of the 2026-10-30 /quest-complete follow-ups): ✅ new m0 quest / note in the host configuration only / drop. - -Public API: none. Wire: none.