Repository navigation
Railway lab /data at 98%: per-lane cargo targets are never reaped #7062
Description
Problem
The Railway lab volume /data (service t27c-lab) is at 98%: df -h /data printed 46G 45G 936M 98% at 2026-10-06T19:41:51Z (lab clock). A release build of t27c + tri needs several hundred MB of target dir, so the next lane that builds from a clean target can fail with No space left on device mid-build, and a half-written target dir makes the space problem worse.
Each lane builds into its own /data/<lane>-target and nothing removes it when the lane's PR merges. The largest entries, measured in the same run (du -sh, newest mtime from find -maxdepth 3):
| Size | Dir | Newest write (UTC) |
|---|---|---|
| 17G | ref |
2026-10-06T14:45 |
| 4.1G | g6446-target |
2026-10-06T12:07 |
| 4.0G | bext |
2026-10-05T15:08 |
| 2.8G | elan |
2026-10-06T06:47 |
| 1.6G | scratch-t27a |
2026-10-06T14:36 |
| 1.1G | vc6655-target |
2026-10-06T19:32 |
| 954M | s6913-target |
2026-10-06T15:22 |
| 725M | w6609-target |
2026-10-05T19:59 |
| 661M | gen-target |
2026-10-04T16:11 |
| 581M | target |
2026-10-06T19:37 |
| 522M | gengate-target |
2026-10-05T17:44 |
| 511M | g7-target |
2026-10-06T14:58 |
| 425M | src |
2026-10-06T19:39 |
| 421M | lane2 |
2026-10-06T13:44 |
Four of them have had no write for more than 24 hours: bext, w6609-target, gen-target, gengate-target (about 5.9G together).
The queen-actors loop (epic #6971) removed only its own: the /data/tm6993 worktree (via git -C /data/src worktree remove), /data/tm6993-target, /data/tm6993-acc and its tm6993* / tm7022* scratch files, after #7043 merged. That freed about 370M. Nothing belonging to another lane was touched.
Proposal
- Each owning lane checks its dir above and removes it if its PR has merged; a
-targetdir whose issue is closed is safe to drop, since it is a cargo cache. - A lab reaper in the shape of the bee runner's
ai.t27.runner-reap: drop/data/*-targetdirs with no write for 48 h, never/data/src,/data/reforelan. It should print what it dropped and what it kept. Its home should be atrisubcommand, not a new*.sh(L7). tri lab df, or a line in an existing lab doctor, should report/datause and warn at 90%, so the next lane sees the number before it builds.
Acceptance
-
df -h /dataon the lab shows under 85% use. - A merged lane's
-targetdir is removed by its lane or by the reaper within 48 h. - The lab doctor or
triprints/datause and warns at 90%.
Refs #6092 (t27c on the lab), #6971 (the loop that found it).
User Scenarios
- Given the Railway lab
/datavolume is at 98% use, when a merged lane's-targetdirectory is older than 48 hours with no writes, then the lab reaper removes it and prints what it dropped. - Given a lane's PR has merged, when the lane runs its cleanup, then it removes its own
/data/<lane>-targetdirectory because it is a cargo cache safe to drop. - Given a developer runs
tri lab dfor the lab doctor, when/datause exceeds 90%, then the command prints the usage and emits a warning before the lane builds.
Requirements
- FR-001: The
triCLI MUST provide areapsubcommand that drops/data/*-targetdirectories with no write for 48 hours, never/data/src,/data/ref, or/data/elan, and prints what it dropped and kept. - FR-002: Each lane's merge automation MUST remove its own
/data/<lane>-targetdirectory when its PR is merged. - FR-003: The
tri lab dfcommand (or existing lab doctor) MUST report/datausage and warn at 90%. - FR-004: After reaper and lane cleanups run,
df -h /dataon the lab MUST show under 85% use.
Boundary
- cli/tri/Cargo.toml
- infra/t27c-lab/railway.json
- bootstrap/src/railway.rs
- railway.toml
- cli/t27b/Cargo.toml
- bootstrap/t27c.py
- backend/trinity-core/Cargo.toml
- Cargo.toml
- Cargo.lock
- ffi/Cargo.toml
- tools/converter/Cargo.toml
- bindings/python/Cargo.toml
Activity
gHashTag commentedon Oct 6, 2026
Measured again at 2026-10-06T19:52:40Z (lab clock):
/data46G, 45G used, 757M free (99%), down from 936M at 19:41:51Z. This loop's own dirs were already removed at 19:41Z, so the growth is another lane's. This loop does not start a lab build under ~2G free, so the mutation pass for #7059 ran as 22 hand mutants on the Mac instead oftri mutate spec.gHashTag commentedon Oct 7, 2026
Measured again at 2026-10-07T06:58Z (lab clock):
/dataat 100%, 160M free (df -B1M: 46627 total, 46452 used).This loop removed only its own dirs, after checking that no process had its cwd in them:
/data/s6913(a clean checkout at 0f5dca2) and/data/s6913-target, pluss6913.outands6913.sh. Their PR, fix(t27c): seal records spec_path relative to the store, refuses a spec outside it (Closes #6913) #6925, merged at 2026-10-07T06:26Z./data/tm7113. Its PR, policy: the L2 GENERATION rule lives in specs/policy/l2_generation.t27; L2 checks a spec changed without its copy (Closes #7103 #7113) #7149, merged earlier.After the removal, at 07:01:02Z: 1410M free, 97%.
The largest entries now (
du -xsh, 06:59Z):ref17G,p1core-target4.2G (written 07-07T06:06),bext4.0G,elan2.8G,vc6655-target1.8G,scratch-t27a1.6G,w6609-target725M,gen-target661M,target581M,gengate-target522M,g7-target511M. They belong to other lanes, and this loop leaves them alone.A cheaper default for every lane: the container's
/tmpis the overlay root, with 604G free at the same time. This loop's cargo target for #7303/#7050 lives at/tmp/tm7303/targetand never touches/data. A cargo target is a cache, and losing it on a redeploy costs one rebuild./datacould be kept for state that must survive a redeploy.gHashTag commentedon Oct 7, 2026
Reopened 2026-10-07: the bulk close of issues without a
## Boundarysection was reverted by the owner. Nothing to do here; this comment replaces the earlier close notice.