Repository navigation
fix(dev-fleet): run the Pull+Build lockfile probe on the repo's filesystem - #11435
Conversation
|
Intent: Make Dev Fleet's Pull + Build installability probe rehearse the real |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify headroom: the probe's scratch coexists with [DESIGN-REVIEWED] 12e29ca |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The sweeper is a recursive delete in the operator's checkout root — verify the marker+age gate once on a real fleet host with a concurrent Pull + Build. What this change shipsInventory (10 items) — 10 justifiedIntent: FIX — Pull + Build refused as "out of space" with 33 GiB free because the install rehearsal ran on a tmpfs whose inodes, not bytes, were exhausted; the rehearsal moves onto the filesystem the real install writes to and the refusal names both budgets. Provenance is a written reproduction with measured numbers (55 175 free inodes vs 59 586 needed), and the fix sits at cause level (rehearsal on the wrong filesystem), not at the message.
Duplication checked: [FIRST-PRINCIPLES-REVIEWED] 12e29ca |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsNo package.json or .npmrc at the repo root, so No findings. [OPUS-REVIEWED] 12e29ca Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
707ef01 to
1b170a2
Compare
|
self-added: yes
|
1b170a2 to
9fccfc6
Compare
9fccfc6 to
8c012c7
Compare
Pushed
|
| Shard | Failing tests | This PR | main 04727ebe0 (run 35172978466) |
|---|---|---|---|
| Windows 3 | test_fargate_launch_engine.py (2 tests, same names) |
2 failed | 2 failed |
| Windows 4 | test_ledger_retention.py, test_ledger_kinds.py |
46 failed | failed |
| Windows 6 | test_remove_slot_for_history_key.py (same test) |
1 failed | 1 failed |
| Linux 2, 3 | same two families | still running | failed |
| Coverage Gate | cascade of the above | - | failed |
Main additionally reds two Linux shards and Coverage Gate, which this PR's run had not
reached yet at the time of reading -- so those are expected to red here too, from the
same cause.
Both breakages are already tracked, with fixes open. Ledger entry validation from
#11317 refuses writes three test files still make without agent/actor: issue #11443,
fix PRs #11445 and #11441. The Fargate engine drifted from the widened LaunchEngine
protocol: PRs #11442, #11432, #11426, #11422.
Nothing in this PR can clear those. It will go green on a rebase once they land; the
review lanes are the gates that are actually measuring this diff.
8c012c7 to
d0561ea
Compare
|
d0561ea to
f77f1d4
Compare
UX Review (Fable 5) — ⏭️ skippedRevision |
|
/ai-review override gpt 30a8c3b: The actor who could plant the trigger (a git-tracked dir at checkout root bearing this module private scratch prefix and private marker filename) already holds write access to that checkout and can corrupt it directly, so the sweep path grants nothing new; the module never commits such an artifact (mkdtemp, git-ignored, finally-removed); and any content lost is git-tracked and recoverable from the object store, so it is not unrecoverable. |
Human judgment recorded@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
1159543 to
11f2d0d
Compare
|
CI note on head The rebase cleared the four One red remains, A pristine detached checkout of current main ( This PR's diff is six files ( |
|
Second main-owned red on head A pristine detached checkout of current main ( Board accounting for this head, then: two reds, both main's and both tracked (#11634 with fix PR #11632, and #11635), plus Rebasing again is not the remedy for either one, because both breakages live on main's own tip. |
|
Disposition of the two verification requests on head Design: nothing must pin First Principles: verify the checkout root and Neither item is going in as its own push. Two reds on this head are main's (#11634 with fix PR #11632, and #11635), so this branch needs one more rebase once those land; the |
…ystem Pull + Build rehearses the incoming lockfile with a real script-free install before it touches the checkout, and that rehearsal ran in TMPDIR. On a host whose TMPDIR is memory-backed the rehearsal answers about the wrong filesystem: it exhausted that filesystem's fixed file-slot budget and reported "not enough room ... free space in the temporary directory", so an operator who checked exactly that saw 33 GiB free and read the refusal as wrong. The real numbers were 55 175 free file slots against 59 586 needed. The rehearsal now installs on the filesystem the real npm ci writes to, but only when git check-ignore confirms the checkout hides the scratch name; otherwise it falls back to TMPDIR as before, because an un-ignored checkout would silently read dirty and fail Prune merged closed. Out of room at scratch creation is the verdict and is never retried elsewhere. Quota exhaustion classifies as out-of-room in the errno and npm-output paths alike. Abandoned scratch trees older than _SCRATCH_STALE_SECS are swept before the new one is created, and only directories carrying an ownership marker written by this module are ever deleted, because the sweep is a recursive delete in the operator's checkout root. A test derives every deadline the module declares from its own source and requires the sweep window to exceed their sum by a wide margin, so raising a timeout without widening the window fails rather than deleting a live probe's scratch. The refusal message names both budgets and both filesystems, and the two staging-failure messages in worktree_ops.py get the same sentence. Fixes #11434
11f2d0d to
12e29ca
Compare
|
Disposition of both verification requests on head First Principles: verify the marker-plus-age gate on a real host with a concurrent Pull + Build. Done in this checkout root, driving the shipped Both guards were exercised in the direction that matters. The concurrent probe is marked, so only its youth saved it; the look-alike is aged, so only the missing marker saved it. Design: verify headroom, since the probe's scratch coexists with os.rename(self.stash, self.backup)
So the real install's peak is the backup tree plus the tree being written, and the probe's peak is Read the other way, the arithmetic argues for this design. A rehearsal that peaked at one tree would pass on a checkout with 1.5 trees of room, and then the real two-tree install would fail after merge -- which is the outcome this preflight exists to prevent. Nothing is recorded in the spec for it, because there is no trade-off to record. |
|
Green on Every marker names this head rather than an earlier one, and all 18 backend shard rows are present and green -- an absent shard row is unmeasured, not passing, which is how a rebase can read as a clearance it has not earned. Earlier reds on this branch are accounted for and none survive: the One item is deliberately not in this PR and is tracked: #11459, where a probe that falls back to Not merging. This awaits a human. |
What is the problem?
Pull + Build refused with
preflight: not enough room in the scratch directoryon a hostwith 33 GiB free in
/tmp. The operator checksdf -h, sees plenty, and the messagehas told them nothing they can act on.
/tmpwas a memory-backed filesystem with its inode count capped at mount time(
nr_inodes=1048576), shared with every process on the box. Unrelated litter had takenthe free file count to 55 175 slots. A
node_modulestree needs 59 586 files. tmpfsreturns
ENOSPCwhen it runs out of INODES, the same errno as a full disk, so the stepreported the only budget it knew about (bytes) and sent the operator to check the one that
was fine.
Why this issue matters to the user
Two harms, and the second is the larger one.
The refusal is unactionable.
df -hsays 33 GiB free, the message says free up space, andPull + Build stays blocked until someone thinks to run
df -i. This is the exact failureclass the preflight exists to remove: it was built so a sync refusal names a cause instead
of echoing the last output line.
And the verdict was answering the wrong question. The preflight exists to rehearse the
real
npm ci, which installs into the checkout. Rehearsing in/tmpmeasures afilesystem the install never touches, so it can pass where the real step will fail for
room, or fail (as here) where the real step would have succeeded. The repo's own
filesystem had 259 million free inodes at that moment.
How our fix solves it
The rehearsal moves onto the filesystem it is rehearsing.
_scratch_parenttakes thescratch from the repo root, so the room budget the probe measures is the one the real
npm ciis charged against. The root rather thanwebsite/keeps the directory outsidethe frontend project
npmresolves config against and outside the subtree thebackend-only skip decision reads.
The repo root is used only where git hides the name. This module ships with the
installed gateway while the ignore rule covering its scratch name is a commit in the
checkout's own history, so a fleet checkout parked on an older ref can run this code with
no rule for it -- and a probe killed in that window leaves an untracked directory in the
checkout root, which reads as dirty and fail-closes
Prune merged. That is this PR's ownbug class arriving from the other side.
_scratch_name_is_ignoredasksgit check-ignoreabout a generated name, and only a yes lets the repo host the scratch.
git check-ignoreis the oracle rather than a read of
.gitignore, because ignore resolution spans severalfiles with precedence and negation. An unanswerable question (missing git, a timeout, not
a repo) reads as NOT ignored: being wrong that way costs a rehearsal on
TMPDIR, thisstep's previous behaviour, while the other way costs a checkout that silently reads dirty.
Abandoned scratch directories are swept, because the move created the need for one.
probedeletes its own in afinally, so what survives is a run that never reached it:SIGKILL, an OOM kill, a reboot.
/tmpwas age-cleaned by the host; the checkout root iscleaned by nobody and the name is now git-ignored, so without a sweeper an abandoned tree
would accumulate there invisibly and permanently.
_sweep_stale_scratchremovesprefix-matching directories older than
_SCRATCH_STALE_SECS(6 hours) and runs beforethe new scratch is created. That order is what makes it a remedy rather than hygiene: the
room the litter holds is charged to the same budgets the incoming install is measured
against, so on a filesystem litter has filled, the sweep is what lets the probe run at
all. It runs whenever the repo could host a scratch, including when the ignore gate then
sends this probe to
TMPDIR, since litter from an earlier gateway build is exactly what anun-ignored checkout needs cleared. Four properties keep it from being its own hazard. It
takes no lock, so the 6-hour window is the concurrency guard, and a live probe's scratch is
bounded by its own timeout plus fixed-timeout helpers, well inside it. It matches only the
scratch prefix and only directories, because it runs in the operator's checkout root. It
deletes only a directory carrying
.kirocrew-probe-owned, a marker_make_scratchwritesinside every scratch it creates, and only when that marker is a REGULAR file -- a symlinked
marker is refused, or whatever planted the link would choose what gets destroyed. A name
prefix is a convention, not proof of authorship, and this is a recursive delete in the
operator's checkout root, so the marker and the age divide the question and neither implies
the other: the marker answers "is this MINE", the age answers "is it still IN USE". Writing
the marker is best-effort, deliberately in that direction -- a failed write leaves the
directory unswept, which costs room and room is reclaimable, while deleting an unmarked
directory costs data nobody can get back. And every failure is swallowed, because
housekeeping must never be why a verification does not happen.
The window and the probe's own deadline are one invariant kept in two places, so a test
pins the comparison instead of the prose asserting it. It reads
probe's default timeoutand each fixed helper timeout out of the module's source, charges a single probe with all
of them back to back, and requires
_SCRATCH_STALE_SECSto exceed that sum by a widemargin -- 21600 s against 1320 s today. Raising a deadline, or adding a helper, without
widening the window fails that test. The ownership marker cannot cover this case: a
running probe's scratch is genuinely marked and genuinely prefix-named, so age is the only
thing that separates it from litter, which is why the two numbers may not drift apart.
Being out of room is the verdict, never a reason to relocate. A scratch creation that
fails for room says the filesystem the real install targets has none, which is the answer.
Only conditions that make the repo unusable as a host at all fall back. Out of room covers
ENOSPCandEDQUOTtogether, since a per-user quota is how a managed host says the samething and only one of them is spelled
ENOSPC.The message stops naming one budget and one filesystem.
EXIT_NO_SPACEnow sends theoperator to check the free bytes (
df -h) and the free file count (df -i), on thecheckout's filesystem and on the package cache's: the install writes both and they need not
be the same one, so a full cache raises the same errno while the scratch's own filesystem
looks healthy. The same correction lands on the two sibling refusals on this button,
could not stage the dependency preflightandcould not stage the sync runnerinworktree_ops.py, which both sent the operator to bytes alone on aTMPDIRENOSPC.Their scratch dirs stay in
TMPDIR: they hold a handful of files, not a dependency tree,so residency is not their problem; the sentence was.
The spec moves with the code.
docs/system-specs/modules/dev-fleet.mdgains scratchresidency, the ignore gate, the sweeper and the no-fallback-on-no-room rule, so the next
change to this step does not reason from a stale contract.
What tests we did
test/test_dev_fleet_npm_preflight.py: 82 passed. black, isort, flake8 and mypy cleanon all three changed source files.
Reproduced first, then verified. Before:
preflight: not enough room ...alongsidenpm error code ENOSPC, withdf -h /tmpshowing 33 GiB free anddf -i /tmpshowing55 175 slots against 59 586 needed. After: the probe consumes 0 tmpfs inodes.
New coverage, each pinned by a mutation run that shows the test fails when the property is
removed (the gate, the conservative reading of an unanswerable
check-ignore, the sweepcall, its prefix filter, its age window, and its ordering before creation):
TMPDIRwhen it does not*does not satisfy the gateTMPDIRprefix is never deleted; nor is one whose marker is a symlink
docstring as an OUTCOME test, because mutation showed
shutil.rmtreerefuses a symlinkby itself, so the
follow_symlinks=Falseflags are not what saves the target; they staybecause reading an mtime through a link answers the wrong staleness question
than merely clearing it; mutation confirms all five variants red it -- a probe timeout
raised past the window, one raised only enough to eat the margin, a shrunken window, a
lengthened helper deadline, and a brand-new long deadline
ENOSPCandEDQUOT, statedunder the condition that makes it apply
errno.EDQUOTis read withgetattrso the module and its tests load on Windows, whereit is not defined
One pre-existing red is unrelated:
test_dev_fleet_app.py::test_trusted_bin_pins_the_resolved_target_not_the_symlinkfailsidentically on pristine main at
04727ebe0f, fromKIROCREW_DEVFLEET_BIN_GHin the hostenvironment.
Any other suggestions on the work
One residual is recorded rather than fixed here. When the ignore gate or an unwritable
checkout sends the probe to
TMPDIR, nothing in the output says the rehearsal ran offtarget, so a pass carries less than it appears to. That is a reporting change to the run's
detail line, not a fix to this diff, and folding it in on top of a converged review would
be the wrong trade. Filed as #11459 so the disclosure is tracked rather than only written
down -- two review lanes pointed at it independently.
Pattern harvest
Rule candidate: grep/semgrep on operator-facing out-of-room text
Pattern: a refusal raised by an out-of-room errno whose remedy names ONE budget.
ENOSPCis returned for free BYTES and for free FILE SLOTS (inodes), and amemory-backed filesystem reaches its inode cap long before its bytes run short --
so "free up space" is unactionable exactly when it is most needed. A refusal
carrying an out-of-room errno should name both budgets (
df -handdf -i), andshould not assert which filesystem ran out when the operation writes more than
one.
Generalizable rather than a one-off: the same sentence was found three times on
the same button.
npm_preflight.EXIT_NO_SPACEis the one #11434 reported;grep 'free space in the temporary directory'then turned upworktree_ops.pystaging the dependency preflight and staging the sync runner,both sending the operator to bytes alone on a
TMPDIRENOSPC. All three arefixed here. A rule is worth having because the next such message will be written
by someone reading one of these as the house style.
Second, weaker candidate, recorded not proposed:
tempfile.mkdtemp()/TemporaryDirectory()with nodir=inside code whose purpose is to REHEARSE orMEASURE work that will happen on a different filesystem. The bare call takes
TMPDIR, which silently answers a question about the wrong filesystem. Hard toexpress mechanically, because the intent -- rehearsal versus ordinary scratch --
is not in the syntax.
Fixes #11434