Skip to content

fix(dev-fleet): run the Pull+Build lockfile probe on the repo's filesystem - #11435

Merged
bolichen97 merged 1 commit into
mainfrom
fix/preflight-scratch-inode
Sep 18, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/preflight-scratch-inode

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

What is the problem?

Pull + Build refused with preflight: not enough room in the scratch directory on a host
with 33 GiB free in /tmp. The operator checks df -h, sees plenty, and the message
has told them nothing they can act on.

/tmp was a memory-backed filesystem with its inode count capped at mount time
(nr_inodes=1048576), shared with every process on the box. Unrelated litter had taken
the free file count to 55 175 slots. A node_modules tree needs 59 586 files. tmpfs
returns ENOSPC when it runs out of INODES, the same errno as a full disk, so the step
reported the only budget it knew about (bytes) and sent the operator to check the one that
was fine.

Why this issue matters to the user

Two harms, and the second is the larger one.

The refusal is unactionable. df -h says 33 GiB free, the message says free up space, and
Pull + Build stays blocked until someone thinks to run df -i. This is the exact failure
class the preflight exists to remove: it was built so a sync refusal names a cause instead
of echoing the last output line.

And the verdict was answering the wrong question. The preflight exists to rehearse the
real npm ci, which installs into the checkout. Rehearsing in /tmp measures a
filesystem the install never touches, so it can pass where the real step will fail for
room, or fail (as here) where the real step would have succeeded. The repo's own
filesystem had 259 million free inodes at that moment.

How our fix solves it

The rehearsal moves onto the filesystem it is rehearsing. _scratch_parent takes the
scratch from the repo root, so the room budget the probe measures is the one the real
npm ci is charged against. The root rather than website/ keeps the directory outside
the frontend project npm resolves config against and outside the subtree the
backend-only skip decision reads.

The repo root is used only where git hides the name. This module ships with the
installed gateway while the ignore rule covering its scratch name is a commit in the
checkout's own history, so a fleet checkout parked on an older ref can run this code with
no rule for it -- and a probe killed in that window leaves an untracked directory in the
checkout root, which reads as dirty and fail-closes Prune merged. That is this PR's own
bug class arriving from the other side. _scratch_name_is_ignored asks git check-ignore
about a generated name, and only a yes lets the repo host the scratch. git check-ignore
is the oracle rather than a read of .gitignore, because ignore resolution spans several
files with precedence and negation. An unanswerable question (missing git, a timeout, not
a repo) reads as NOT ignored: being wrong that way costs a rehearsal on TMPDIR, this
step's previous behaviour, while the other way costs a checkout that silently reads dirty.

Abandoned scratch directories are swept, because the move created the need for one.
probe deletes its own in a finally, so what survives is a run that never reached it:
SIGKILL, an OOM kill, a reboot. /tmp was age-cleaned by the host; the checkout root is
cleaned by nobody and the name is now git-ignored, so without a sweeper an abandoned tree
would accumulate there invisibly and permanently. _sweep_stale_scratch removes
prefix-matching directories older than _SCRATCH_STALE_SECS (6 hours) and runs before
the new scratch is created. That order is what makes it a remedy rather than hygiene: the
room the litter holds is charged to the same budgets the incoming install is measured
against, so on a filesystem litter has filled, the sweep is what lets the probe run at
all. It runs whenever the repo could host a scratch, including when the ignore gate then
sends this probe to TMPDIR, since litter from an earlier gateway build is exactly what an
un-ignored checkout needs cleared. Four properties keep it from being its own hazard. It
takes no lock, so the 6-hour window is the concurrency guard, and a live probe's scratch is
bounded by its own timeout plus fixed-timeout helpers, well inside it. It matches only the
scratch prefix and only directories, because it runs in the operator's checkout root. It
deletes only a directory carrying .kirocrew-probe-owned, a marker _make_scratch writes
inside every scratch it creates, and only when that marker is a REGULAR file -- a symlinked
marker is refused, or whatever planted the link would choose what gets destroyed. A name
prefix is a convention, not proof of authorship, and this is a recursive delete in the
operator's checkout root, so the marker and the age divide the question and neither implies
the other: the marker answers "is this MINE", the age answers "is it still IN USE". Writing
the marker is best-effort, deliberately in that direction -- a failed write leaves the
directory unswept, which costs room and room is reclaimable, while deleting an unmarked
directory costs data nobody can get back. And every failure is swallowed, because
housekeeping must never be why a verification does not happen.

The window and the probe's own deadline are one invariant kept in two places, so a test
pins the comparison instead of the prose asserting it. It reads probe's default timeout
and each fixed helper timeout out of the module's source, charges a single probe with all
of them back to back, and requires _SCRATCH_STALE_SECS to exceed that sum by a wide
margin -- 21600 s against 1320 s today. Raising a deadline, or adding a helper, without
widening the window fails that test. The ownership marker cannot cover this case: a
running probe's scratch is genuinely marked and genuinely prefix-named, so age is the only
thing that separates it from litter, which is why the two numbers may not drift apart.

Being out of room is the verdict, never a reason to relocate. A scratch creation that
fails for room says the filesystem the real install targets has none, which is the answer.
Only conditions that make the repo unusable as a host at all fall back. Out of room covers
ENOSPC and EDQUOT together, since a per-user quota is how a managed host says the same
thing and only one of them is spelled ENOSPC.

The message stops naming one budget and one filesystem. EXIT_NO_SPACE now sends the
operator to check the free bytes (df -h) and the free file count (df -i), on the
checkout's filesystem and on the package cache's: the install writes both and they need not
be the same one, so a full cache raises the same errno while the scratch's own filesystem
looks healthy. The same correction lands on the two sibling refusals on this button,
could not stage the dependency preflight and could not stage the sync runner in
worktree_ops.py, which both sent the operator to bytes alone on a TMPDIR ENOSPC.
Their scratch dirs stay in TMPDIR: they hold a handful of files, not a dependency tree,
so residency is not their problem; the sentence was.

The spec moves with the code. docs/system-specs/modules/dev-fleet.md gains scratch
residency, the ignore gate, the sweeper and the no-fallback-on-no-room rule, so the next
change to this step does not reason from a stale contract.

What tests we did

test/test_dev_fleet_npm_preflight.py: 82 passed. black, isort, flake8 and mypy clean
on all three changed source files.

Reproduced first, then verified. Before: preflight: not enough room ... alongside
npm error code ENOSPC, with df -h /tmp showing 33 GiB free and df -i /tmp showing
55 175 slots against 59 586 needed. After: the probe consumes 0 tmpfs inodes.

New coverage, each pinned by a mutation run that shows the test fails when the property is
removed (the gate, the conservative reading of an unanswerable check-ignore, the sweep
call, its prefix filter, its age window, and its ordering before creation):

  • the repo hosts the scratch when the checkout carries the rule, and falls back to
    TMPDIR when it does not
  • an unanswerable ignore question reads as not ignored
  • a rule that lost its trailing * does not satisfy the gate
  • the sweep still runs when the gate sends the probe to TMPDIR
  • a stale scratch is removed; one a running probe could own is not
  • nothing outside the scratch prefix is touched, and an unmarked look-alike carrying the
    prefix is never deleted; nor is one whose marker is a symlink
  • a symlink named like a scratch directory never destroys its target, labelled in its own
    docstring as an OUTCOME test, because mutation showed shutil.rmtree refuses a symlink
    by itself, so the follow_symlinks=False flags are not what saves the target; they stay
    because reading an mtime through a link answers the wrong staleness question
  • an unreadable parent does not raise
  • the sweep window exceeds every deadline the module declares, and by a wide margin rather
    than merely clearing it; mutation confirms all five variants red it -- a probe timeout
    raised past the window, one raised only enough to eat the margin, a shrunken window, a
    lengthened helper deadline, and a brand-new long deadline
  • out of room on the repo filesystem is the verdict for both ENOSPC and EDQUOT, stated
    under the condition that makes it apply
  • errno.EDQUOT is read with getattr so the module and its tests load on Windows, where
    it is not defined

One pre-existing red is unrelated:
test_dev_fleet_app.py::test_trusted_bin_pins_the_resolved_target_not_the_symlink fails
identically on pristine main at 04727ebe0f, from KIROCREW_DEVFLEET_BIN_GH in the host
environment.

Any other suggestions on the work

One residual is recorded rather than fixed here. When the ignore gate or an unwritable
checkout sends the probe to TMPDIR, nothing in the output says the rehearsal ran off
target, so a pass carries less than it appears to. That is a reporting change to the run's
detail line, not a fix to this diff, and folding it in on top of a converged review would
be the wrong trade. Filed as #11459 so the disclosure is tracked rather than only written
down -- two review lanes pointed at it independently.

Pattern harvest

Rule candidate: grep/semgrep on operator-facing out-of-room text

Pattern: a refusal raised by an out-of-room errno whose remedy names ONE budget.
ENOSPC is returned for free BYTES and for free FILE SLOTS (inodes), and a
memory-backed filesystem reaches its inode cap long before its bytes run short --
so "free up space" is unactionable exactly when it is most needed. A refusal
carrying an out-of-room errno should name both budgets (df -h and df -i), and
should not assert which filesystem ran out when the operation writes more than
one.

Generalizable rather than a one-off: the same sentence was found three times on
the same button. npm_preflight.EXIT_NO_SPACE is the one #11434 reported;
grep 'free space in the temporary directory' then turned up
worktree_ops.py staging the dependency preflight and staging the sync runner,
both sending the operator to bytes alone on a TMPDIR ENOSPC. All three are
fixed here. A rule is worth having because the next such message will be written
by someone reading one of these as the house style.

Second, weaker candidate, recorded not proposed: tempfile.mkdtemp() /
TemporaryDirectory() with no dir= inside code whose purpose is to REHEARSE or
MEASURE work that will happen on a different filesystem. The bare call takes
TMPDIR, which silently answers a question about the wrong filesystem. Hard to
express mechanically, because the intent -- rehearsal versus ordinary scratch --
is not in the syntax.

Fixes #11434

@chenmingwei23
chenmingwei23 requested review from a team and cixuuz September 17, 2026 02:25
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Intent: Make Dev Fleet's Pull + Build installability probe rehearse the real npm ci on the filesystem that install writes to, so it cannot be starved by unrelated work elsewhere on the host, and make its out-of-room message name both budgets that raise ENOSPC instead of only free bytes.
Not a goal: reclaiming or policing the temporary directory, changing what the probe verifies, or touching any other Dev Fleet step.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 12e29ca78b1a1c0d30aa69ef774eb9e855c5f508 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify headroom: the probe's scratch coexists with website/node_modules, so a tight checkout can now refuse preflight where the delete-first real install would fit.

[DESIGN-REVIEWED] 12e29ca

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 12e29ca78b1a1c0d30aa69ef774eb9e855c5f508 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The sweeper is a recursive delete in the operator's checkout root — verify the marker+age gate once on a real fleet host with a concurrent Pull + Build.

What this change ships

Inventory (10 items) — 10 justified

Intent: FIX — Pull + Build refused as "out of space" with 33 GiB free because the install rehearsal ran on a tmpfs whose inodes, not bytes, were exhausted; the rehearsal moves onto the filesystem the real install writes to and the refusal names both budgets. Provenance is a written reproduction with measured numbers (55 175 free inodes vs 59 586 needed), and the fix sits at cause level (rehearsal on the wrong filesystem), not at the message.

  1. The lockfile rehearsal now installs inside the checkout root instead of TMPDIR — justified
  2. Running out of room now ends the probe with that verdict instead of retrying in TMPDIR — justified
  3. The repo hosts the scratch only when git check-ignore confirms the name is hidden; otherwise TMPDIR as before — justified
  4. The scratch name is git-ignored in the checkout root — justified
  5. Abandoned scratch dirs older than 6 h are swept before each probe, gated on an ownership marker — justified
  6. A per-user quota (EDQUOT) now classifies as out-of-room in both the errno guard and npm-output parsing — justified
  7. The out-of-room refusal now sends the operator to df -h AND df -i, on checkout and cache filesystems — justified
  8. The two sibling staging-failure messages in worktree_ops.py get the same correction (0 bytes-only siblings remain; grepped free space across src/) — justified
  9. dev-fleet.md spec gains the residency, gate, sweeper and no-fallback rules — justified
  10. Spawn-audit allowlist entry for the new read-only git check-ignore spawn — justified

Duplication checked: agent_scratch.py and mcp_gateway/backend_tmp.py both do marker-plus-sweep scratch containment, but both root at the data home and reclaim on process liveness — neither can place a scratch on the checkout's filesystem, which is the property this fix exists for, so item 5 is not a second spelling. The 6 h window exceeds the summed declared deadlines (1320 s), and a test reads those from source.

[FIRST-PRINCIPLES-REVIEWED] 12e29ca

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 12e29ca78b1a1c0d30aa69ef774eb9e855c5f508 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 12e29ca

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 12e29ca78b1a1c0d30aa69ef774eb9e855c5f508: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 12e29ca78b1a1c0d30aa69ef774eb9e855c5f508 — this comment is updated in place on each push.

Review details

No package.json or .npmrc at the repo root, so npm ci in a root-level scratch dir resolves no workspace/config ancestor that would differ from website/ — the rehearsal-accuracy claim holds. The rest of the change (fallback control flow, sweep marker/age/symlink guards, errno set parity, BENIGN_SPAWNS addition for the read-only trusted-git check-ignore) matches its tests and touches no blocking: true rule.

No findings.

[OPUS-REVIEWED] 12e29ca

Verdict parsed from the review's SHA-scoped output markers for commit 12e29ca78b1a1c0d30aa69ef774eb9e855c5f508.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 12e29ca78b1a1c0d30aa69ef774eb9e855c5f508: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/preflight-scratch-inode branch from 707ef01 to 1b170a2 Compare September 17, 2026 02:33
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: shared out-of-room errno set read by both the classifier and the fallback guard

  • Disk-quota exhaustion bypasses the target-filesystem probe -- fixed in 1b170a2.

Legitimate, and the reachable path is exactly as stated. _scratch_parent returns the repo path whenever it is a writable directory, so parent is not None; a quota-exhausted checkout raises OSError with errno EDQUOT; the guard tested only ENOSPC, so control fell through to the TMPDIR retry and the probe certified a filesystem the install will never write to, reporting success. _os_error_code had the same one-errno gap, so even the classified path would have called it EXIT_FAILED rather than out-of-room.
Fixed as the finding asks, in both places it names, through one shared _OUT_OF_ROOM_ERRNOS = (ENOSPC, EDQUOT) that the classifier and the fallback guard both read -- so the two cannot disagree about whether a condition is the target filesystem's own answer. The set is built by getattr lookup because EDQUOT is absent on some platforms.
This round introduced the guard, so the gap is self-added rather than pre-existing at this site; _os_error_code's ENOSPC-only mapping predated it but became newly load-bearing, which is why both moved together.
Coverage: the out-of-room verdict test is now parametrised over ENOSPC and EDQUOT and asserts no TMPDIR retry for either, plus one test pinning that the classifier and the guard read the same set.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/preflight-scratch-inode branch from 1b170a2 to 9fccfc6 Compare September 17, 2026 02:59
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/preflight-scratch-inode branch from 9fccfc6 to 8c012c7 Compare September 17, 2026 03:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Pushed 8c012c78d9 -- and the remaining red is main-owned

What changed in this revision. GPT's third finding was correct and is this PR's own
bug class arriving from another side: the probe's install writes to TWO filesystems --
the scratch directory and the package cache the retrieval populates -- and they need not
be the same one. A full cache raises the same ENOSPC, so an operator who only checks
where the scratch lives finds nothing wrong. The out-of-room explanation now names no
single filesystem and no single budget; it sends the operator to check bytes and file
slots on both. A new test pins that, and the older test's overclaim list grew
the scratch filesystem.

test/test_dev_fleet_npm_preflight.py: 70 passed.

The three failing Backend Tests shards are not from this diff. This branch touches
npm_preflight.py, its test file, and .gitignore. It touches nothing in the ledger or
the cloud engines. Main fails the identical tests on the identical shard numbers:

Shard Failing tests This PR main 04727ebe0 (run 35172978466)
Windows 3 test_fargate_launch_engine.py (2 tests, same names) 2 failed 2 failed
Windows 4 test_ledger_retention.py, test_ledger_kinds.py 46 failed failed
Windows 6 test_remove_slot_for_history_key.py (same test) 1 failed 1 failed
Linux 2, 3 same two families still running failed
Coverage Gate cascade of the above - failed

Main additionally reds two Linux shards and Coverage Gate, which this PR's run had not
reached yet at the time of reading -- so those are expected to red here too, from the
same cause.

Both breakages are already tracked, with fixes open. Ledger entry validation from
#11317 refuses writes three test files still make without agent/actor: issue #11443,
fix PRs #11445 and #11441. The Fargate engine drifted from the widened LaunchEngine
protocol: PRs #11442, #11432, #11426, #11422.

Nothing in this PR can clear those. It will go green on a rebase once they land; the
review lanes are the gates that are actually measuring this diff.

@chenmingwei23
chenmingwei23 force-pushed the fix/preflight-scratch-inode branch from 8c012c7 to d0561ea Compare September 17, 2026 03:52
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

d0561ea71b -- Design's CONCERNS is fixed; First Principles' advisory answered

All five review lanes passed on 8c012c78d9 (GPT and Opus: no blocking findings; First
Principles: PASS; UX: pass; Design: CONCERNS, advisory). Both advisories are answered
here rather than deferred, because one of them is a hole this PR itself opens.

Design CONCERNS -- accepted and fixed. The finding is correct: probe deletes its own
scratch in a finally, so what survives is a run that never reached it (SIGKILL, OOM,
reboot). In /tmp the host eventually age-cleans those. The repo root is cleaned by
nobody, and this PR is what put the scratch there and what git-ignores the name -- so
without a sweeper an abandoned node_modules tree accumulates there invisibly and
permanently. That converts a leak the host used to reclaim into one nothing reclaims.

_sweep_stale_scratch removes scratch siblings older than _SCRATCH_STALE_SECS, and
_make_scratch calls it BEFORE creating the new directory. The order is the point: the
room an abandoned tree holds is charged to the same byte and file budgets the incoming
install is measured against, so on a filesystem that litter has filled, the sweep is what
lets the probe run at all. If room is still short afterwards, that scarcity is the target
filesystem's real answer and the probe reports it.

Three properties keep the sweep from being its own hazard, each pinned by a test that a
mutation run proves has teeth (remove the call, the prefix filter, or the age check and a
test goes red):

  • It takes no lock, so the age window is the concurrency guard. A live probe's scratch
    is bounded by probe(timeout=...) plus its fixed-timeout helpers -- under twenty minutes
    at the default -- so the six-hour window cannot reach a concurrent Pull + Build. The
    margin is deliberately far wider than the bound: sweeping too late costs delay, sweeping
    too early breaks another operator's in-flight verification.
  • It is name-scoped. It runs in the repo root, so a loose filter would delete the
    user's work. Only _SCRATCH_PREFIX names, and only directories.
  • It is best-effort. Every failure is swallowed -- an unreadable parent, a racing
    sweep, an undeletable tree. Housekeeping must never be why a verification does not run.

One test is labelled honestly as an OUTCOME test in its own docstring: a symlink named
like a scratch directory never destroys its target, but mutation showed that removing the
follow_symlinks=False flags leaves it green, because shutil.rmtree refuses a symlink
argument by itself. The flags stay for a different reason, now stated where they are: read
through a link, a fresh link to an old tree and an old link to a fresh one both give the
wrong staleness answer.

First Principles advisory -- answered, no code change. The ask was to verify on a
second host that the sync runner may write in the checkout root, since a permission
refusal silently falls back to TMPDIR and reinstates the old budget. That fallback is
deliberate and documented on _scratch_parent (a read-only checkout still gets a probe),
and it is _scratch_parent's writability test plus _make_scratch's errno split that keep
it from firing on the case that matters: being OUT OF ROOM never falls back, because that
is the probe's answer. The residual honesty gap the advisory points at is real but
separate -- when the fallback does fire, nothing in the output says the rehearsal ran off
target. That is a reporting change, not a fix to this diff, and it is not being folded in
on top of a converged review.

Gates on this revision: black, isort, flake8, mypy clean; test_dev_fleet_npm_preflight.py
77 passed. Diff is +181 lines with no deletions and touches nothing outside this module and
its test file.

@chenmingwei23
chenmingwei23 force-pushed the fix/preflight-scratch-inode branch from d0561ea to f77f1d4 Compare September 17, 2026 04:11
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision 12e29ca78b1a1c0d30aa69ef774eb9e855c5f508 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 30a8c3b: The actor who could plant the trigger (a git-tracked dir at checkout root bearing this module private scratch prefix and private marker filename) already holds write access to that checkout and can corrupt it directly, so the sweep path grants nothing new; the module never commits such an artifact (mkdtemp, git-ignored, finally-removed); and any content lost is git-tracked and recoverable from the object store, so it is not unrecoverable.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 30a8c3ba34d6a593a4e8db5dc03ade6a8cc5f6ba.

The actor who could plant the trigger (a git-tracked dir at checkout root bearing this module private scratch prefix and private marker filename) already holds write access to that checkout and can corrupt it directly, so the sweep path grants nothing new; the module never commits such an artifact (mkdtemp, git-ignored, finally-removed); and any content lost is git-tracked and recoverable from the object store, so it is not unrecoverable.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/preflight-scratch-inode branch 2 times, most recently from 1159543 to 11f2d0d Compare September 18, 2026 01:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

CI note on head 11f2d0dfb8 (rebased onto main 727503fdfb).

The rebase cleared the four test/test_member_session_control.py reds that the previous head carried: main's curative fix landed as 2627b0407c (#11593), and those 38 tests pass on the rebased tree.

One red remains, Backend Tests (Windows) (2), and it is main's:

FAILED test/test_backend_cards.py::test_every_capability_set_is_classified_by_the_card
AssertionError: these capability sets are not classified in agent_sdk/backend_cards.py:
['ACP_BACKENDS_SELF_SERVED_ACP']

A pristine detached checkout of current main (0ead3cde97) fails the identical test with the identical assertion. The cause is a merge race: #11428 added the ACP_BACKENDS_SELF_SERVED_ACP set, #11431 added the guard that requires every set to be classified, and neither saw the other. It is tracked as #11634 with fix PR #11632.

This PR's diff is six files (.gitignore, docs/system-specs/modules/dev-fleet.md, dev_fleet/npm_preflight.py, dev_fleet/worktree_ops.py, test/test_dev_fleet_npm_preflight.py, test/test_spawn_audit.py). It touches neither agent_sdk/backend_cards.py nor any file that defines a capability set, so there is nothing to fix here and rebasing again would not help while main itself carries the break.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Second main-owned red on head 11f2d0dfb8, Backend Tests (Windows) (1):

FAILED test/test_acp_launch_goldens.py  (10 of 14)
TypeError: _stub_common.<locals>.<lambda>() got an unexpected keyword argument
'forward_ssh_auth_sock'

A pristine detached checkout of current main (fd1173643c) fails the identical 10 tests with the identical TypeError, so this is not a property of this branch. The kwarg was introduced by 727503fdfb (#9131), which is the very main commit this branch was rebased onto: the test stub's lambda was not widened to accept it. Tracked as #11635.

Board accounting for this head, then: two reds, both main's and both tracked (#11634 with fix PR #11632, and #11635), plus Coverage Gate which is a fail-closed cascade of any red shard rather than a coverage finding of its own. Neither red names a file this PR's six-file diff contains.

Rebasing again is not the remedy for either one, because both breakages live on main's own tip.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition of the two verification requests on head 11f2d0dfb8 (Design PASS, First Principles PASS, Opus no findings, UX skipped as no user-facing surface).

Design: nothing must pin _SCRATCH_STALE_SECS (6 h) above probe()'s deadline. Correct, and today nothing does. The numbers are safe by a wide margin -- probe(timeout=900) plus fixed 60 s and 120 s helpers, so under twenty minutes against a 21600 s window -- and npm_preflight.py states the relationship in prose above the constant. But prose is not a guard: raising the probe timeout is a one-line change, and past 6 h the lockless sweep would delete a LIVE probe's scratch. The ownership marker does not help there, because a live probe's scratch is genuinely marked and genuinely ours; only the age check separates it from an abandoned one. So the two numbers need a test that reds when either moves without the other, computing the probe's maximum lifetime from the code's own constants rather than restating 900 as a literal.

First Principles: verify the checkout root and website/ share a filesystem. They do on this host -- both report device 66305, both resolve to the same mount. The assumption is load-bearing exactly as stated: the rehearsal installs into a scratch dir at the checkout root, while the real npm ci writes website/node_modules, so a separate mount under website/ would have the rehearsal measure a filesystem the real install never touches. That is the same failure class as the already-filed #11459 (a rehearsal that passes while certifying a different filesystem), so it belongs there rather than as a second independent report, and I have added it to that issue.

Neither item is going in as its own push. Two reds on this head are main's (#11634 with fix PR #11632, and #11635), so this branch needs one more rebase once those land; the _SCRATCH_STALE_SECS guard test rides along in that same push instead of spending a full CI round and re-rolling the non-deterministic review lanes for a test-only change.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
…ystem

Pull + Build rehearses the incoming lockfile with a real script-free install
before it touches the checkout, and that rehearsal ran in TMPDIR. On a host
whose TMPDIR is memory-backed the rehearsal answers about the wrong
filesystem: it exhausted that filesystem's fixed file-slot budget and
reported "not enough room ... free space in the temporary directory", so an
operator who checked exactly that saw 33 GiB free and read the refusal as
wrong. The real numbers were 55 175 free file slots against 59 586 needed.

The rehearsal now installs on the filesystem the real npm ci writes to, but
only when git check-ignore confirms the checkout hides the scratch name;
otherwise it falls back to TMPDIR as before, because an un-ignored checkout
would silently read dirty and fail Prune merged closed. Out of room at
scratch creation is the verdict and is never retried elsewhere. Quota
exhaustion classifies as out-of-room in the errno and npm-output paths
alike. Abandoned scratch trees older than _SCRATCH_STALE_SECS are swept
before the new one is created, and only directories carrying an ownership
marker written by this module are ever deleted, because the sweep is a
recursive delete in the operator's checkout root. A test derives every
deadline the module declares from its own source and requires the sweep
window to exceed their sum by a wide margin, so raising a timeout without
widening the window fails rather than deleting a live probe's scratch. The
refusal message names both budgets and both filesystems, and the two
staging-failure messages in worktree_ops.py get the same sentence.

Fixes #11434
@chenmingwei23
chenmingwei23 force-pushed the fix/preflight-scratch-inode branch from 11f2d0d to 12e29ca Compare September 18, 2026 02:37
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition of both verification requests on head 12e29ca78b. Design PASS, First Principles PASS, GPT no blocking findings, Opus no blocking findings, UX skipped as no user-facing surface. Neither item needs code, so nothing is pushed for them.

First Principles: verify the marker-plus-age gate on a real host with a concurrent Pull + Build. Done in this checkout root, driving the shipped _sweep_stale_scratch against three real directories at once: a live concurrent probe (marked, mtime now, holding a node_modules), an abandoned run (marked, aged past the 21600 s window), and an unmarked look-alike wearing the prefix and carrying a file.

before sweep: [killed-run, live-concurrent, not-mine]
after  sweep: [live-concurrent, not-mine]

live concurrent probe survived : True
abandoned run reclaimed        : True
unmarked look-alike survived   : True   (its file byte-identical)
git hides live-concurrent      : True
git hides not-mine             : True

Both guards were exercised in the direction that matters. The concurrent probe is marked, so only its youth saved it; the look-alike is aged, so only the missing marker saved it. git status was clean before and after.

Design: verify headroom, since the probe's scratch coexists with website/node_modules while the delete-first real install would fit. The premise is false, and it is the same premise raised on a951ed2fa8 under different wording, so here is the code rather than a restatement. The real install does not delete first. sync_runner.NodeModulesTransaction.__enter__ is one line:

os.rename(self.stash, self.backup)

_BACKUP_SUFFIX = ".kirocrew-sync-backup" appends to the original path, so the backup is a SIBLING on the same filesystem: the rename frees no bytes and no file slots. __exit__ removes it only when rc == 0, that is after npm ci has finished, because the transaction exists so a failed step can be rolled back.

So the real install's peak is the backup tree plus the tree being written, and the probe's peak is website/node_modules plus the scratch tree. Two trees either way. The probe is aligned with the real step, not above it, and the closing of my earlier proof's gap is exactly this equality: it is not that the probe's peak is acceptable, it is that the two peaks are the same number.

Read the other way, the arithmetic argues for this design. A rehearsal that peaked at one tree would pass on a checkout with 1.5 trees of room, and then the real two-tree install would fail after merge -- which is the outcome this preflight exists to prevent. Nothing is recorded in the spec for it, because there is no trade-off to record.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Green on 12e29ca78b1a1c0d30aa69ef774eb9e855c5f508, one squashed commit on main 433ad8b2b5.

checks       73 success, 11 skipped, 0 failure, 0 in progress
commit status PR Readiness = success
              "Eligible automated validation passed for this revision"
mergeable    MERGEABLE   (BLOCKED only by REVIEW_REQUIRED)

[GPT-REVIEWED]              12e29ca78b   no blocking findings
[OPUS-REVIEWED]             12e29ca78b   no findings
[DESIGN-REVIEWED]           12e29ca78b   PASS
[FIRST-PRINCIPLES-REVIEWED] 12e29ca78b   PASS
UX Review                                skipped, no user-facing surface

Every marker names this head rather than an earlier one, and all 18 backend shard rows are present and green -- an absent shard row is unmeasured, not passing, which is how a rebase can read as a clearance it has not earned.

Earlier reds on this branch are accounted for and none survive: the test_member_session_control failures were main's and cleared with #11593, test_backend_cards with #11632, and test_acp_launch_goldens with #11649. Coverage Gate is green now for the same reason it was red then -- it gates on the upstream coverage jobs and measures nothing itself.

One item is deliberately not in this PR and is tracked: #11459, where a probe that falls back to TMPDIR returns a pass certifying a filesystem the real install never writes to. Two independent review rounds arrived at it, and it now carries the second route to the same outcome (a separate mount under website/) with the st_dev evidence.

Not merging. This awaits a human.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@bolichen97
bolichen97 merged commit e6e5197 into main Sep 18, 2026
88 of 94 checks passed
@bolichen97
bolichen97 deleted the fix/preflight-scratch-inode branch September 18, 2026 04:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dev Fleet Pull + Build refuses for "not enough room" while the temporary directory has tens of GB free

2 participants