Repository navigation
fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) - #497
Open
mengmengmengqiang wants to merge 1 commit into
Open
mengmengmengqiang wants to merge 1 commit into
mengmengmengqiang wants to merge 1 commit into
Conversation
mengmengmengqiang
pushed a commit
to tokenaissance/layerfs
that referenced
this pull request
Oct 28, 2026
This was referenced Oct 28, 2026
Contributor
Author
|
Follow-up note (no action): after this PR the two waits are already separate constants — |
mengmengmengqiang
added a commit
to tokenaissance/layerfs
that referenced
this pull request
Oct 28, 2026
…Ephemeral-AI-Lab#426, Ephemeral-AI-Lab#460) A normal unmount whose connection outlives its mount waits for the receive loops to join. A healthy join follows the detach in milliseconds, but a connection kept alive by another mount namespace never joins, so the wait spends the whole drain_wait (5 s) before the Join stop is reported and the abort of Ephemeral-AI-Lab#496 can end it. Observe the join for its own shorter grace (JOIN_WAIT = 1 s) once every request is quiescent. Requests keep the whole drain_wait; only the loop join, which follows a known detach and can only finish in microseconds or never, is bounded shorter. A stalled unmount now ends in about 1.1 s instead of about 5.1 s; a healthy one is unchanged. Measurements on macOS 26.5.1 / OrbStack 7.2.9, the machine of Ephemeral-AI-Lab#496: - normal unmount: n=20, min 20 ms, p50 23 ms, p90 46 ms, max 82 ms - early-mount stall with drain_wait 5 s: n=20, 5036-5558 ms, p50 5075 - early-mount stall with JOIN_WAIT 1 s: n=20, 1021-1467 ms, p50 1082 - normal unmount with JOIN_WAIT 1 s: n=20, 17-44 ms, p50 23 - a 500 ms prototype (not this change) once saw a healthy join at 374 ms, which is why the grace is 1 s and not shorter. The stalled_unmount test also asserts the stalled unmount returns in under 3 s: it took 1.12 s with this change and 3.23 s with the full wait. Production LOC: 102831 -> 102833 (delta +2) core 38395 -> 38395 (0), engine 61777 -> 61779 (+2), api 2659 (0).
mengmengmengqiang
force-pushed
the
fix/460-join-grace-1s
branch
from
October 28, 2026 21:39
0bfd085 to
9a03429
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto
mainafter #500 — #496 was reworked and merged as #500 (69fedcac), so this is now a single commit onmain(35d36575) and the file diff shows only this change.Problem
#496 ends a normal unmount stalled at
Joinwith the FUSE abort write, but it does so when the drain deadline (drain_wait, 5 s) expires. That deadline also covers requests, yet the join it then waits for follows a known detach: a healthy connection joins in milliseconds, while a connection kept alive by another mount namespace never joins (#426, #460). Paying 5 s for a wait that can only end in microseconds or never is the conservative part, not a necessary one.Change
Once every request is quiescent, observe the receive-loop join for its own shorter grace,
JOIN_WAIT = 1 s, instead of the remainingdrain_wait. Requests keep the wholedrain_wait; only the join is bounded shorter, and the abort of #496 then runs at ~1 s rather than ~5 s. Nothing else changes: the same abort write, the same second drain, the sameUnmounted.Measurements (same machine as #496: macOS 26.5.1, OrbStack 7.2.9, kernel
7.2.9-orbstack-00391, Docker 29.8.2)drain_wait5 s (#496)JOIN_WAIT1 s (this PR)JOIN_WAIT1 s (this PR)40/40 in the 1 s runs returned
UnmountedandClosed. The tail after the deadline (abort + force drain + revoke + close) was 36–558 ms, p50 75 ms, in the 5 s build.Why 1 s and not shorter: a throwaway 500 ms prototype (not in this PR) made a stalled unmount 519–565 ms, but one healthy unmount in that run took 374 ms — the normal tail moves under load, so 500 ms leaves ~1.3× margin. 1 s keeps ~2.7× over the slowest healthy join we saw and still cuts ~4 s per stall.
The signal to count is the grace path:
abort_attemptedin the follow-up event (draft: tokenaissance#2) is true exactly when the 1 s expired and #496's abort ran.Cost estimate for the scenario matrix
The 10 scripted scenarios contain 211
workspace.unmountcalls; #466's failing runs reported roughly 66 failed steps in one matrix pass. At ~4 s saved per stalled unmount, one full matrix run saves about 1–5 minutes, and a single agent call goes from ~5.1 s to ~1.1 s.Tests
stalled_unmountnow also asserts the stalled unmount returns in under 3 s (it took 1.12 s with this change, 3.23 s with the full wait); that assertion would fail on #496's build. The other daemon binaries are unchanged by this diff.Production LOC: 102831 -> 102833 (delta +2): core 0, engine +2, api 0.
Risk and rollback
The grace applies only after requests are quiescent, so it cannot cut off a running request; the worst case for a healthy-but-slow join is that it is treated as stalled and the abort ends a connection whose mount is already gone. Reverting is one line (
JOIN_WAITback todrain_wait), and the 5 s behaviour is #496.Visual evidence walk-through
The bug/fix board already shows both fixes: #496's recovery (now merged as #500) and this PR's grace. A one-picture guide first (what the bug is, what #496/#497 change, how to read the rest), then two context pictures (architecture, sequence), then the evidence: the fix board, the latency panel that isolates this PR's 1 s grace, and the options board that places it next to the alternatives. All measured on macOS 26.5.1, OrbStack (kernel
7.2.9-orbstack-00391), Docker 29.8.2.Start here. Left: the bug in one column. Middle: what #496 and #497 change, and what each costs. Right: how to read the rest of the pictures (lanes, numbered unmount steps, colors). All numbers in the middle column are measured on the machine named below.
Where this change lives. Core crates (content, history, storage) import no engine, FUSE, Docker or JavaScript code; the daemon/FUSE pair is in the adapter band.
JOIN_WAITis a timing constant inside that band — no architecture or dependency change.One bash call from create to unmount. The grace applies inside step 13: once requests are quiescent, the loop join is observed for 1 s instead of the remaining drain wait.
The left lane is the wedge that #496 recovers from; the right lane is this PR's
JOIN_WAIT = 1 s: requests keep the usual 5 sdrain_wait, then the receive-loop join is observed for only 1 s. Early-mount stall: n=20, 1021–1467 ms, p50 1082 (~1.08 s, down from ~5.1 s with #496 alone).The latency panel isolates the grace: a fast path (joined → return) and a stall path (grace expires → #496's abort). Normal unmount with the grace n=20 at 17–44 ms; without it n=20 at 20–82 ms.
The latency column is this PR, shown next to recovery, root-cause and rejected options. (The private-mount-namespace card is a separate experiment, now open as draft #499; its 22 ms is the prototype's early-mount measurement, not part of this PR.) The 1 s value leaves margin over the 374 ms healthy unmount measured under load, which is why 500 ms was rejected.