Skip to content

fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) - #497

Open
mengmengmengqiang wants to merge 1 commit into
Ephemeral-AI-Lab:mainfrom
tokenaissance:fix/460-join-grace-1s
Open

mengmengmengqiang wants to merge 1 commit into
Ephemeral-AI-Lab:mainfrom
tokenaissance:fix/460-join-grace-1s

Conversation

@mengmengmengqiang

@mengmengmengqiang mengmengmengqiang commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Rebased onto main after #500 — #496 was reworked and merged as #500 (69fedcac), so this is now a single commit on main (35d36575) and the file diff shows only this change.

Problem

#496 ends a normal unmount stalled at Join with the FUSE abort write, but it does so when the drain deadline (drain_wait, 5 s) expires. That deadline also covers requests, yet the join it then waits for follows a known detach: a healthy connection joins in milliseconds, while a connection kept alive by another mount namespace never joins (#426, #460). Paying 5 s for a wait that can only end in microseconds or never is the conservative part, not a necessary one.

Change

Once every request is quiescent, observe the receive-loop join for its own shorter grace, JOIN_WAIT = 1 s, instead of the remaining drain_wait. Requests keep the whole drain_wait; only the join is bounded shorter, and the abort of #496 then runs at ~1 s rather than ~5 s. Nothing else changes: the same abort write, the same second drain, the same Unmounted.

Measurements (same machine as #496: macOS 26.5.1, OrbStack 7.2.9, kernel 7.2.9-orbstack-00391, Docker 29.8.2)

Run n min p50 p90 max
normal unmount (5 s build) 20 20 ms 23 ms 46 ms 82 ms
normal unmount after 64 MB read (5 s build) 10 25 ms 32 ms 71 ms 71 ms
early-mount stall, drain_wait 5 s (#496) 20 5036 ms 5075 ms 5228 ms 5558 ms
early-mount stall, JOIN_WAIT 1 s (this PR) 20 1021 ms 1082 ms 1351 ms 1467 ms
normal unmount, JOIN_WAIT 1 s (this PR) 20 17 ms 23 ms 34 ms 44 ms

40/40 in the 1 s runs returned Unmounted and Closed. The tail after the deadline (abort + force drain + revoke + close) was 36–558 ms, p50 75 ms, in the 5 s build.

Why 1 s and not shorter: a throwaway 500 ms prototype (not in this PR) made a stalled unmount 519–565 ms, but one healthy unmount in that run took 374 ms — the normal tail moves under load, so 500 ms leaves ~1.3× margin. 1 s keeps ~2.7× over the slowest healthy join we saw and still cuts ~4 s per stall.

The signal to count is the grace path: abort_attempted in the follow-up event (draft: tokenaissance#2) is true exactly when the 1 s expired and #496's abort ran.

Cost estimate for the scenario matrix

The 10 scripted scenarios contain 211 workspace.unmount calls; #466's failing runs reported roughly 66 failed steps in one matrix pass. At ~4 s saved per stalled unmount, one full matrix run saves about 1–5 minutes, and a single agent call goes from ~5.1 s to ~1.1 s.

Tests

stalled_unmount now also asserts the stalled unmount returns in under 3 s (it took 1.12 s with this change, 3.23 s with the full wait); that assertion would fail on #496's build. The other daemon binaries are unchanged by this diff.

Production LOC: 102831 -> 102833 (delta +2): core 0, engine +2, api 0.

Risk and rollback

The grace applies only after requests are quiescent, so it cannot cut off a running request; the worst case for a healthy-but-slow join is that it is treated as stalled and the abort ends a connection whose mount is already gone. Reverting is one line (JOIN_WAIT back to drain_wait), and the 5 s behaviour is #496.

Visual evidence walk-through

The bug/fix board already shows both fixes: #496's recovery (now merged as #500) and this PR's grace. A one-picture guide first (what the bug is, what #496/#497 change, how to read the rest), then two context pictures (architecture, sequence), then the evidence: the fix board, the latency panel that isolates this PR's 1 s grace, and the options board that places it next to the alternatives. All measured on macOS 26.5.1, OrbStack (kernel 7.2.9-orbstack-00391), Docker 29.8.2.

Guide: what changes, and how to read these pictures

Start here. Left: the bug in one column. Middle: what #496 and #497 change, and what each costs. Right: how to read the rest of the pictures (lanes, numbered unmount steps, colors). All numbers in the middle column are measured on the machine named below.

LayerFS architecture

Where this change lives. Core crates (content, history, storage) import no engine, FUSE, Docker or JavaScript code; the daemon/FUSE pair is in the adapter band. JOIN_WAIT is a timing constant inside that band — no architecture or dependency change.

Interaction sequence

One bash call from create to unmount. The grace applies inside step 13: once requests are quiescent, the loop join is observed for 1 s instead of the remaining drain wait.

Join wedge: cause and fix with #496 + #497

The left lane is the wedge that #496 recovers from; the right lane is this PR's JOIN_WAIT = 1 s: requests keep the usual 5 s drain_wait, then the receive-loop join is observed for only 1 s. Early-mount stall: n=20, 1021–1467 ms, p50 1082 (~1.08 s, down from ~5.1 s with #496 alone).

Cause and fix, latency lane

The latency panel isolates the grace: a fast path (joined → return) and a stall path (grace expires → #496's abort). Normal unmount with the grace n=20 at 17–44 ms; without it n=20 at 20–82 ms.

Fix options: recovery, latency, root cause

The latency column is this PR, shown next to recovery, root-cause and rejected options. (The private-mount-namespace card is a separate experiment, now open as draft #499; its 22 ms is the prototype's early-mount measurement, not part of this PR.) The 1 s value leaves margin over the 374 ms healthy unmount measured under load, which is why 500 ms was rejected.

@mengmengmengqiang

Copy link
Copy Markdown
Contributor Author

Follow-up note (no action): after this PR the two waits are already separate constants — DRAIN_WAIT (5 s) still bounds the request wait, JOIN_WAIT (1 s) bounds the join observation. A future refactor must keep it that way: the abort fires on the request deadline, and the grace only shortens observation. (Tracked in the fork as tokenaissance#4, now closed in favour of this note.)

mengmengmengqiang added a commit to tokenaissance/layerfs that referenced this pull request Oct 28, 2026
…Ephemeral-AI-Lab#426, Ephemeral-AI-Lab#460)

A normal unmount whose connection outlives its mount waits for the receive
loops to join. A healthy join follows the detach in milliseconds, but a
connection kept alive by another mount namespace never joins, so the wait
spends the whole drain_wait (5 s) before the Join stop is reported and the
abort of Ephemeral-AI-Lab#496 can end it.

Observe the join for its own shorter grace (JOIN_WAIT = 1 s) once every
request is quiescent. Requests keep the whole drain_wait; only the loop
join, which follows a known detach and can only finish in microseconds or
never, is bounded shorter. A stalled unmount now ends in about 1.1 s
instead of about 5.1 s; a healthy one is unchanged.

Measurements on macOS 26.5.1 / OrbStack 7.2.9, the machine of Ephemeral-AI-Lab#496:
- normal unmount: n=20, min 20 ms, p50 23 ms, p90 46 ms, max 82 ms
- early-mount stall with drain_wait 5 s: n=20, 5036-5558 ms, p50 5075
- early-mount stall with JOIN_WAIT 1 s: n=20, 1021-1467 ms, p50 1082
- normal unmount with JOIN_WAIT 1 s: n=20, 17-44 ms, p50 23
- a 500 ms prototype (not this change) once saw a healthy join at 374 ms,
  which is why the grace is 1 s and not shorter.

The stalled_unmount test also asserts the stalled unmount returns in under
3 s: it took 1.12 s with this change and 3.23 s with the full wait.

Production LOC: 102831 -> 102833 (delta +2)
  core 38395 -> 38395 (0), engine 61777 -> 61779 (+2), api 2659 (0).
@mengmengmengqiang mengmengmengqiang changed the title fix(fuse): bound the receive-loop join by a 1 s grace (stacked on #496) fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant