Repository navigation
fix(daemon): end a stalled normal unmount at Join with its own abort write (#426, #460) - #496
mengmengmengqiang wants to merge 1 commit into
Conversation
…write (Ephemeral-AI-Lab#426, Ephemeral-AI-Lab#460) A normal unmount can detach successfully and still stop at Join: when another mount of the same superblock outlives the detach, the kernel never ends the connection, its receive loops stay blocked, and the entry was kept as Retained at Join with no way out but a forced unmount — or, where no abort control is bound, no way out at all (Ephemeral-AI-Lab#426, Ephemeral-AI-Lab#460). The unmount now makes the same single abort write a forced unmount makes, on the already-detached connection, and drains again. It then completes as Unmounted, revokes and closes, so the stalled connection ends where the unmount asked. Only a refused or short write, or a second stop, keeps the custody, exactly as any other stop does; such an entry is not forceable either, because its abort control has been used. - layerfs-fuse: Undrained::end_stalled, one abort_detached write then force_drain, only for a Join stop. - layerfs-daemon: unmount_kernel calls it on a Join stop; a second stop keeps the evidence as before. - tests: stalled_unmount now asserts the normal unmount ends the stall itself (the copy of the mount answers ENOENT) and keeps no custody. - docs: native mount session and forced teardown describe the automatic abort; the SDK skill and the unmount comment no longer send callers to --force for this case. Tests: - cargo test -p layerfs-daemon --test stalled_unmount --release in a privileged Linux container: passes; the parent's code keeps custody, so the test fails without this change. - also ran mounted_drain, fp21_detach_stages, control_root, mounted_fusectl, forced_unmount and forced_unavailable; all pass except the environment bound fp31 reclaim observation in mounted_drain (container cgroup memory reclaim, unrelated to unmount). Production LOC: 102763 -> 102777 (delta +14) core 38395 -> 38395 (0), engine 61709 -> 61723 (+14), api 2659 (0).
|
One review note with data: the 5 s in this PR is conservative, not necessary. I measured the same machine (macOS 26.5.1, OrbStack 7.2.9, kernel
Three points:
For the scenario matrix: the 10 scenarios contain 211 I put the change in #497 ( |
Fixes the Join wedge for a normal unmount: the daemon now makes the same single abort write a forced unmount makes when the drain stops at
Join, and drains again, so the unmount completes instead of keeping custody.Refs #426, #460, #466.
Problem
A normal unmount can detach successfully and still stop at
Join: an external mount-namespace holder keeps a reference to the Workspace mount alive (an engine or VM agent outside the Sandbox), so the kernel never ends the connection and the receive loops stay blocked (#426, #460). #481 made that custody forceable; a caller that does not passforcestill gets the refusal, and where no abort control is bound nothing can end it.Reproduced on macOS 26.5.1 / OrbStack, kernel
7.2.9-orbstack-00391, Docker 29.8.2 (API 1.54), at582bc786:sandbox.create()→ mount immediately → 1.5 s → plainworkspace.unmount→Failed(Join) after 5.0–5.5 s, 6/6; onlyworkspace.unmount(ws, true, true)then closes the Sandbox (6/6). Mounted 2.5 s after the Sandbox starts, plain unmount passes (3/3).Change
On a
Joinstop the unmount makes oneabort_detachedwrite on the already-detached connection and calls the same forced drain; the unmount then revokes, closes and repliesUnmounted.layerfs-fuse:Undrained::end_stalled, one abort write thenforce_drain, only for aJoinstop.layerfs-daemon:unmount_kernelcalls it; a refused or short write, or a second stop, keeps the custody exactly as before.stalled_unmounttest: asserts the normal unmount ends the stall itself and the copied mount answers ENOENT; no custody is retained. It fails on the parent, which keepsRetained { stage: Join }.--forcestill exists for forced teardown of a live connection; for a still-stalled custody whose abort control is unusable the entry is not forceable, as the docs now say.After
Same six runs with this branch's daemon on the same machine: plain unmount
Unmounted6/6 (5.05–5.08 s: the drain deadline, then the abort),sandbox.closeClosed6/6, no force needed. Delayed-mount control: 3/3Unmounted.Cost: the completion waits out the 5 s
DRAIN_WAITbefore the abort; that deadline is unchanged.Tests
cargo test -p layerfs-daemon --test stalled_unmount --releasein a privileged Linux container: passes; the parent fails it.mounted_drain,fp21_detach_stages,control_root,mounted_fusectl,forced_unmount,forced_unavailable. All pass exceptmounted_drain's fp31 reclaim observation, which is bound to container cgroup memory reclaim and unrelated to unmount (it fails before any unmount).Production LOC: 102763 -> 102777 (delta +14): core 0, engine +14, api 0.
Visual evidence walk-through
A one-picture guide first (what the bug is, what this PR changes, how to read the rest), then two context pictures: the architecture and one unmount's sequence. Then the evidence for this PR: the bug/fix comparison and the recovery panel. All measured on macOS 26.5.1, OrbStack (kernel
7.2.9-orbstack-00391), Docker 29.8.2.Start here. Left: the bug in one column. Middle: what #496 and #497 change, and what each costs. Right: how to read the rest of the pictures (lanes, numbered unmount steps, colors). All numbers in the middle column are measured on the machine named below.
Where this change lives. Core crates (content, history, storage) import no engine, FUSE, Docker or JavaScript code. The daemon/FUSE pair and the JS SDK/host sit in the adapter band; the fix in this PR is inside that band, and the dependency direction of everything else is unchanged.
The sequence shows where the Join stop sits:
workspace.unmountdetaches (step 12), the daemon drains and joins the FUSE receive loops (step 13), then revokes and closes (step 14). The wedge is step 13.The board is drawn for the stacked pair (#496 + #497): the left lane is the wedge, the right lane is #496's abort write with #497's 1 s join grace on top. On 582bc78 a plain
workspace.unmountstalled atJoinin 6/6 runs (Failedwith aJoincustody; the first v1.1.0 run wedged, see #466). With this PR's single abort write alone: 6/6 complete at ~5.1 s (the 5 s drain deadline), 0/6 wedged; with #497 stacked on it: 20/20 at ~1.08 s.The recovery panel replays the fix as a message sequence: before, the holder we could not identify keeps a reference to the mount, so the superblock outlives the detach and the receive loops stay blocked in
read()(the entryRetainedatJoin); after, oneabort_detachedwrite wakes them, the second drain succeeds, thenrevoke→Close, and the caller getsUnmounted.The wedge was in step 13: the loop join could wait on a blocked
read()with nothing to wake it. The singleabort_detachedwrite on the detached connection wakes the loops, so the second drain reachesRevoke→Closeand the caller getsUnmountedinstead ofFailedwith aJoincustody.