Skip to content

fix(daemon): end a stalled normal unmount at Join with its own abort write (#426, #460) - #496

Open
mengmengmengqiang wants to merge 1 commit into
Ephemeral-AI-Lab:mainfrom
tokenaissance:fix/460-stalled-unmount-abort
Open

mengmengmengqiang wants to merge 1 commit into
Ephemeral-AI-Lab:mainfrom
tokenaissance:fix/460-stalled-unmount-abort

Conversation

@mengmengmengqiang

@mengmengmengqiang mengmengmengqiang commented Oct 28, 2026 •

Copy link
Copy Markdown

Fixes the Join wedge for a normal unmount: the daemon now makes the same single abort write a forced unmount makes when the drain stops at Join, and drains again, so the unmount completes instead of keeping custody.

Refs #426, #460, #466.

Problem

A normal unmount can detach successfully and still stop at Join: an external mount-namespace holder keeps a reference to the Workspace mount alive (an engine or VM agent outside the Sandbox), so the kernel never ends the connection and the receive loops stay blocked (#426, #460). #481 made that custody forceable; a caller that does not pass force still gets the refusal, and where no abort control is bound nothing can end it.

Reproduced on macOS 26.5.1 / OrbStack, kernel 7.2.9-orbstack-00391, Docker 29.8.2 (API 1.54), at 582bc786: sandbox.create() → mount immediately → 1.5 s → plain workspace.unmount → Failed (Join) after 5.0–5.5 s, 6/6; only workspace.unmount(ws, true, true) then closes the Sandbox (6/6). Mounted 2.5 s after the Sandbox starts, plain unmount passes (3/3).

Change

On a Join stop the unmount makes one abort_detached write on the already-detached connection and calls the same forced drain; the unmount then revokes, closes and replies Unmounted.

  • layerfs-fuse: Undrained::end_stalled, one abort write then force_drain, only for a Join stop.
  • layerfs-daemon: unmount_kernel calls it; a refused or short write, or a second stop, keeps the custody exactly as before.
  • stalled_unmount test: asserts the normal unmount ends the stall itself and the copied mount answers ENOENT; no custody is retained. It fails on the parent, which keeps Retained { stage: Join }.
  • Docs: native mount session, forced teardown, the SDK skill reference and the unmount comment.

--force still exists for forced teardown of a live connection; for a still-stalled custody whose abort control is unusable the entry is not forceable, as the docs now say.

After

Same six runs with this branch's daemon on the same machine: plain unmount Unmounted 6/6 (5.05–5.08 s: the drain deadline, then the abort), sandbox.close Closed 6/6, no force needed. Delayed-mount control: 3/3 Unmounted.

Cost: the completion waits out the 5 s DRAIN_WAIT before the abort; that deadline is unchanged.

Tests

  • cargo test -p layerfs-daemon --test stalled_unmount --release in a privileged Linux container: passes; the parent fails it.
  • Also ran mounted_drain, fp21_detach_stages, control_root, mounted_fusectl, forced_unmount, forced_unavailable. All pass except mounted_drain's fp31 reclaim observation, which is bound to container cgroup memory reclaim and unrelated to unmount (it fails before any unmount).

Production LOC: 102763 -> 102777 (delta +14): core 0, engine +14, api 0.

Visual evidence walk-through

A one-picture guide first (what the bug is, what this PR changes, how to read the rest), then two context pictures: the architecture and one unmount's sequence. Then the evidence for this PR: the bug/fix comparison and the recovery panel. All measured on macOS 26.5.1, OrbStack (kernel 7.2.9-orbstack-00391), Docker 29.8.2.

Guide: what changes, and how to read these pictures

Start here. Left: the bug in one column. Middle: what #496 and #497 change, and what each costs. Right: how to read the rest of the pictures (lanes, numbered unmount steps, colors). All numbers in the middle column are measured on the machine named below.

LayerFS architecture: core crates, the engine band, and the adapters

Where this change lives. Core crates (content, history, storage) import no engine, FUSE, Docker or JavaScript code. The daemon/FUSE pair and the JS SDK/host sit in the adapter band; the fix in this PR is inside that band, and the dependency direction of everything else is unchanged.

Interaction sequence: one bash call from create to unmount

The sequence shows where the Join stop sits: workspace.unmount detaches (step 12), the daemon drains and joins the FUSE receive loops (step 13), then revokes and closes (step 14). The wedge is step 13.

Join wedge: before and after the abort write

The board is drawn for the stacked pair (#496 + #497): the left lane is the wedge, the right lane is #496's abort write with #497's 1 s join grace on top. On 582bc78 a plain workspace.unmount stalled at Join in 6/6 runs (Failed with a Join custody; the first v1.1.0 run wedged, see #466). With this PR's single abort write alone: 6/6 complete at ~5.1 s (the 5 s drain deadline), 0/6 wedged; with #497 stacked on it: 20/20 at ~1.08 s.

Cause and fix, recovery lane

The recovery panel replays the fix as a message sequence: before, the holder we could not identify keeps a reference to the mount, so the superblock outlives the detach and the receive loops stay blocked in read() (the entry Retained at Join); after, one abort_detached write wakes them, the second drain succeeds, then revoke → Close, and the caller gets Unmounted.

The wedge was in step 13: the loop join could wait on a blocked read() with nothing to wake it. The single abort_detached write on the detached connection wakes the loops, so the second drain reaches Revoke → Close and the caller gets Unmounted instead of Failed with a Join custody.

…write (Ephemeral-AI-Lab#426, Ephemeral-AI-Lab#460)

A normal unmount can detach successfully and still stop at Join: when
another mount of the same superblock outlives the detach, the kernel never
ends the connection, its receive loops stay blocked, and the entry was kept
as Retained at Join with no way out but a forced unmount — or, where no
abort control is bound, no way out at all (Ephemeral-AI-Lab#426, Ephemeral-AI-Lab#460).

The unmount now makes the same single abort write a forced unmount makes,
on the already-detached connection, and drains again. It then completes as
Unmounted, revokes and closes, so the stalled connection ends where the
unmount asked. Only a refused or short write, or a second stop, keeps the
custody, exactly as any other stop does; such an entry is not forceable
either, because its abort control has been used.

- layerfs-fuse: Undrained::end_stalled, one abort_detached write then
  force_drain, only for a Join stop.
- layerfs-daemon: unmount_kernel calls it on a Join stop; a second stop
  keeps the evidence as before.
- tests: stalled_unmount now asserts the normal unmount ends the stall
  itself (the copy of the mount answers ENOENT) and keeps no custody.
- docs: native mount session and forced teardown describe the automatic
  abort; the SDK skill and the unmount comment no longer send callers to
  --force for this case.

Tests:
- cargo test -p layerfs-daemon --test stalled_unmount --release in a
  privileged Linux container: passes; the parent's code keeps custody, so
  the test fails without this change.
- also ran mounted_drain, fp21_detach_stages, control_root, mounted_fusectl,
  forced_unmount and forced_unavailable; all pass except the environment
  bound fp31 reclaim observation in mounted_drain (container cgroup memory
  reclaim, unrelated to unmount).

Production LOC: 102763 -> 102777 (delta +14)
  core 38395 -> 38395 (0), engine 61709 -> 61723 (+14), api 2659 (0).
@mengmengmengqiang

Copy link
Copy Markdown
Author

One review note with data: the 5 s in this PR is conservative, not necessary.

I measured the same machine (macOS 26.5.1, OrbStack 7.2.9, kernel 7.2.9-orbstack-00391, Docker 29.8.2), swapping the built sealed bundle's daemon under the running dsh worker so every run is the same topology as the scenario matrix:

Run n min p50 p90 max
normal unmount (this PR) 20 20 ms 23 ms 46 ms 82 ms
normal unmount after 64 MB read 10 25 ms 32 ms 71 ms 71 ms
early-mount stall (this PR) 20 5036 ms 5075 ms 5228 ms 5558 ms
early-mount stall with a 1 s join grace (PR #497) 20 1021 ms 1082 ms 1351 ms 1467 ms
normal unmount with the 1 s grace 20 17 ms 23 ms 34 ms 44 ms

Three points:

  1. The healthy join is tens of milliseconds (p50 23 ms, max 82 ms over 30 runs, including I/O). The 5 s deadline only ever waits for a connection that will not join: at the deadline loops_exited = 0, and Workspace mounted while its Sandbox starts cannot be unmounted (TeardownCustody { stage: Join }) and stays wedged #426 showed a retry 20 s later still refused. The wait is pure patience, not progress.
  2. The abort is cheap: after the deadline, abort + force drain + revoke + close took 36–558 ms (p50 75 ms), 20/20 Unmounted + Closed.
  3. Shortening it works, but not to 500 ms: a throwaway 500 ms prototype made stalls 519–565 ms, yet one healthy unmount in that run took 374 ms — the normal tail moves under load. That is why the follow-up uses a 1 s grace for the loop join only, keeping drain_wait for requests.

For the scenario matrix: the 10 scenarios contain 211 workspace.unmount calls, and the failing runs in #466 reported roughly 66 failed steps per pass. At ~4 s saved per stalled unmount that is about 1–5 minutes per matrix run, and a single agent call goes from ~5.1 s to ~1.1 s.

I put the change in #497 (fix(fuse): bound the receive-loop join by a 1 s grace), stacked on this PR: it only touches the join wait after quiescence, leaves the request deadline at 5 s, and stalled_unmount now asserts the stalled unmount returns in under 3 s (1.12 s with the grace, 3.23 s with the full wait). If you would rather keep this PR minimal, ignore #497 here and treat these numbers as the argument for a follow-up after this merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant