FUSE readers remain blocked after successful unmount: independent reproducer and Linux comparison (related to #426)

Summary

Related to #426, especially the intermittent TeardownCustody { stage: Join, detached: true, loops_exited: 0 } variant. This report adds kernel wait-state observations, a connection-specific abort experiment, and a small FUSE reproducer that does not use LayerFS, Rust/fuser, or SQLite.

On our macOS/OrbStack environment, normal unmount can remove the mount while both FUSE receive threads remain blocked. A minimal Go FUSE server also intermittently fails to join its readers after successful plain unmount. The same minimal test did not fail in our bounded Ubuntu comparison.

This is diagnostic evidence, not a confirmed OrbStack-specific root cause or a proposed production fix. The Docker Desktop report in #426 is relevant counter-evidence to an OrbStack-only attribution. The minimal test's relationship to the exact LayerFS defect still needs confirmation.

Environments

	macOS / OrbStack	Separate Ubuntu Linux host
Architecture	arm64	x86_64
Linux kernel	`7.0.14-orbstack-00380-ga7e0a2dc9535`	`6.8.0-138-generic`
Docker Engine	29.4.0, API 1.54	28.2.2, max API 1.50
containerd	2.2.2	1.7.28
runc	1.5.1	1.3.0-0ubuntu2~22.04.1
OrbStack	2.2.3 (2020300), commit `c83556b0ef8f1ba9a33abbb194622b6b7a1c0307`	N/A

LayerFS was built from ad78914e36b258b360daf13aec164b5655814689, using a sealed Linux ARM64 release bundle and the macOS worker/sibling-sandbox topology. LayerFS workloads used a tiny disposable Git/npm fixture, not a private application repository.

LayerFS observations

A read-only SDK exec running cat sum.js was sufficient to trigger the failure. A separate npm test case also failed. Other cases succeeded, so this was not an assertion that every unmount fails.
Normal unmount failed at Join after approximately 5 seconds, with detached: true, two readers entered, zero exited/joined, and active: 0, retained: 0.
The workspace mount was present before unmount and absent from mountinfo afterwards.
Inspecting the failed daemon from a diagnostic sidecar showed both receive threads blocked in fuse_dev_do_read (read syscall on the FUSE fd). The mount owner and fence threads were waiting on futexes.
The corresponding FUSE connection's waiting value was 0.
A read-only process/mount/fd scan found no obvious surviving workload process holding that workspace. This does not prove that no kernel or mount-namespace reference remained.

Connection-specific abort experiment

On one failed disposable test connection, after matching its FUSE device/connection ID, we wrote 1 to that connection's fusectl abort file. Both FUSE receiver threads and the mount/fence threads then disappeared immediately.

Captured thread states:

Before:
layerfs-fence-1 futex_wait
fuser-0 fuse_dev_do_read
fuser-1 fuse_dev_do_read
layerfs-daemon inet_csk_accept
layerfs-idle hrtimer_nanosleep
layerfs-control wait_woken
layerfs-mount-1 futex_wait

After connection-specific abort:
layerfs-daemon inet_csk_accept
layerfs-idle hrtimer_nanosleep
layerfs-control wait_woken

This establishes that aborting that connection releases the blocked receivers; it does not establish why automatic connection shutdown did not happen. It is not evidence of successful public SDK cleanup or a recommendation to abort unrelated connections.

Independent minimal FUSE experiment

The attached inline reproducer implements only enough FUSE protocol for cat file: two goroutines read the same /dev/fuse fd, then the main goroutine calls plain umount2(..., 0) and waits up to 2 seconds for both readers.

The external driver executes cat through docker exec and verifies the exact output hi\n. It optionally inserts 100 ms before unmount. This is different from delaying the initial mount after sandbox creation, as discussed in #426; we have not tested that workaround here.

Minimal test	OrbStack	Ubuntu host
External docker exec; 30 iterations; no pre-unmount delay	29 joined, 1 timeout	30 joined, 0 timeouts
External docker exec; 30 iterations; 100 ms pre-unmount delay	29 joined, 1 timeout	30 joined, 0 timeouts

Additional earlier runs: direct-child cat joined 10/10 on each environment; external docker exec joined 9/10 on OrbStack and 10/10 on Ubuntu.

Example failing sequence from the OrbStack 30-iteration run:

case=10 cat="" err=<nil> umount= <nil>
JOIN FAIL 10

Here cat="" is the server's unused direct-child output variable in EXTERNAL mode, not the external command's result. The Python driver separately asserts hi\n.

Interpretation limits

These are small bounded samples, not estimates of production failure probability. No failures on Ubuntu does not prove absence of the issue.
Architecture, kernel, Docker, containerd and runc differ together; this comparison cannot isolate any one component.
The driver signals completion with touch inside the same docker-exec shell, so unmount may race with that shell/runtime's final exit. A 100 ms delay is not proof that all runtime references have been released. A useful next test is a completion signal sent only after the original docker exec fully returns.
The Go program is a deliberately minimal protocol implementation, not a conformant general filesystem. A reader timeout is diagnostic evidence only.
On a timeout the program retains the fd/readers until container removal and continues the bounded run; subsequent iterations are therefore not fully independent.
We did not run the full LayerFS SDK on the Ubuntu host: its Docker API maximum is 1.50, while this pinned LayerFS implementation uses 1.54. We did not upgrade the shared host.
All experiment containers were removed after collecting evidence.

Reproduction source

Save the following as minifuse-v2.go and drive-minifuse.py. Build the Go file for the Linux architecture of the Docker engine, e.g. with Go 1.26:

CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o minifuse-v2 minifuse-v2.go
# Use GOARCH=amd64 for an x86_64 Linux engine.
python3 drive-minifuse.py fuse-join-probe node:26 "$PWD/minifuse-v2" 30 0 results.txt
python3 drive-minifuse.py fuse-join-probe-delay node:26 "$PWD/minifuse-v2" 30 100 results-delay.txt

Run only against a disposable test container on a FUSE-capable Docker engine. The driver grants that container SYS_ADMIN and /dev/fuse and removes its named container in finally. Our runs used already-local Node 26 images (Ubuntu: node:26.5.0-bookworm); image identity is another uncontrolled difference.

minifuse-v2.go

package main
import("encoding/binary";"fmt";"os";"os/exec";"syscall";"sync";"time";"strconv")
var le=binary.LittleEndian
func attr(ino uint64)[]byte{b:=make([]byte,88);le.PutUint64(b,ino);le.PutUint32(b[64:],1);le.PutUint32(b[80:],4096);if ino==1{le.PutUint32(b[64:],2);le.PutUint32(b[60:],0040755)}else{le.PutUint64(b[8:],3);le.PutUint32(b[60:],0100644)};return b}
func main(){count:=10;if v,e:=strconv.Atoi(os.Getenv("ROUNDS"));e==nil{count=v};for n:=0;n<count;n++{run(n)}}
func run(n int){dir:=fmt.Sprintf("/tmp/minifuse-%d",n);os.Mkdir(dir,0755);fd,e:=syscall.Open("/dev/fuse",syscall.O_RDWR,0);if e!=nil{panic(e)};e=syscall.Mount("mini",dir,"fuse",syscall.MS_NOSUID|syscall.MS_NODEV,fmt.Sprintf("fd=%d,rootmode=40000,user_id=0,group_id=0,max_read=131072",fd));if e!=nil{panic(e)}
var wg sync.WaitGroup;wg.Add(2);for i:=0;i<2;i++{go func(){defer wg.Done();for{b:=make([]byte,135168);size,e:=syscall.Read(fd,b);if e!=nil{fmt.Println("read exit",n,e);return};if size<40{return};op:=le.Uint32(b[4:]);ino:=le.Uint64(b[16:]);var p []byte;errnum:=int32(0);switch op{case 26:p=make([]byte,64);le.PutUint32(p,7);le.PutUint32(p[4:],31);le.PutUint32(p[20:],131072);le.PutUint32(p[24:],1)
case 1:p=make([]byte,40);le.PutUint64(p,2);p=append(p,attr(2)...)
case 3:p=append(make([]byte,16),attr(ino)...)
case 14:p=make([]byte,16);le.PutUint64(p,1)
case 15:p=[]byte("hi\n");off:=le.Uint64(b[48:]);if off>=3{p=nil}else{p=p[off:]}
case 18: p=nil
case 2,42:continue
case 38: return
 default:errnum=-38}
out:=make([]byte,16);le.PutUint32(out,uint32(16+len(p)));le.PutUint32(out[4:],uint32(errnum));copy(out[8:],b[8:16]);out=append(out,p...);if _,e=syscall.Write(fd,out);e!=nil{fmt.Println("write error",e);return}}}()}
var out []byte; if os.Getenv("EXTERNAL")=="1" {os.WriteFile("/tmp/ready",[]byte(fmt.Sprint(n)),0644);for {if _,err:=os.Stat(fmt.Sprintf("/tmp/go-%d",n));err==nil{break};time.Sleep(10*time.Millisecond)}}else{out,e=exec.Command("cat",dir+"/file").CombinedOutput()};fmt.Printf("case=%d cat=%q err=%v ",n,out,e);if v,e:=strconv.Atoi(os.Getenv("DELAY_MS"));e==nil{time.Sleep(time.Duration(v)*time.Millisecond)};e=syscall.Unmount(dir,0);fmt.Println("umount=",e);done:=make(chan bool);go func(){wg.Wait();done<-true}();select{case <-done:fmt.Println("JOIN PASS",n);syscall.Close(fd);case <-time.After(2*time.Second):fmt.Println("JOIN FAIL",n);if os.Getenv("HOLD_FAIL")=="1"{os.WriteFile("/tmp/failed",[]byte(fmt.Sprint(n)),0644);time.Sleep(40*time.Second);os.Exit(2)}} }

drive-minifuse.py

import subprocess,time,json,sys
name,image,binary,rounds,delay,out=sys.argv[1:]
rounds=int(rounds)
def cmd(args,**kw):return subprocess.run(['docker',*args],check=True,**kw)
cmd(['run','-d','--name',name,'--cpus=2','--memory=1g','--cap-add','SYS_ADMIN','--device','/dev/fuse','--security-opt','apparmor=unconfined','-e','EXTERNAL=1','-e','ROUNDS='+str(rounds),'-e','DELAY_MS='+delay,'-v',binary+':/minifuse:ro',image,'/minifuse'],stdout=subprocess.DEVNULL)
started=time.monotonic()
try:
 for n in range(rounds):
  deadline=time.monotonic()+6
  while time.monotonic()<deadline:
   r=subprocess.run(['docker','exec',name,'cat','/tmp/ready'],capture_output=True,text=True)
   if r.stdout.strip()==str(n):break
   time.sleep(.05)
  else:raise Exception('not ready: '+str(n))
  r=cmd(['exec',name,'sh','-c',f'cat /tmp/minifuse-{n}/file && touch /tmp/go-{n}'],capture_output=True,text=True)
  assert r.stdout=='hi\n',repr(r.stdout)
 cmd(['wait',name],timeout=8,stdout=subprocess.DEVNULL)
finally:
 r=subprocess.run(['docker','logs',name],capture_output=True,text=True)
 logs=r.stdout+r.stderr
 open(out,'w').write(logs)
 print(json.dumps({'name':name,'passes':logs.count('JOIN PASS'),'failures':logs.count('JOIN FAIL'),'seconds':round(time.monotonic()-started,2),'delayMs':delay}))
 cmd(['rm','-f',name],stdout=subprocess.DEVNULL)

Requested follow-up

Could maintainers correlate these observations with #426, especially mount/connection ownership during container startup and exec teardown? It would help to determine whether the detached mount still has references outside the inspected namespace, and whether LayerFS needs a recoverable teardown state when the connection does not terminate.

No fix is claimed here. Feel free to consolidate this report into #426 if that is easier to track.

Merge pull request #481 from Ephemeral-AI-Lab/issue426-forced-unmount

fix(daemon): end a normal unmount stalled at Join with one forced unm…

We prototyped the private-mount-namespace mitigation and completed the command path. It removes the Join stall — early-mount unmount median 22 ms (n=20) — and the daemon hands its mount-namespace fd to a small ns-exec helper over a private socket (SCM_RIGHTS), so workspace.exec runs inside the daemon's namespace instead of the container's initial one. E2E on a real Sandbox: exec 10/10 (p50 54 ms), early-mount unmount p50 15 ms; with the flag off, exec works and unmount stays ~5.0 s. The open question is now a design decision: is this direction wanted, or does the abort path in #496/#497 stay the answer?

_{Reading guide. Left: the bug in one column. Middle: what #496 and #497 change, and what each costs. Right: how to read the panels below (lanes, numbered unmount steps, colors). The prototype below is about the third direction, the private namespace.}

_{CAUSE (top): the mount outlives the detach even though we cannot find the holder in any namespace we can enumerate, so the receive loops stay blocked. SOLUTION (bottom): the daemon unshares CLONE_NEWNS with private propagation before mounting FUSE; measured early-mount unmount p50 15 ms, exec 10/10 (p50 54 ms). The helper receives the daemon's mount-ns fd over ns-exec.sock and runs the command there.}

Environment: macOS 26.5.1, OrbStack (kernel 7.2.9-orbstack-00391), Docker 29.8.2 (API 1.54). LayerFS 582bc786 + #496/#497: the prototype is stacked on fix/460-join-grace-1s, so the measured build contains them; its own effect does not rely on them.

Diagnosis (measured on the running stack, inside a real Sandbox)

/ in the Sandbox is already private; only PID 1 (docker-init) and the daemon share mnt:[4026532712], and the FUSE row (0:77 / /workspaces/1 … - fuse layerfs) exists only there.
No process fd/cwd points into /workspaces; a privileged --pid=host scan of the whole Docker VM finds the mount in those two processes only. We could not locate an external holder — every namespace we can enumerate has already lost the mount row by the time the stall is visible, yet the connection outlives the detach.
During a stalled unmount (250 ms in), the mount row is already gone, yet the receive loops stay unjoined and the call returns only at the ~5 s Join deadline. (We did not use the absence of a /sys/fs/fuse/connections entry as evidence: the daemon mounts its own fusectl, so the init namespace is not the right place to look; what matters is that the abort fd the daemon holds at attach still works when asked.)
The trigger is a timing window. 2×2 (delay before mount / delay before unmount; two runs each):

before mount	before unmount	unmount
0 ms	300 ms	22 / 19 ms
2500 ms	1500 ms	43 / 15 ms
0 ms	1500 ms	5066 / 5029 ms
2500 ms	300 ms	12 / 9 ms

So the mount must be created immediately after sandbox.create() and live ~1.5 s for the stall to appear.

Correction to an assumption in our earlier notes (it also appeared in our design sketches): the Sandbox seccomp profile does not deny unshare/clone in general — the masked rule matches CLONE_NEWUSER (0x10000000) only; CLONE_NEWNS (0x20000) passes. Verified in a live Sandbox under the unchanged profile (unshare -m --propagation private … → rc 0). No policy change was needed.

Prototype

Daemon, before it starts its serving threads: unshare(CLONE_NEWNS) + mount(NULL, "/", MS_REC|MS_PRIVATE), so the FUSE mount is created only in the daemon's own namespace. Patch: 4 files, +31/−2 (production LOC +14, all engine), commit 644556d0 on branch experiment/private-mount-ns, based on fix/460-join-grace-1s. cargo check passes; the existing stalled_unmount test still passes (1.07 s on this run; #497 recorded 1.12 s) — it covers the grace + abort on the base only: the harness is in-process, never calls the daemon entry that unshares, and makes its own unshare -m copy, so it does not yet cover the new topology.

Measurement (isolated worker from the patched bundle; running install untouched)

Probe	pre-patch bundle (same isolated worker)	private-ns build
early-mount unmount (n per column)	5028, 5030, 5040 ms (n=3)	21, 17, 26, 8, 25, 29, 293, 41, 30, 45, 41, 12, 23, 21, 25, 17, 21, 14, 20, 13 (median 22; 19 of 20 ≤ 45 ms; one 293 ms scheduling outlier)
delayed mount (n per column)	18, 31 ms (n=2)	13, 13, 9, 17, 12, 13, 17, 9, 15, 15 (n=10, 9–17 ms)

The running install's own baseline over the same period: early-mount 5022–5066 ms (n=5), delayed 9–58 ms (n=5). All runs in the private-ns build returned Unmounted; the 1 s join grace (from #497) was never reached and no Failed with a Join custody appeared. The single 293 ms sample is most likely scheduling, not the grace.

What we did not establish

The identity of the holder: not visible inside the Docker VM; the mount row is gone from every namespace we can enumerate. The mechanism is inferred from the timing and from the fix's effect, not from an identified actor.
Whether the same window exists on other engines/kernels (Docker Desktop, Linux hosts) — only this OrbStack kernel was exercised.
Whether the workaround is needed once the holder is identified and removed at the source.
docker exec visibility: it still does not see /workspaces/* in a private-ns Sandbox; only workspace.exec is routed. macOS and multi-tenant hardening of the socket were not exercised.

Where this leaves the design

Private namespace, with exec in it. Run commands from the daemon's namespace: either a daemon exec endpoint, or a small in-container helper that setns-es into the daemon's mount namespace and then execs. That is a bigger change than this prototype and the first thing to design.
Abort path only. Keep fix(daemon): end a stalled normal unmount at Join with its own abort write (#426, #460) #496/fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) #497 as the answer: recovery at the drain deadline (5 s) and the 1 s join grace. This is shipped-able today and needs no namespace change.
Hiding the symptom (private ns) without identifying the holder is a third option, but it buys 22 ms at the cost of the command path; not recommended as-is.

Which direction would you prefer? The fd-passing prototype is ready for review in the fork (tokenaissance#5, with the design notes in tokenaissance#3); if the direction is not wanted, we will leave the abort path as the answer and close the experiment.

FUSE: a plain `umount2` detach does not wake readers blocked in `read()` while the connection stays referenced

Kind: diagnostic evidence and an open behavior question for the fs/fuse and OrbStack kernel sides. This is not a confirmed kernel bug, not a patch, and not a claim that the behavior is OrbStack-specific. A related layer-level report exists against Docker Desktop (LayerFS #426), which is relevant counter-evidence to an OrbStack-only attribution.

TL;DR

LayerFS runs a userspace FUSE daemon inside a Docker container on macOS/OrbStack. A plain Workspace unmount detaches the mount (umount2) and the mount row disappears, but the daemon's two FUSE receive loops stay blocked in read(/dev/fuse) and teardown stops at Join until a deadline.

It reproduces deterministically when the mount is created immediately after the container is started and lives about 1.5 s: 6/6 at LayerFS 582bc786, 1/1 at v1.1.0.
At 250 ms into a stalled unmount the mount row is already gone from the container's mount namespace, the receive loops are not joined, and the connection is still alive.
Writing 1 to the connection's fusectl abort file ends it. After the abort, drain + revoke + close completes in p50 75 ms (max 558 ms). With a 1 s join grace the stall becomes about 1.08 s.
If the daemon creates the mount inside its own private mount namespace (unshare(CLONE_NEWNS)), the stall does not occur: early-mount unmount p50 15 ms (n=10, 7.2–35.0 ms); the same mount with the flag off stalls about 5.0 s.

We could not locate the reference that keeps the connection alive in any namespace we can enumerate. The question below asks whether the wake-up expectation is wrong, how userspace should learn that no mount/user remains, and whether an OrbStack kernel change is involved.

Environment

	Value
Host	macOS 26.5.1 (Darwin 25.5.0, arm64)
Container runtime	OrbStack 7.2.9
Docker Engine	29.8.2, API 1.54
Linux kernel	`7.2.9-orbstack-00391-g2b298d05fbd4`
LayerFS	FUSE daemon in a container; `582bc786` (and v1.1.0 for one confirmation run)

The FUSE daemon is a normal unprivileged userspace server for the Workspace mount; the container has CAP_SYS_ADMIN and /dev/fuse. The daemon is non-dumpable (PR_SET_DUMPABLE=0), and the container has no CAP_SYS_PTRACE; this limits what the reproducer can inspect, not the kernel behavior under test.

Reproducer

The probe uses the public LayerFS JavaScript SDK and must run against a disposable project and Sandbox.

Open an existing project.
sandbox.create().
Optionally wait BEFORE_MOUNT_MS, then workspace.mount(sandbox, branch) immediately.
Optionally wait BEFORE_UNMOUNT_MS (the failing case: 1500 ms).
Call plain workspace.unmount(ws) and record its wall time and result.

// probe.mjs -- requires @layerfs/agent and an existing project and branch.
import { LayerFS } from "@layerfs/agent";

const project = process.env.LAYERFS_PROJECT;      // existing project
const branch = process.env.LAYERFS_BRANCH;        // existing branch
const beforeMountMs = Number(process.env.BEFORE_MOUNT_MS ?? 0);
const beforeUnmountMs = Number(process.env.BEFORE_UNMOUNT_MS ?? 1500);
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

const layerfs = await LayerFS.open(project);
const { sandbox } = await layerfs.sandbox.create();   // start the Sandbox
if (beforeMountMs) await sleep(beforeMountMs);
const ws = await layerfs.workspace.mount(sandbox, branch); // mount

await sleep(beforeUnmountMs);

const started = performance.now();
let outcome;
try {
  outcome = { result: await layerfs.workspace.unmount(ws) };
} catch (error) {
  outcome = { code: error.code, message: error.message, detail: error.detail };
}
outcome.unmountMs = Math.round(performance.now() - started);
console.log(JSON.stringify(outcome, null, 2));

// A stalled unmount keeps custody; force it before closing the Sandbox.
if (outcome.code === "Failed") {
  try { await layerfs.workspace.unmount(ws, true, true); } catch {}
}
try { await layerfs.sandbox.close(sandbox); } catch {}
await layerfs.close();

On the affected environment the (BEFORE_MOUNT_MS=0, BEFORE_UNMOUNT_MS=1500) arm returns a retained Join custody after about 5.0 s (the SDK surfaces it as Failed with stage: Join in the message/detail); the other arms return Unmounted in tens of milliseconds. Use a disposable Sandbox: a stalled unmount keeps kernel custody until it is force-ended.

To inspect the stall itself, an independent side observation is enough: 250 ms after the unmount starts, read /proc/<pid>/mountinfo (or the namespace's mountinfo) and the daemon's thread states. The daemon's own logs record the retained custody (stage: Join, detached: true, loops_exited: 0).

Observed stall state

At 250 ms into a stalled unmount:

The mount row is already gone from the container's mount namespace, yet the receive loops are not joined.
The connection outlives the detach: the FUSE connection is still alive and both readers remain blocked in fuse_dev_do_read / read(/dev/fuse).
The unmount call returns only at the ~5 s Join deadline. The retained custody reports detached: true, loops_exited: 0 (readers entered, none exited).

We could not find the holder of the remaining reference in any namespace we can enumerate. No process fd or cwd pointed into /workspaces; a privileged --pid=host scan of the Docker VM found the FUSE mount only in the container's PID-1/daemon mount namespace, not in an additional holder. We therefore infer that something keeps a mount/superblock reference outside the container's visible namespace, but we did not identify the actor.

Timing: the stall is a window, not every unmount

BEFORE_MOUNT_MS / BEFORE_UNMOUNT_MS, two runs each; values are unmount wall time:

before mount	before unmount	unmount
0 ms	300 ms	22 / 19 ms
2500 ms	1500 ms	43 / 15 ms
0 ms	1500 ms	5066 / 5029 ms
2500 ms	300 ms	12 / 9 ms

The stall requires the mount to be created immediately after sandbox.create() and to live about 1.5 s. Delaying the mount by 2.5 s, or unmounting after 300 ms, avoids it. The long values are the 5 s drain/join deadline, not work being done.

What ends it: the `fusectl` abort

The daemon holds the per-connection abort control from attach (it mounts its own fusectl). Writing 1 to /sys/fs/fuse/connections/<n>/abort aborts the connection, wakes the blocked readers, and lets teardown continue.

After the abort, drain + revoke + close completes in p50 75 ms (max 558 ms; tail measured on the 5 s build).
LayerFS fix(daemon): end a stalled normal unmount at Join with its own abort write (#426, #460) #496 makes this abort write on the already-detached connection when the drain stops at Join; fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) #497 adds a 1 s join grace, so the same stall ends at p50 1.08 s (n=20, 1021–1467 ms) instead of about 5.1 s.

This is recovery, not a diagnosis. It deliberately ends the connection; if another namespace legitimately holds the mount, aborting would break its users. Knowing whether such a holder exists is exactly what userspace lacks.

Control: a private mount namespace avoids the stall

When the daemon calls unshare(CLONE_NEWNS) and mount(NULL, "/", MS_REC|MS_PRIVATE) before mounting, the FUSE mount is created only in the daemon's own mount namespace and never appears in the container's initial mount namespace.

Early-mount unmount: p50 15 ms (n=10, range 7.2–35.0 ms), no stall.
The same mount with the flag off: about 5.0 s.

This is a control, not a proposed fix: it shows that the stall depends on where the mount lives relative to the container's initial namespace, and it avoids the unknown holder rather than identifying it. It does not establish that the holder is benign.

The question for the kernel / OrbStack side

We did not instrument or patch the kernel. All observations are from userspace inside the container. We also could not read the source of 7.2.9-orbstack-00391: the public orbstack/linux-macvirt tree we can inspect has an older base, so we cannot say whether the running kernel carries a change in this area. The same layer-level symptom has been reported against Docker Desktop (LayerFS #426), so this is not established as OrbStack-specific.

Should a plain umount2 detach wake or abort the blocked FUSE readers when the superblock is kept alive only by a mount in another namespace? In the kernel source we can read, a FUSE connection is aborted on teardown only when the mount being removed is the last one for that connection; a plain detach in one namespace therefore leaves the connection and its blocked readers alone while another reference exists. Is that the intended behavior for the case where the caller's namespace no longer has the mount at all?
Can the kernel expose a reliable "no active mounts/users" signal — for example a per-connection mount-reference count or a "detached but still referenced" state — so userspace can decide to abort deterministically instead of guessing a grace period? Today userspace sees blocked readers and a connection that outlives its namespace, but no supported way to tell "nothing can ever use this connection again" from "teardown is merely slow".
Is there an OrbStack kernel patch in this area (FUSE abort/teardown or mount-namespace reference handling) that could explain why the remaining reference is invisible to every namespace we can enumerate? We could not inspect that kernel version.

What we did not establish

The identity of the holder. The mount row is gone from the container's namespace and no other namespace showed it; the mechanism is inferred from the blocked readers and from the effect of the abort and private-namespace controls.
Mainline behavior. Only this OrbStack kernel was exercised; we cannot say whether mainline fs/fuse behaves the same.
Other engines/kernels. Docker Desktop is implicated by the related report but was not tested here.
Whether the abort is safe in the presence of an unknown holder. The abort ends the connection for every mount of it, including one we cannot see.
Whether the private-namespace path is viable for a product. It changes where commands must run; the control above measured mount/unmount only.

References

LayerFS issue FUSE readers remain blocked after successful unmount: independent reproducer and Linux comparison (related to #426) #460 (this thread): FUSE readers remain blocked after successful unmount: independent reproducer and Linux comparison (related to #426) #460
LayerFS issue Workspace mounted while its Sandbox starts cannot be unmounted (TeardownCustody { stage: Join }) and stays wedged #426 (earlier report, including a Docker Desktop occurrence): Workspace mounted while its Sandbox starts cannot be unmounted (TeardownCustody { stage: Join }) and stays wedged #426
LayerFS fix(daemon): end a stalled normal unmount at Join with its own abort write (#426, #460) #496 (abort write at the drain deadline): fix(daemon): end a stalled normal unmount at Join with its own abort write (#426, #460) #496
LayerFS fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) #497 (1 s join grace): fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) #497
LayerFS repository: https://github.com/Ephemeral-AI-Lab/layerfs

Where this sits

_{Recover (#496/#497) is shipped and bounds the stall; prevent (private mount namespace + ns-exec, #499 / fork #5) is a verified prototype that removes it but changes docker exec visibility; the kernel fix described here would remove it without either.}

fix(fuse): bound the receive-loop join by a 1 s grace, not drain_wait (…

fix(daemon): end a stalled normal unmount at Join with its own abort …

Navigation Menu

Repository navigation

FUSE readers remain blocked after successful unmount: independent reproducer and Linux comparison (related to #426) #460

Summary

Environments

LayerFS observations

Connection-specific abort experiment

Independent minimal FUSE experiment

Interpretation limits

Reproduction source

Requested follow-up

Diagnosis (measured on the running stack, inside a real Sandbox)

Prototype

Measurement (isolated worker from the patched bundle; running install untouched)

What we did not establish

Where this leaves the design

FUSE: a plain `umount2` detach does not wake readers blocked in `read()` while the connection stays referenced

TL;DR

Environment

Reproducer

Observed stall state

Timing: the stall is a window, not every unmount

What ends it: the `fusectl` abort

Control: a private mount namespace avoids the stall

The question for the kernel / OrbStack side

What we did not establish

References

Where this sits

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

FUSE readers remain blocked after successful unmount: independent reproducer and Linux comparison (related to #426) #460

Description

Summary

Environments

LayerFS observations

Connection-specific abort experiment

Independent minimal FUSE experiment

Interpretation limits

Reproduction source

Requested follow-up

Activity

mengmengmengqiang commented on Oct 28, 2026

Diagnosis (measured on the running stack, inside a real Sandbox)

Prototype

Measurement (isolated worker from the patched bundle; running install untouched)

What we did not establish

Where this leaves the design

mengmengmengqiang commented on Oct 28, 2026

FUSE: a plain umount2 detach does not wake readers blocked in read() while the connection stays referenced

TL;DR

Environment

Reproducer

Observed stall state

Timing: the stall is a window, not every unmount

What ends it: the fusectl abort

Control: a private mount namespace avoids the stall

The question for the kernel / OrbStack side

What we did not establish

References

Where this sits

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

FUSE: a plain `umount2` detach does not wake readers blocked in `read()` while the connection stays referenced

What ends it: the `fusectl` abort