[mengmengmengqiang]

Status: prototype complete — design decision required. The private mount namespace and the ns-exec fd-passing helper are implemented and E2E-verified on a real Sandbox (fork tokenaissance#5). The remaining question is whether upstream wants this direction — commands running in the daemon's namespace — or whether the abort path in #496/#497 stays the answer.

Branch experiment/private-mount-ns (6554e7fd), based on main; it now contains the full implementation (private ns + exec helper).

Visual evidence walk-through

Start with the architecture and the unmount sequence so the change has a frame, then the cause/fix panel and the options board. All measured on macOS 26.5.1, OrbStack (kernel 7.2.9-orbstack-00391), Docker 29.8.2.

_{Where the change lives. The daemon/FUSE pair is the adapter band; the private namespace is a startup property of the daemon, and the exec helper is an adapter that joins it — no dependency direction or crate boundary changes.}

_{One bash call from create to unmount. This prototype removes the condition in step 13 entirely: with no external reference, the loop join follows the detach in milliseconds. Step 8 (workspace.exec) reaches the mount through the fd-passing helper (see "The exec path").}

_{Top: the mount outlives the detach even though we cannot find the holder in any namespace we can enumerate, so the loops stay blocked and the entry ends as Retained at Join. Bottom: the daemon unshares CLONE_NEWNS and makes propagation private before mounting, so no external namespace can take a copy; measured early-mount unmount median 22 ms. The red line is the answer: the helper takes the daemon's namespace fd over a socket and runs the command there.}

_{Where this sits among the options: the private-namespace card is marked measured · prototype (exec prototype ready in fork #5); recovery (#496) and the 1 s grace (#497) stay as the safety net.}

Problem

#496/#497 recover from the Join stall and bound its cost, but the stall still happens: on this machine a mount created immediately after sandbox.create() and kept ~1.5 s is held by something we could not identify — every namespace we can enumerate has already lost the mount row — so umount2 detaches while the FUSE receive loops stay blocked.

Where the reference is not: not in any fd/cwd, and not in the Docker VM's mount table for other processes; by the time the drain runs, the mount row is already gone while the loops stay blocked. (The daemon's own fusectl means the init namespace's /sys/fs/fuse/connections is not a reliable observation point, so we do not use it as evidence.) We could not identify the holder; the mechanism is inferred from timing and from the fix's effect. 2×2 timing (delay before mount / delay before unmount; two runs each) isolates the window:

before mount	before unmount	unmount
0 ms	300 ms	22 / 19 ms
2500 ms	1500 ms	43 / 15 ms
0 ms	1500 ms	5066 / 5029 ms
2500 ms	300 ms	12 / 9 ms

Change

• daemon: opt-in private_mount_ns; unshare(CLONE_NEWNS) + MS_REC|MS_PRIVATE on / before any serving thread; writes daemon.pid; serves its mount-namespace fd over /layerfs-local/ns-exec.sock (mode 0600, peer uid 0 in the same pid namespace, one SCM_RIGHTS message, nothing read).
• helper layerfs-daemon ns-exec --socket … --directory … --uid … --gid … -- bash -c …: receive fd → setns(CLONE_NEWNS) → chdir --directory → drop to the command uid/gid → exec.
• API: wraps the Engine argv only when the Sandbox record carries the socket; otherwise the argv is byte-identical to before.
• the pid-file route is removed: the daemon is non-dumpable and the Sandbox has no CAP_SYS_PTRACE, so /proc/<pid>/ns/mnt can never be opened in deployment.

The exec path (fd passing)

workspace.exec is a Docker Engine exec in the container's initial mount namespace; the daemon is not involved. The wrapper starts layerfs-daemon ns-exec as root (the command identity cannot open the namespace), which receives the daemon's mount-ns fd over the socket, setns, re-enters the workspace directory, drops to 501:20, and execs the user command. The Engine entry stays, so output streaming, cancellation and exit codes are unchanged.

Measurement

Isolated worker (private ns, mount/unmount only): early-mount unmount median 22 ms (n=20).

E2E on a real Sandbox (sealed bundle 8fdaaf92, LAYERFS_PRIVATE_MOUNT_NS=1):

test	n	exec	unmount
early mount	10	10/10 exit 0, p50 54.2 ms	10/10 Unmounted, p50 15.0 ms (7.2–35.0)
control, flag off	5	5/5 exit 0	5/5 refused, p50 5036 ms (the old wedge)
delayed mount	5	p50 101.5 ms	p50 20.0 ms
normal path	5	p50 37.4 ms	p50 10.8 ms

The command runs in the daemon's namespace (readlink /proc/self/ns/mnt = the daemon's mnt:[4026533424]), sees the layerfs /workspaces/1 fuse row, runs as 501:20, and passes exit codes/stdout/stderr. A marker written and read back, and committed, in 10/10 runs.

_{Per-approach matrix: this PR's row (private ns + fd passing) is early-mount unmount p50 15.0 ms with exec 10/10; the #501 row is the telemetry PR on its own head (no private ns, no 1 s grace), so its ~5 s is the recover path, not a regression.}

Tests

• cargo check for the touched crates passes; bridge 2, api 2, daemon lib 5, ns_exec integration 11, engine_protocol 17 pass in rust:1.85.1-bookworm --cap-add SYS_ADMIN.
• E2E as above. cargo fmt --check was not run (rustfmt absent from that image).

What this does not prove

• docker exec into a private-ns Sandbox still cannot see /workspaces/* (by design; only workspace.exec is routed).
• A Sandbox created before this change is refused clearly (Unsupported: … close it and create a new Sandbox); there is no live migration.
• macOS and multi-tenant hardening of the socket were not exercised; the socket is root-only in the container's pid namespace.
• The identity of the external holder remains unknown.

Trade-offs / open question

1. Wrapped Execs start as root and the helper drops to the command identity before exec; this is a deliberate capability in exchange for not adding CAP_SYS_PTRACE.
2. docker exec visibility into /workspaces/* is gone for private-ns Sandboxes.
3. The namespace dies with the daemon (better cleanup, but recovery paths that expect a second process to see the mount no longer work).
4. Opt-in; the abort path (fix(daemon): end a stalled normal unmount at Join with its own abort write (#426, #460) #496/fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) #497) stays the safety net.

Related

#426, #460, #466, #496, #497; implementation in tokenaissance#5; design notes in tokenaissance#3.

[mengmengmengqiang]

Prototype for the blocker: tokenaissance#5 implements the setns helper (opt-in daemon private ns + pid file + Engine argv wrapper). Unit and unshare -m integration tests pass; a real Sandbox end-to-end is still missing, so this PR stays draft.