[mengmengmengqiang]

fd passing (commit 6554e7fd)

The helper no longer opens /proc/<pid>/ns/mnt. The daemon serves its own mount-namespace descriptor over a private Unix socket (ns-exec.sock, mode 0600): the peer must be uid 0 in the same pid namespace, and exactly one SCM_RIGHTS message is sent. The helper receives the fd, setns(CLONE_NEWNS), re-enters --directory, drops to the command uid/gid and execs.

• The pid-file route is removed: it could never work in a deployment (the daemon is non-dumpable and the Sandbox has no CAP_SYS_PTRACE; the earlier E2E returned 10/10 EACCES).
• MountNamespace is now socket-only; DaemonSetup::check() requires the socket; old records derive the socket path (a pre-socket daemon never bound it, so exec refuses until the Sandbox is recreated).
• Tests in rust:1.85.1-bookworm --cap-add SYS_ADMIN: cargo check (5 crates) passes; bridge 2, api 2, daemon lib 5, ns_exec integration 11, engine_protocol 17 pass. cargo fmt --check was not run (rustfmt not installed in that image).

E2E passed (2026-10-28)

Separate worker, sealed bundle build-linux-8fdaaf92b71b4d149bf1c46fb8976edb (head 6554e7fd), LAYERFS_PRIVATE_MOUNT_NS=1:

• early mount, n=10: workspace.exec 10/10 exit 0, pwd = /workspaces/1, marker written/read/committed 10/10; exec p50 54.2 ms (45.9–186.9); unmount p50 15.0 ms (7.2–35.0), 10/10 Unmounted; close 10/10 Closed.
• the command really runs in the daemon's namespace: readlink /proc/self/ns/mnt = the daemon's mnt:[4026533424] (init ns is a different one with 0 /workspaces rows); the FUSE row is visible; uid 501 / gid 20; exit 7 probe returns exitCode 7 with stdout/stderr; background process.start/cancel work.
• control, flag off, n=5: exec OK, unmount 5/5 refused after p50 5036 ms — the old wedge.
• delayed mount, n=5: exec p50 101.5 ms, unmount p50 20.0 ms. Normal path, n=5: exec p50 37.4 ms, unmount p50 10.8 ms.
• old Sandbox across the upgrade is refused clearly: Unsupported: Sandbox 1 was created by a daemon that does not serve its private mount namespace; close it and create a new Sandbox ….

Earlier pid-file E2E (below) failed 10/10 with EACCES; that route is removed.

E2E result (2026-10-28): the pid-file route could not open the daemon's namespace

The branch was built into a sealed Linux bundle and run against a separate worker with LAYERFS_PRIVATE_MOUNT_NS=1:

• workspace.exec refused 10/10: exit 125, layerfs-daemon ns-exec: open mount namespace /proc/6/ns/mnt: Permission denied (os error 13); the command never starts.
• The private namespace itself works: /layerfs-local/daemon.pid = 6; the container's init namespace has no /workspaces row; the daemon namespace (probed with CAP_SYS_PTRACE) has the FUSE row; early-mount unmount p50 27.8 ms (n=10, max 182.1 ms), no ~5 s stall.
• Control (flag off, same bundle): exec 5/5 OK (sees /workspaces/1 and the FUSE row); unmount 5/5 refused after p50 5022 ms — the old wedge is back. The flag isolates cleanly.
• Root cause: the daemon makes itself non-dumpable (layerfs-daemon/src/application/config.rs:105, prctl::set_dumpable(false)); the Sandbox has CAP_SYS_ADMIN but not CAP_SYS_PTRACE, and the Engine topology check rejects adding it. Root in the container is denied too; a dumpable sibling is joinable, and a CAP_SYS_PTRACE probe can join the daemon.
• Fixed in this revision: the daemon now hands its mount-namespace descriptor to the helper over a Unix socket with SCM_RIGHTS (commit 6554e7fd, section below). The pid-file route is deleted: it cannot work in a Sandbox, so the helper no longer names a path that only returns EACCES.

This PR stays draft until the rebuilt bundle passes the same E2E.

Update: the daemon hands its mount-namespace fd to the helper

Commit 6554e7fd on feat/ns-exec-helper:

• Daemon side (layerfs-daemon/src/application/ns_fd.rs, only when private_mount_ns is on): binds <overlay grandparent>/ns-exec.sock (mode 0600, a stale path is removed, the path is unlinked on drop) and serves it from one background thread. On each accepted connection it reads SO_PEERCRED and requires uid 0 and this process's pid namespace (/proc/<pid>/ns/pid compared with /proc/self/ns/pid), opens /proc/self/ns/mnt itself and sends exactly one SCM_RIGHTS message with that one descriptor, then closes. A refused peer is closed without sending anything.
• Helper (application/ns_exec.rs): ns-exec --socket <path> [--directory <path>] [--uid <uid> --gid <gid>] -- <command> [args...] receives the descriptor, setns(CLONE_NEWNS), then keeps the existing --directory re-entry, uid/gid drop (supplementary groups cleared) and exec. The received raw descriptor is reopened through /proc/self/fd so the owned-file API works without unsafe code, and its label is checked to be mnt:. The pid-file exec route and its --pid-file flag are deleted; the daemon still writes <local>/daemon.pid as a diagnostic marker for the E2E.
• Wiring: the Engine argv becomes ["/usr/local/bin/layerfs-daemon", "ns-exec", "--socket", "/layerfs-local/ns-exec.sock", "--directory", …, "--uid", "501", "--gid", "20", "--", "bash", "-c", <command>]; the Engine starts the helper as root (needed for setns) and the helper drops to the ordinary 501:20 identity. Without the flag the argv and Engine user are byte-for-byte what they were.
• WorkspaceApi::exec now derives the namespace from the owner's SandboxCreate.setup, so a private-ns Sandbox cannot run that public path in the initial namespace.
• DaemonSetup::check requires the socket path for the private-namespace mode; the pid file is optional evidence. The socket path is saved in the sandbox record (nsExecSocket), so every exec of a running Sandbox is wrapped consistently.
• Upgrade: a live Sandbox whose record predates the socket (nsExecPidFile without nsExecSocket) is created by a daemon that never bound one. workspace.exec refuses it with code Unsupported and a message to close and recreate the Sandbox — no derived socket that cannot work, and no silent initial-namespace run.
• One entry path: layerfs_sandbox::DAEMON_ENTRY now feeds the create request, the observed-topology check and the helper argv, so the three cannot drift.

What and why

workspace.exec is a Docker Engine exec: the Engine starts bash -c <command> in the container's initial mount namespace, and the filesystem daemon is not involved (engine/crates/layerfs-node/src/exec.rs, engine/crates/layerfs-api/src/sandbox/exec.rs). The private-mount-namespace prototype (644556d0, issues #3/Ephemeral-AI-Lab#499) makes the daemon unshare(CLONE_NEWNS) at startup and mount FUSE only there. That keeps Workspace mounts out of the namespace Engine execs and docker exec enter, but it also means workspace.exec no longer sees the Workspace.

This PR is the follow-up: a small hidden setns helper so workspace.exec runs inside the daemon's private mount namespace. The Engine still performs the Exec, streams the output, and owns cancellation and the exit code; only the entry process changes, and only for a Sandbox that opts in.

What changed

• layerfs-daemon ns-exec (hidden subcommand, application/ns_exec.rs): joins the descriptor the daemon serves, re-enters the mount directory, drops the root entry identity to the ordinary command identity, and execs the command. Every refusal (missing/refused socket, no descriptor, a non-namespace descriptor, setns/chdir/drop failure) prints layerfs-daemon ns-exec: <reason> and exits 125 without running the command; the host sees the exit code and stderr.
• application/ns_fd.rs: the 0600 socket and its root + same-pid-namespace SCM_RIGHTS server described above.
• Daemon startup (application/cli.rs, layerfs-fuse/src/mount/syscalls.rs, layerfs-bridge/src/daemon_setup.rs): DaemonSetup gains an opt-in private_mount_ns flag (config record LFSD\x06). When set, before the listener and any worker thread, the daemon takes unshare(CLONE_NEWNS) + MS_REC|MS_PRIVATE on /, writes daemon.pid atomically, and starts the fd socket. When off, no namespace, pid file or socket exists.
• Exec wiring (layerfs-api/src/sandbox/exec.rs, layerfs-sandbox/src/backend/docker/request.rs, layerfs-node/src/{sandbox,discovery,registry,exec}.rs): the wrapper argv is built from the Sandbox's saved socket, and ExecRequest.root_launcher starts the helper as root (the Engine user becomes 0:0 only for wrapped Execs).
• Opt-in: deploy.privateMountNs or LAYERFS_PRIVATE_MOUNT_NS=1 when the Sandbox starts; off by default.

Tests run

Environment: Docker arm64, rust:1.85.1-bookworm, --cap-add SYS_ADMIN. The committed tree was copied into a Docker volume and every command ran from there, so a stale bind-mount view could not affect the result.

cargo check -p layerfs-daemon -p layerfs-api -p layerfs-sandbox -p layerfs-node -p layerfs-bridge --release
    Finished `release` profile ... exit=0

cargo test -p layerfs-node --lib
    test result: ok. 3 passed; 0 failed
    (record -> mount_namespace() -> wrapped argv; pid-only record requires recreation; plain command unchanged)

cargo test -p layerfs-bridge --lib daemon_setup
    test result: ok. 2 passed; 0 failed

cargo test -p layerfs-api --lib sandbox::exec
    test result: ok. 2 passed; 0 failed

cargo test -p layerfs-daemon --lib application::
    test result: ok. 7 passed; 0 failed

cargo test -p layerfs-daemon --test ns_exec -- --test-threads=1
    test result: ok. 11 passed; 0 failed
    a_command_joins_the_namespace_served_over_the_daemons_socket ... ok
    the_socket_command_starts_in_the_served_namespaces_directory ... ok
    the_socket_command_runs_as_the_identity_the_helper_drops_to ... ok
    a_missing_socket_refuses_and_runs_nothing ... ok
    a_socket_that_sends_nothing_refuses_and_runs_nothing ... ok
    a_socket_that_sends_another_file_refuses_and_runs_nothing ... ok
    a_helper_without_a_namespace_source_refuses_and_runs_nothing ... ok
    a_helper_in_another_pid_namespace_is_refused_by_the_daemons_socket ... ok
    the_daemons_socket_is_private_replaces_a_stale_path_and_sends_one_message ... ok
    the_daemons_socket_refuses_a_non_root_peer ... ok
    the_daemon_serves_its_socket_and_pid_file_only_when_asked ... ok

cargo test -p layerfs-sandbox --test engine_protocol
    test result: ok. 17 passed; 0 failed

cargo fmt --all -- --check
    clean

The integration test builds a real private namespace with unshare -m --propagation private, mounts a tmpfs + marker visible only there, and serves that namespace's fd over a test socket; read and chdir succeed only after setns, and the three refusal tests run a command that would touch a file and assert it does not exist. The socket tests cover mode 0600, stale-path replacement, exactly one SCM_RIGHTS message (one byte, one fd, then EOF), a non-root peer, and a peer in another pid namespace. The CAP_SYS_ADMIN/CAP_SYS_PTRACE premise was checked directly: docker exec -u 501:20 nsenter -t <pid> -m fails with Permission denied, while root succeeds.

Sealed E2E bundle: /Users/reina/Project/.layerfs-worktrees-nsexec/layerfs-nsexec-sealed-200d21420d8f/artifacts/build-linux-e8640431b70544b8bba0cee2e6bae72f (manifest.json sha256 2f249952f7dfc91f1b0639569085653616dc4536e706106548c00a1a0c2998ae; layerfs-daemon sha256 d393f9173fc81823d66114b8fa9d2e9293f2d972386a7742cf03cf1cdb4e76ce; layerfs.node sha256 ed013154d5683be3f2966017b8e54b09bf4b2a53ae3ed7f7337fc5cf23a7c013), built by tools/worktree.py register/prepare/build --platform linux --profile release --jobs 6 from a clean detached worktree at 6554e7fd.

Not exercised

• No real Sandbox end-to-end in this revision. The rebuilt bundle has not been rerun against the E2E worker yet; that is the next step and the PR stays draft for it.
• Daemon serving its own socket is tested directly (mode, credentials, one message, drop); the daemon's full startup (Store/FUSE) under the flag is not part of this test set.
• docker exec still cannot see the Workspace mount; only workspace.exec is routed through the helper.
• No macOS run: the daemon and these tests are Linux-only.
• The dsh-layerfs integration is unchanged; enabling this in a deployment needs LAYERFS_PRIVATE_MOUNT_NS=1 in the SDK worker and the matching daemon binary installed in the Sandbox image.
• No performance measurement in this revision.

Limitations

• Prototype, opt-in, not product behaviour. Hidden subcommand, default-off config flag.
• Root entry process. Joining the namespace needs a capability only root holds. Wrapped Execs start as root and drop to 501:20 in the helper before exec'ing; that widens the entry privilege for wrapped commands and needs a security review before production use. An unwrapped Sandbox keeps the old non-root entry.
• CAP_SYS_ADMIN dependency: the Sandbox container has it and its seccomp profile only denies CLONE_NEWUSER; a stricter runtime would refuse startup or exec.
• Mixed versions are deliberate refusals. New node with an old container: the record lacks nsExecSocket, so workspace.exec refuses Unsupported and asks to recreate the Sandbox. Old node with a new container: the old argv names --pid-file, which the new helper no longer accepts, so it exits 125 with a usage message. Both keep the command out of the wrong namespace; neither is silently unwrapped.
• Engine WorkingDir: the Engine must enter the mount directory in the initial namespace. It does while a Workspace is mounted, because the daemon creates the mountpoint directory in the container's root filesystem; a real product change should prove this in the host wiring and a real-topology test.
• A real product change still needs: deployment wiring for the flag, an image that ships this daemon, a real-topology test (mount + exec + cancel + exit code), and the privilege review of the root entry.

Refs #3, tokenaissance/layerfs#499

Closes #3.

[mengmengmengqiang]

Superseded by the upstream PR Ephemeral-AI-Lab#499 (same head commit 386f1b2); this fork PR is kept only as a mirror and is closed so the upstream PR is the single review surface.

[mengmengmengqiang]

Reopened per request: kept as the fork-side mirror of the upstream PR Ephemeral-AI-Lab#499 (same head 386f1b2).