fix(daemon): integrate stalled unmount recovery with exact abort custody (#496)

.agents/skills/layerfs-js-sdk/references/methods/workspace.md

@@ -282,7 +282,7 @@ Unmounts a Workspace. A Sandbox left with no Workspace closes itself two minutes
 | --- | --- |
 | `ws` | The Workspace from `mount` |
 | `discard` | `true` to drop uncommitted changes instead of refusing |
-| `force` | `true` to abort the Workspace's kernel connection instead; drops uncommitted changes. Use it only for an unmount that stalled (below) |
+| `force` | `true` to abort the Workspace's kernel connection instead; drops uncommitted changes. A normal unmount that stalls at `Join` ends itself with one abort write, use `force` for a live connection with a usable abort control, not to replay an attempted abort |
 
 **Returns:** `{ kind: "Unmounted" }`, or with `force`
 `{ kind: "ForceUnmounted", abort, detach, cleanup }`.
@@ -294,7 +294,8 @@ Unmounts a Workspace. A Sandbox left with no Workspace closes itself two minutes
 | `Dirty` | Uncommitted changes; `detail.changed` lists paths | Commit them, so the step is recorded; discard only a Workspace you're abandoning |
 | `Busy` | A background process still runs there | Stop it (`backend.daemon.process.cancel`, in the observe skill), or wait for it |
 | `NotMounted` | Already unmounted, or the wrong Workspace | Use the Workspace from `mount` |
-| `Failed` with `TeardownCustody { stage: Join` in the message | The mount is gone but its connection did not end, so the unmount stalled; every later plain unmount answers the same | `workspace.unmount(ws, true, true)` once |
+| `Failed` with `TeardownCustody { stage: Join` in the message | The mount is gone but its drain did not finish; the custody records whether an abort was attempted. Later unmounts retain that custody | Preserve the custody; do not retry an attempted abort. Replace the Sandbox if its connection cannot be ended |
+| `Failed` with `TeardownCustody { stage: Abort` in the message | The one abort write failed or returned a short count; the original cause and abort facts are retained | Preserve the custody and replace the Sandbox; neither normal nor forced unmount repeats the abort |
 
 ```js
 // ws: from workspace.mount. Every mount creates a new Workspace; never hard-code one.

api/agent/src/workspace/unmount.js

@@ -1,7 +1,9 @@
 // workspace.unmount: ends the Workspace; `discard` drops its uncommitted
-// changes. `force` aborts its kernel connection instead, also after a normal
-// unmount stalled at Join, and drops uncommitted changes. A Sandbox left with
-// no Workspace closes once it has been idle.
+// changes. A normal unmount whose drain stops at Join makes one abort write
+// on the already-detached connection and ends there. `force` aborts the
+// live connection when an abort control is usable, and drops uncommitted
+// changes. An attempted abort is never repeated. A Sandbox left with no Workspace closes once it
+// has been idle.
 import { invokeClient, workspaceArgs } from "../client.js";
 export function unmount(client, ws, discard = false, force = false) {
   return invokeClient(client, "workspace.unmount", { ...workspaceArgs(ws), discard, force });

core/docs/architecture/76-native-mount-session.md

@@ -189,19 +189,27 @@ its existing Commit-custody admission and then, for an attached Workspace:
 3. **Drain.** [`drain`](../../crates/layerfs-fuse/src/session/drain.rs)
    establishes that every created loop is joined, the owner and watcher
    returned, no received, admitted or retained request remains, and releases the
-   lane. `drain_wait` is an observation deadline.
+   lane. `drain_wait` is an observation deadline. A stop at `Join` means the
+   connection outlived its mount (another mount namespace holds a copy of it,
+   #426 and #460): the unmount makes one abort write on the already-detached
+   connection and drains again, so the stalled connection still ends. A write
+   that is not accepted whole, or a second stop, keeps the entry `Retained`
+   exactly as any other stop does.
 4. **Revocation.** `NativeJob::Revoke` logically revokes the engine mount.
 5. **Close and registry removal** through the existing terminal Close.
 
 Any step that is not established leaves the entry `Retained` with its stage
-(`Detach`, `Join`, `Owner`, `Requests`, `Lane`, `Revoke`, `Close` or `Registry`),
+(`Detach`, `Join`, `Abort`, `Owner`, `Requests`, `Lane`, `Revoke`, `Close` or `Registry`),
 the observer and the original evidence. The reply is `Retained` with
 `TeardownCustody`; it is not reported as an uncertain mutation and nothing is
 replayed. Later Attach or Unmount on that entry returns the same custody, and
-so does ForceUnmount, except after a detached drain stopped at `Join`: that one
-[stalled normal unmount](80-forced-teardown.md#stalled-normal-unmount) is ended
-by one forced unmount. The normal path never uses the abort control and never
-sets the terminal fence.
+so does ForceUnmount, except for the one kind of stop that can still be ended:
+a detached drain stopped at `Join` whose abort control is bound and unused
+([stalled normal unmount](80-forced-teardown.md#stalled-normal-unmount)). The
+normal path uses the abort control only there. An accepted abort sets the
+terminal service fence before the second drain. A failed or short abort reports
+`Abort` with its original errno or byte count; later drain, Revoke and Close
+failures retain the recorded abort effects in `TeardownCustody.forced`.
 
 ## Forced unmount
 

core/docs/architecture/80-forced-teardown.md

@@ -14,8 +14,10 @@ The daemon half is [`control/force.rs`](../../crates/layerfs-daemon/src/control/
 The connection half (the abort write, the terminal fence and the forced drain)
 belongs to Fuse and is described in
 [native request service](75-native-request-service.md#terminal-fence-and-gated-ports).
-The normal path is unchanged; see
-[native mount session](76-native-mount-session.md#normal-unmount).
+A normal unmount that stalls at `Join` makes the same single abort write
+itself ([stalled normal unmount](#stalled-normal-unmount)); `ForceUnmount`
+remains for the one stalled custody that could not be aborted, and for
+aborting a live connection without waiting for its processes.
 
 ## Request and replies
 
@@ -122,11 +124,24 @@ A normal unmount can detach successfully and still stop at `Join`: when
 another mount of the same superblock outlives the detach, for example a copy
 in a mount namespace made by a process outside the Sandbox, the kernel never
 ends the connection, so its receive loops stay blocked in their reads
-(issues #426 and #460). The entry keeps that undrained session as `Retained`
-at `Join` with no forced facts, and every later normal `Unmount` returns that
-custody unchanged.
-
-`ForceUnmount` is the one way out. It takes the session back and makes the
+(issues #426 and #460). The unmount now makes one abort write itself
+(`NativeSession::abort_detached`) and drains again, so the connection ends and
+the unmount completes with `Unmounted`.
+
+When no abort control is bound, the entry stays `Retained` at `Join` with
+no abort facts. A failed or short write stops at `Abort`, preserving the
+original errno or count in `detail` and the disposition in `forced`. An accepted
+write followed by a failed second drain keeps that drain's stage and records
+`abort: Written`. Revoke and Close failures also retain these facts. Every
+attempted abort records `detach: Detached` and `commit: Absent`, since normal
+admission established no Commit custody before the plain detach.
+
+A later normal or forced unmount returns the retained stage, detail and abort
+facts unchanged. An unavailable control cannot be forced, and a consumed
+control is never written again. Successful completion still replies `Unmounted`.
+
+`ForceUnmount` remains for a stalled custody that is still forceable (one
+whose abort was not attempted) and for a live connection. It takes the session back and makes the
 same sequence with two differences. Step 1 is `NativeSession::abort_detached`,
 which refuses unless the connection is detached and not yet aborted. Step 2
 makes no detach, because the normal unmount's plain detach is the

core/docs/issues/496/ABORT-CUSTODY-FIX-20261029.md

@@ -0,0 +1,93 @@
+# Normal unmount abort custody
+
+> **Status:** Dated implementation checkpoint; not release qualification.
+
+PR #496 at `3a0564b6bb235dc7a0a62d732a5d8ec58e0ad865` ended a normal
+unmount stalled at Join by attempting one abort and draining again. Its failed
+abort return still carried the pre-abort `Undrained.forced`, and normal daemon
+teardown supplied no abort facts to retained drain, Revoke or Close replies.
+
+## Change
+
+- Refresh the returned FUSE receipt from the session's recorded abort outcome.
+- Report short/failed writes at `Abort`, with the original byte count/errno.
+- Carry `Written` through a failed second drain, Revoke and Close. Normal
+  admission supplies `commit: Absent`; its completed detach supplies `Detached`.
+- Keep successful normal replies `Unmounted`. Keep control consumption, one
+  attempt, drain predicates and custody ownership unchanged.
+- Document that retained abort facts also apply to a stalled normal unmount.
+  Update the SDK guidance: an unavailable or consumed control is not forceable.
+- Record the expanded test's functional duration in the CI slow inventory while
+  retaining its existing Linux mount lane.
+
+No test was deleted. No dependency, wire encoding or persistence profile changed.
+
+## Regression evidence
+
+The new failed/short-write cases were run before the production fix. Both failed:
+`stage: Join`, `forced: None`, generic drain detail. The existing success case
+passed. The post-fix six-case binary passes:
+
+1. Copied mount namespace: normal unmount completes, with a Written abort receipt.
+2. Kernel-denied abort write: EIO and Failed remain in custody.
+3. Zero-byte abort write: exact count and Short remain in custody.
+4. An acknowledged write followed by another Join stop: Written remains in custody.
+5. A failed owner job before Revoke: Written remains in Revoke custody.
+6. An already-closed overlay: Revoke succeeds, Close refuses, Written remains in custody.
+
+The write failures use a test-only thread-local seccomp filter on the exact
+already-bound descriptor of the test's own connection. Case 4 uses seccomp user
+notification to simulate a successful write without ending the connection; it
+checks propagation after a second drain failure, not a naturally reproduced
+kernel failure. No descriptor is replaced, no other connection is inspected,
+and no test code writes fusectl. The copied namespace is released before final
+assertions, and subsequent normal/forced replies preserve the stage, detail and
+abort facts without another abort. Work counters remain live observations.
+
+An initial Close fixture used a missing-object read and expected quarantine.
+That expectation failed: definite missing-object reads do not quarantine the
+reader. The fixture was replaced with a public owner Close to stage its exact
+refusal, without weakening the expected custody assertion.
+
+## Validation
+
+Linux ARM64 Docker, Rust 1.85.1, locked dependencies, repository ARM64 flags;
+Disposable Store explicitly selected by the fixture. Builds precede execution.
+Every mounted invocation has a 100-second wall timeout; the FUSE/Bridge test
+invocation has a 90-second execution timeout and tooling tests a 120-second cap.
+These are functional tests, not a performance campaign.
+
+- `stalled_unmount`: 6 passed (21.28 s); the strengthened success receipt assertion
+  is additionally checked with only that case selected.
+- `forced_unmount`: 3 passed; `forced_unavailable`: 1 passed;
+  `fp21_detach_stages`: 2 passed.
+- `layerfs-fuse` and `layerfs-bridge`, all targets: 33 tests passed.
+- Warning-denying Clippy for daemon/FUSE/Bridge, all targets: passed; the final
+  strengthened stalled-unmount test is checked separately.
+- Product boundary: 956 production files passed; authorized fuser integrity
+  passed; 155 tooling tests passed, followed by 6 CI-selection tests after the
+  slow-inventory update.
+- Rust formatting, diff whitespace and Ruff 0.17.2 formatting: passed.
+  An optional Ruff lint invocation reported the pre-existing EXE001 on
+  `tools/ci_tests.py` (the parent tree has a shebang and mode 100644).
+- Durable: NOT_RUN — disabled by owner. No full SDK E2E or benchmark run.
+
+Machine-local append-only receipts, including the original failures, are under
+`~/.cache/layerfs-worktrees/layerfs-pr496-fix-8cc8b1c1603d/builds/`.
+The pre-fix receipt is `pr496-mount-stalled_unmount-7f1b9e7200234dcda9fc2312db1e4b87`;
+the complete passing six-case receipt is
+`pr496-mount-stalled_unmount-1b8071b206174205b677283a4ff02124`.
+
+## Production source size
+
+Production LOC: 102777 -> 102816 (delta +39).
+Core: 38395 -> 38395 (0); engine: 61723 -> 61762 (+39);
+API: 2659 -> 2659 (0). Reference absent (retired R9c; last counted 65,417
+at tree `498dd1917812`).
+
+Method: `tools/production_loc.py` blob
+`3341e4d0362c822b6d9a08419c432942b3a13800`, `--root <exact exported tree> --json`,
+against the first parent and final staged tree. First-party production Rust,
+runtime SQL and JavaScript only; tests, tooling, docs, generated and third-party
+code excluded. Exact tree IDs and the confirmed comparison are in the commit
+message. This is a source-size comparison, not a performance claim.

engine/crates/layerfs-bridge/src/control_mount.rs

@@ -148,7 +148,8 @@ pub struct TeardownCustody {
     pub work: Option<MountWork>,
     /// Bounded original cause description; never parsed to decide custody.
     pub detail: String,
-    /// Forced-teardown effects; absent from a normal unmount or Attach custody.
+    /// Abort effects, including a normal unmount's stalled connection; absent
+    /// when no abort was attempted.
     pub forced: Option<crate::control_forced::ForcedFacts>,
 }
 fn put_receipt(out: &mut Writer, value: &MountReceipt) -> Result<(), ControlError> {

engine/crates/layerfs-daemon/src/control/detach.rs

@@ -1,4 +1,7 @@
 //! Linux half of normal unmount: plain detach once, then the drain barrier.
+//! A drain that stops at `Join` makes the one abort write a forced unmount
+//! would and drains again, so a connection that outlived its mount is ended
+//! rather than kept (#426, #460).
 use super::{
     kernel::Kernel,
     registry::bound_mut,
@@ -7,11 +10,30 @@ use super::{
 };
 use crate::store::BoundWorkspace;
 use layerfs_bridge::control::{
-    Activity, ControlCode, ForcedFacts, MountPhase, TeardownStage, WorkspaceToken,
+    AbortDisposition, Activity, CommitKnowledge, ControlCode, DetachDisposition, ForcedFacts,
+    MountPhase, TeardownStage, WorkspaceToken,
+};
+use layerfs_fuse::{
+    mount::AbortWrite,
+    session::{Detach, DrainStage, Forced, SessionObserver},
 };
-use layerfs_fuse::session::{Detach, DrainStage, SessionObserver};
 use std::{fmt, sync::Arc};
 
+/// Abort effects after normal admission, which established no Commit custody
+/// and completed the one plain detach before attempting the write.
+pub(super) fn detached_abort_facts(forced: Option<Forced>) -> Option<ForcedFacts> {
+    forced.map(|forced| ForcedFacts {
+        abort: match forced.abort {
+            AbortWrite::Written => AbortDisposition::Written,
+            AbortWrite::Short(_) => AbortDisposition::Short,
+            AbortWrite::Failed(_) => AbortDisposition::Failed,
+        },
+        detach: DetachDisposition::Detached,
+        commit: CommitKnowledge::Absent,
+        fenced: forced.terminal_replies,
+    })
+}
+
 /// Terminal admission over a connection-owning entry. No daemon-side request
 /// count refuses here: the kernel's own reversible answer is the fence, because
 /// every blocked caller, descriptor, working directory and mapping holds a
@@ -84,21 +106,42 @@ impl Service {
         self.leaving(token, MountPhase::Draining);
         let drained = match session.drain() {
             Ok(drained) => drained,
-            Err(undrained) => {
-                let stopped = stage(undrained.stage);
-                let detail = format!(
-                    "connection drain stopped at {:?}: loops {:?}, work {:?}",
-                    undrained.stage, undrained.loops, undrained.work
-                );
-                // Detached yet unjoined: the connection outlived its mount,
-                // and one forced unmount may still end it.
-                let evidence = match undrained.stage {
-                    DrainStage::Join => Evidence::Stalled(undrained),
-                    _ => Evidence::Other(undrained),
-                };
-                return Err(self.keep(token, stopped, detail, Some(observer), evidence, None));
-            }
+            // Detached yet unjoined: the connection outlived its mount because
+            // another mount namespace holds a copy of it (an engine or VM agent
+            // outside the Sandbox). Make the same one abort write a forced
+            // unmount makes, then drain again; a connection whose control is
+            // unusable, or one that still stops, keeps its custody (#426, #460).
+            Err(undrained) => match undrained.end_stalled() {
+                Ok(drained) => drained,
+                Err(undrained) => {
+                    let facts = detached_abort_facts(undrained.forced);
+                    let (stopped, detail) = match undrained.forced.map(|forced| forced.abort) {
+                        Some(AbortWrite::Failed(errno)) => (
+                            TeardownStage::Abort,
+                            format!("abort write: {errno}; nothing further attempted"),
+                        ),
+                        Some(AbortWrite::Short(count)) => (
+                            TeardownStage::Abort,
+                            format!("abort write returned {count} of 1 byte; nothing further attempted"),
+                        ),
+                        abort => (
+                            stage(undrained.stage),
+                            format!(
+                                "connection drain stopped at {:?}: abort {abort:?}, abort_bound {}, loops {:?}, work {:?}",
+                                undrained.stage, observer.facts().abort_bound,
+                                undrained.loops, undrained.work
+                            ),
+                        ),
+                    };
+                    let evidence = match undrained.stage {
+                        DrainStage::Join => Evidence::Stalled(undrained),
+                        _ => Evidence::Other(undrained),
+                    };
+                    return Err(self.keep(token, stopped, detail, Some(observer), evidence, facts));
+                }
+            },
         };
+        let facts = detached_abort_facts(drained.forced);
         // Drain barrier holds: one fixed logical mark, then indexed retirement.
         match self.job(
             workspace.route(),
@@ -112,7 +155,7 @@ impl Service {
                     failure.wire().detail,
                     Some(observer),
                     Box::new((failure, drained)),
-                    None,
+                    facts,
                 ));
             }
         };

engine/crates/layerfs-daemon/src/control/serving.rs

@@ -58,8 +58,8 @@ pub(super) struct Kept {
     pub observer: Option<SessionObserver>,
     /// The original owner, exactly as the stopping boundary left it.
     pub evidence: Evidence,
-    /// Effects of a forced teardown as recorded where it stopped; absent for
-    /// a normal unmount or an Attach.
+    /// Abort effects as recorded where teardown stopped, including a normal
+    /// unmount's stalled connection. Absent when no abort was attempted.
     pub forced: Option<ForcedFacts>,
 }
 impl fmt::Debug for Kept {

engine/crates/layerfs-daemon/src/control/unmount.rs

@@ -78,7 +78,12 @@ impl Service {
         workspace: &BoundWorkspace,
         kernel: Receipt,
     ) -> Result<Success, Failure> {
-        let kernel = self.closed(token, workspace, kernel, None)?;
+        #[cfg(target_os = "linux")]
+        let forced =
+            super::detach::detached_abort_facts(kernel.as_ref().and_then(|done| done.forced));
+        #[cfg(not(target_os = "linux"))]
+        let forced = None;
+        let kernel = self.closed(token, workspace, kernel, forced)?;
         self.depart(
             token,
             Success {