Repository navigation
[draft] Observability: count and expose unmount Join stalls #2
Description
Draft — review before filing upstream. Small observability follow-up; no behavior change.
Problem
After Ephemeral-AI-Lab/layerfs#500 (the reworked Ephemeral-AI-Lab#496), a normal unmount that stalls at Join makes one abort_detached write and carries exact abort custody when it fails: TeardownCustody { stage: Abort|Join|Revoke|Close, detached, forced: { abort: Written|Short|Failed, detach: NotAttempted|Detached|Busy|Failed, commit: Absent|Published|Unknown, fenced }, work: MountWork, detail }. But:
- the caller sees this only in the failure reply; the daemon emits no structured log or metric of its own, and its container disappears with the Sandbox;
- a successful abort-on-Join still returns
Unmountedwithout the abort facts, so "how often did the abort path fire?" is invisible even though the latency (~5 s, ~1 s with fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) Ephemeral-AI-Lab/layerfs#497) hints at it; - without
LAYERFS_TRACEthere is no record at all.
Design (four layers)
The outcome is a value object derived from the new custody; the Host owns the use case; the daemon only renders one debug line.
| Layer | Content | Why |
|---|---|---|
| Entities | UnmountOutcome from the new custody: stage (Join / Abort / Revoke / Close), detached, forced.abort (Written / Short / Failed, with the byte count/errno in detail), forced.detach (NotAttempted / Detached / Busy / Failed), forced.commit (Absent / Published / Unknown), work (loops_exited / loops_joined / active / retained), waited — in the bridge/SDK, no I/O |
stable shape, no transport |
| Use Cases | record_unmount_outcome, count_unmount_outcomes, emit_stall_alert in the Host |
aggregation belongs where the process is durable |
| Adapters | daemon: one stderr WARN at the Abort stage; SDK: trace-line serializer with a structured custody field; Host: journal writer + counter exporter |
format only; no policy |
| Frameworks & Drivers | docker logs (ephemeral), stateDir/lines/*.jsonl (durable), Session plugin record, optional metrics backend |
sinks |
Arrows: runtime flow is engine -> SDK -> Host (outcomes move outward); source dependencies are the reverse — Host -> SDK -> wire schema, and the engine depends only on the wire schema, never on the Host. The daemon must not own counters or metric names.
Proposed changes (minimum)
- daemon: one structured WARN when the abort path runs: stage
Abort, abort disposition (Written / Short / Failed + byte count/errno), detached,MountWorkcounters, waited. Debug only — docker logs are not durable. - SDK: put the custody fields into the existing trace line as structured data on failure (stage, forced.abort/detach/commit, work, waited), not only the message string; and expose the abort disposition on the success path (
Unmounted) so fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) Ephemeral-AI-Lab/layerfs#497's grace path is countable — an optional field or a separate event. - Host: count from the journal (
stateDir/lines/*.jsonl) by stage/abort disposition; a counter only if metrics exist. - no logging on the normal path.
Not doing (anti-over-engineering)
- no metrics endpoint or OTel inside the ephemeral daemon;
- no event bus;
- no per-request log lines;
- no new interface abstraction until a second sink exists.
Acceptance
- injected Join stall: daemon WARN; trace line carries structured custody with the new fields; journal/count +1;
- successful abort-on-Join: the abort disposition is visible (new optional field on the success reply or a separate event);
- normal unmount: no stall event;
- documented schema; no wire-encoding change beyond the new optional field.
Diagram
Activity