Fleet role finalization and closed receiver continuation

[forhappy]

Summary

Continue docs/fleet-operations-plan.md after merged PRs #37 and #56. Minion is the canonical executable at crates/cellule-host/minion.

• Wire role settlement and Finalize through the existing host drain owner, with exact boot withdrawal and durable retirement before Completed. Add writer, reader, live-follower and failed-owner maintenance scenarios.
• Retain bounded receiver continuation routes and immutable recovery basis/evidence. Recover lost Recovering/Serving claims through canonical failed-session proof and the existing native takeover path, preserving original errors, acknowledged receipts and unresolved movement charges.
• Add the runnable receiver-loss command, controller reconstruction, dropped replies, interrupted publication/materialization and inherited sealed-suffix fault cases. Keep one canonical acquisition/recovery path and preserve an ordinary acquisition winner.
• Refresh native SettleRoles after a committed result's reply is lost. Preserve the original accepted envelope/key and validate the fresh opaque proof at the current request's head/registry.
• Add same-claimant reconstruction and new-claimant replacement after real controller lease expiry, with old-controller fencing, exact-root and original request-result readback, final withdrawal and joined ledgers.
• Share fresh signed directory traversals across bounded follower windows. Keep existing cursor/digest bytes and exact expired/fenced obligations. Whole-set recheck failure invalidates all confirmations without restamping originals.
• Remove redundant authentication at immutable canonical codec and historical enrollment fingerprint boundaries. Fresh canonical reads still authenticate every record. Add byte-contract, signature-count, forgery, changed-record and full recheck regressions.
• Remove the temporary diagnostic probes after reproducing and fixing the hosted observer deadline failures. Original profiles, deadlines and assertions are unchanged.

Latest checkpoint: prepared Blob admission and fixture CI race

Prepared Blob command execution through the configured client now uses the same original store admission and retained native owner. Clones and restored snapshots refuse new dispatch after closure; accepted execution survives caller loss. Convenience mutation retains its one whole-operation owner across staging and dispatch, using the same native command path. Request identity, input/digest, snapshot bytes, resolution and durability are unchanged.

Add three public regressions for closed prepared clones/restores, cancelled accepted execution behind real SQLite work, and exact open replay with one upload. Resolve the original upload outcome bytes/sequence after closure. Reproduce and fix the full Rust CI fixture race: a prior reply can arrive before its supervisor releases ownership. Fixture setup now joins prior jobs before measuring the next paused job. The delayed-cleanup probe reproduced 2 versus 1 and passed after the test fix; the probe is removed and the one-job assertion remains exact.

Local Rust 1.97: eight Blob/Cron cases, five snapshot contracts and the typed application consumer passed; warning-denied lint/API docs, format, architecture/layout and document/SQL/peer gates passed. Final native qualification passed all 215 cases, warning-denied lint/API docs and static gates on exact snapshot f8174764; full new-head CI remains required. BlobInventory still blocks maintenance until Cell-scoped pins, complete global retention and uncertain remote outcomes are covered.

Earlier checkpoint: original Blob operation lifetime and minion CI fix

• Retain at most 64 original Blob operations after caller cancellation. Shared irreversible admission spans mutation staging/publication, range metadata and all part reads, and complete GC references. Preserve original native failure/panic sources. Forced Tokio runtime loss records unjoined work and refuses closure.
• Add CellNode::install_blob_artifact_store through the existing ordered facility drain; keep the configured provider and client capability on one shared owner. No separate scheduler or drain lane.
• Fix the reproduced minion CI failure: inconsistent native identity is correctly refused by observation construction. Preserve exact errors and expand corruption coverage from eight to fourteen cases, including coherent substituted fences.
• Final Blob qualification passed 206 cases, warning-denied host/runtime lint and API docs, and static gates on exact snapshot 7fd0a2e5.
• Minion qualification passed all 382 cases with unchanged profiles and deadlines, ten selected observation cases and the typed application consumer on snapshot 9dad2e91. This snapshot precedes the final forced-runtime-loss guard; full new-head CI remains required.

BlobInventory still blocks maintenance: local joining does not prove returned PreparedCommand lifetimes, Cell-scoped pins, complete global retention, remote outcomes or safe movement. These remain plan work.

Earlier checkpoint: parallel maintenance cleanup regression

Fix the full Rust CI maintenance cleanup test. Default hosts intentionally share a process-wide disk budget; an unrelated held 4096-byte reservation reproduced the final disk-zero failure. Give each simulated donor/successor a distinct budget at unchanged default capacity, retain donor zero assertions, add successor disk-zero and require the unrelated reservation to remain intact. Production behavior is unchanged.

Exact native run 37267463895 passed two complete parallel runtime suites (231 passed/4 ignored each) and warning-denied lint on snapshot a3cfd497.

The retained earlier Compose campaign passed reader smoke/object-only routing and failed leased local expired-burst query p99 (2.13677x baseline; 3.059677 ms versus 1.431917 ms). Cause is unproven; gates remain unchanged.

Earlier checkpoint: busy SQL maintenance

Add canonical minion maintenance-busy: two continuous command lanes use the same public reconciler to cordon the donor, prepare receivers, dispatch native busy release, move all twelve Cells and finalize the exact boot. Every acknowledged request is resolved on its canonical successor with its original digest, sequence and result; an exact audit count detects duplicated or missing effects. Both lanes must be refused before their finite command bound. Client tasks join on success and failure before node/journal cleanup; startup failure retains its native source.

Fix two reproduced starvation paths. Exact quiescence/release now use the immutable activation fence while accepted publication owns the publisher. Advisory owner inventory retains that fence; the planner and reference observer can retain explicit maintenance demand for an independently rechecked writer after root advancement. Complete counts and ordinary movement still invalidate; receiver preparation precedes source quiescence, and native settlement/final publication/role barriers remain required. Wrong identity, missing cost, Blob and foreign-role guards remain blocking. The planner digest advances to v15 and binds the fence; persisted ID, root and action codec bytes are unchanged.

Local Rust 1.97: 29 selected regressions, production busy command, warning-denied host/runtime lint and API docs, format, architecture/layout, document and SQL/peer gates passed. Production readback covered 88 acknowledged commands plus twelve original receipts, final counts [0, 6, 6], and all three joined/retired boots. The exact sixteen Rust paths passed native job 111619387502 on snapshot e3f6dac and Rust 1.99: all 36 selected regressions, three production commands and warning-denied host/runtime lint. Native busy readback preserved all 384 acknowledgements, twelve original receipts and exact audit rows; artifact busy-maintenance-37264858825-1 retains source/binary/environment/limits and raw results. This is finite in-process SQL evidence; full W4–W10 qualification remains required.

Earlier checkpoint: combined reader/follower maintenance

Add canonical minion maintenance-roles using the same four managed boots, supervisor, signed native peers and public reconciler. A donor holds both a reader and a foreign follower tail. Follower policy alone cannot authorize Finalize; native reader evacuation refuses an absent selected replacement and preserves the original Established open view. Both replacements must establish readers before joined original retirement and immutable reader policy. Fresh complete observation must prove both roles before SettleRoles and Finalize. Check both replacement receipts, original request history, renewed writer acknowledgement and [1, 0, 0, 0] ownership; join all four nodes and eleven enrollment rows.

Native combined job 111610477201 passed on exact snapshot 8f64d2f: 19 selected regressions, both production commands and warning-denied Clippy on Rust 1.99. Local Rust 1.97 regressions, command, API docs and document/boundary gates passed. A reproduced stack overflow was fixed by splitting reader phases and boxing complete reconciliation at the scenario boundary; final runs use normal stack limits with no probes. No assertions, profiles or deadlines changed.

The earlier parent f9a476e complete Compose campaign passed, including original constrained reader scaling and leased/object-only routing. The earlier intermittent availability failure remains unexplained. This adds an in-process role combination; full primitive/process/provider qualification remains required.

Earlier checkpoint: executable live-follower maintenance

Add maintenance-follower to canonical minion using four managed boots and the existing durability supervisor. Zero local writers cannot hide a retained foreign follower lane. Missing replacement policy remains blocking; canonical writer drain covers the original tail, both original member retirements are confirmed, and the installed epoch 2 ensemble has two eligible members. Canonical acquisition resumes the writer on its original node and acknowledges another command. Fresh immutable policy and complete role observation authorize native Finalize and exact withdrawal. Canonical-root readback covers both acknowledgements and preserves the original request digest, expiry, sequence and stored outcome. Cleanup joins four nodes, both ensembles and every boot, with final writer counts [1, 0, 0, 0].

Native follower job 111605650581 passed on the exact 11 changed Rust files: 18 selected regressions, production command and warning-denied host Clippy on Rust 1.99. Local executable, observation/reader/balance checks, host API docs and document/boundary/contract gates passed. The finite adapter retains two epochs and exact close barriers; it provides no failed-owner recovery capability or external process/provider qualification.

Parent f9a476e full Rust workspace job succeeded with all 376 minion cases, provider/process smoke, local/replica LTX and lint gates. Follower/object proof and other quality checks passed. Its Compose smoke and reader-scaling steps also passed; entity/routing measurements remain live. The earlier constrained-reader availability failure remains unexplained: two independent diagnostic attempts passed original 3/5/10/20-node profiles with all 300 scheduled writes committed per scale. No production runtime fix is claimed from passing reproductions. The new published head requires complete CI.

Executable reader maintenance

Add maintenance-reader to canonical minion. The public driver cordons a managed reader boot, preserves its usable reader while replacement policy is missing, opens a canonical selected replacement, publishes immutable evacuation evidence, settles the full role graph and finalizes native shutdown/withdrawal. Verify replacement value 29, the original stored mutation result, unchanged writer ownership and all joined runtime/enrollment ledgers. Share the signed native peer adapter with existing reader tests. Wait for real selection to observe the cordon through its bounded cache; sign only actual samples matching current local admission mode.

Native reader job 111598959724 succeeded on the exact Rust source: the production CLI, 1 executable regression, 10 existing reader fault cases, 2 reader observer contracts, 3 follower observer contracts and warning-denied host Clippy.

The parent 9257b18 Compose smoke failed in unchanged constrained mixed-reader load with ReplicaUnavailable. Driver/arrival/container/provider evidence is retained in its artifact; cause is not yet proven. Complete current-head CI and this availability regression are the highest next priorities.

Role-result publication and cancelled owners

Fix a reproduced deadlock after unpublished native SettleRoles results. A failed retry returns the original journal error and retires only the joined read-only role proof. The next pass requires complete fresh evidence at the current head/registry; original acceptance and historical results remain unchanged. Physical-effect receipts remain retained without reexecution.

Add before-write and after-commit faults under same-claimant renewal and replacement after actual lease expiry, plus cancellation while the original native owner is paused. Exact duplicates join that owner and competing proofs remain refused until it finishes. Cases verify Closing/Completed, Stopped, withdrawal, exact-root and original request-result readback and joined ledgers.

Native Ubuntu publication job 111594331530 passed on the exact Rust source with no probes: 6 restart cases, 2 cancelled owners, 3 unchanged observer cases, 7 physical-action retention cases, 3 opaque-proof refusals, and warning-denied host Clippy on Rust 1.99. Original profiles, deadlines and assertions are unchanged.

Parent ab5eb0b full Rust workspace job succeeded with all 369 minion cases, provider/process smoke, local/replica LTX and lint gates. Complete source hashes and dated scope are in the progress record.

Remaining plan work

W4–W10 remain incomplete. Highest priorities are current-head CI and unexplained constrained-reader availability; busy primitive and external-owner maintenance, successive boot/lineage and remaining role faults including external Cron/Blob owners; recorded W9 process/provider, load and mixed-version campaigns; and exercised W10 rollout/rollback/recovery runbooks. In-process closures do not qualify OS crashes or external job supervision.

Main synchronization

This branch includes main at 80c4fd9. PR #37 is already merged; its reported conflict files have no conflict markers here. No unresolved index entries remain.