ENOSPC recovery: corrupted accounting failed open, autonomous dispatch continued, and safe same-campaign recovery required manual intervention

[fanyangCS]

Summary

During an OpenPAI security-remediation campaign on September 17–18, 2026, filesystem exhaustion caused partial journal writes and failed call finalization. Argus then admitted further provider work because malformed accounting records were silently omitted. Planner waiting was not a durable safety stop, and a successor could run while the original item retained a stale running claim.

This issue targets lbx154/Argus runtime integrity and recovery, not OpenPAI application code. We recovered the same campaign using reviewed local fixes, evidence-preserving journal recovery, explicit acknowledgement of unknown liabilities, and supported ownership reconciliation. Stored execution receipts record a real engineer start at 02:08:02 UTC on September 18. That establishes resumed execution, not campaign completion.

Affected source: 310bc3bc, PR130-only baseline, reporting v0.1.7. Recovery source: 7b11f04b, a reviewed local build, not an upstream merged fix or published release. Local patches and sanitized reproductions are available on request.

What failed

1. Confirmed incident trigger: storage exhaustion damaged persistence

ENOSPC interrupted usage/event appends and both usage persistence and unknown-cost settlement for an interrupted call. Preserved bytes show:

• An incomplete usage-record prefix immediately followed by a complete later priced record on the same physical JSONL line, without a separating newline.
• A truncated completion event concatenated with a complete reservation event.
• A separate truncated planner-start event.

The complete priced suffix was corroborated by a canonical usage event and retained provider billing records. The interrupted call lacked a complete billing receipt; zero observations were not evidence of zero charge.

Large regenerable search-index caches contributed storage pressure and were subsequently cleaned up; evidence does not establish them as the sole cause of exhaustion. Freeing space did not repair damaged evidence.

2. Confirmed enforcement failures, with attribution limits

Fail-open accounting. The real usage reader skipped the malformed physical line, hiding both the incomplete call and the complete priced suffix. An isolated incident-copy replay returned an empty admission reason and actually obtained a synthetic reservation through reserve_call_budget, without invoking a provider. This was a failure inside normal admission, not a transport bypass.

Lossy reconciliation risk. Reconciliation worked only on parsed rows and could rewrite the ledger without malformed bytes when another row changed. The observed replay updated zero rows and retained the damaged journals; destructive rewriting is a source-established conditional risk, not a claim that it occurred in this incident.

Failed finalization did not durably block dispatch. Usage-write and settlement exceptions were logged without a durable failed-finalizer barrier. An unrecorded reservation could disappear when its owner PID died and observations were zero; old next-day projection also discarded retained obligations/acknowledgements in copy tests. These are reproduced loss paths, not evidence that the live recovery erased those liabilities.

Soft waiting was mistaken for containment. Unchanged-input suppression expires after 1,800 seconds. A subsequent durable wait also woke when a watched campaign file's revision changed, without checking whether accounting repair had happened. Timer expiry is a high-confidence explanation of the first retry, not a captured historical branch trace. The later revision_changed wake is recorded directly; an observation-only edit was a strongly supported contributor, not an exclusively proven writer attribution.

Continuation ownership was incomplete. Serial claiming admitted a successor alongside the original running row. Using the original item's ID as the successor's node_key did not establish exclusive ownership. This produced duplicate lifecycle claims; it does not demonstrate two simultaneously executing engineers.

Fix reviews also exposed canonical-owner propagation gaps in alternate reviewer/supervisor callers and failure-result handling weaknesses. These are review-discovered coverage defects, not established causes of the original ENOSPC incident.

Recovery chronology — UTC

Time	Action or observation
Sept 17, by 10:25:36	Usage persistence and unknown settlement failed with ENOSPC; damaged journals and an unfinalized reservation remained.
11:15 / 11:20	Planning restarted; the later wake explicitly recorded a watched-file revision change.
11:24–11:52	A successor engineer and then an actual reviewer ran; the original claim remained stale.
11:56:51	Continuous planning was disabled. This was partial containment, not a universal no-new-calls fence or full writer stop.
Sept 17–18	Separate execution/accounting fixes and their integration were reviewed locally; storage cleanup restored headroom.
Sept 18, 00:28–00:37	Workflow-only checkpoint fallback was rehearsed while retaining current accounting. It still failed integrity/admission and was not committed live.
00:33:05, during that attempt	Graceful legacy-daemon stop completed after evidence preservation; the legacy web writer was also stopped and its autostart disabled. Subsequent checks found the identified provider children gone.
01:29–02:01	Bounded incident recovery was implemented and reviewed. The first candidate failed destination-path tests; the corrected revision passed the bounded rereview.
Before 02:08	Reviewed recovery and authorized liability acknowledgements were applied; the stale original was skipped through a supported operator API, preserving the genuinely reviewed successor.
02:08:02	Stored provider-start evidence records an engineer running in the same campaign, advancing to the alert-parser mission.

The debt-preserving owner-stop obstacle is resolved, not a remaining blocker. Writer checks used same-user visibility, not privileged-process attestation.

Why rollback alone was insufficient, and what worked

An older workflow checkpoint cannot erase newer obligations. Workflow-only restoration retained corrupt journals and failed admission; restoring old accounting would risk dropping current charges and acknowledgements. A new project or clean ledger would hide the problem.

The inspected runtime initially lacked an incident-repair API for this damage. The successful local recovery added a bounded prepare/review/apply transaction:

1. Archive original journals and damaged segments; bind inputs and projected outputs to a reviewed manifest with content-hash compare-and-swap checks.
2. Recover only independently corroborated complete records, counting the known priced receipt exactly once. Preserve unreconstructable fragments without inventing missing fields.
3. Carry the failed reservation into an explicit unresolved-unknown obligation with original identity, provenance and lower bounds—not a fake active lease, successful settlement or zero-cost usage row.
4. Block admission with a pending transaction marker; support crash resumption and committed duplicate no-op. Use fsynced replacements and descriptor-relative, no-follow path handling. The initial byte-only implementation failed review because unchanged bytes behind a redirected parent path could be written outside the intended root.
5. Preserve current acknowledgements, configuration and budgets; reconcile workflow ownership separately from financial recovery.

Once corruption was repaired, the full global accounting gate exposed 148 older unknown calls across 34 projects. These are actual aggregated counts, not new calls caused by this incident. They had been masked by the earlier integrity failure. The operator explicitly accepted imperfect historical accounting through supported estimated-liability acknowledgements: 149 new decisions including the recovered planner, while preserving the prior decision. All unknowns remained unknown; no known charge was reduced and no budget reset or policy bypass was needed.

Minimal safe reproduction and validation

Proposed synthetic reproduction, not a CLI command or a claim of newly executed tests: use an isolated temporary state directory, deny networking, and replace only final provider transport with a counter/fake.

• Create schema-valid synthetic usage and reservation fixtures using the version's normal writers.
• Inject OSError(ENOSPC) during append/finalization, or seed a truncated prefix followed immediately by a valid record. Do not fill a real filesystem.
• Call the real reader, reconciliation and reservation entry points. Require typed refusal before transport and byte preservation.
• Simulate owner death and next-day reads. Require retained unknown obligations and acknowledgements.
• Advance waiting beyond 30 minutes or edit a watched observation file. Require that no safety block clears and no new transport starts.
• Attempt a same-origin successor while the original claim remains active. Require explicit, audited reconciliation before eligibility.

Retained validation: preserved incident copies showed baseline admission versus patched refusal, without provider calls. The final recovery rereview records 60 targeted tests and 648 selected compatibility tests passing; these suites overlap and must not be summed. Two unchanged destination-path safety probes failed on the rejected recovery parent and passed on the revision. Full-store recovery plus the authorized historical batch admitted and released a real test reservation without transport. The later live provider-start receipt is separate evidence of actual execution.

Acceptance criteria / requested upstream behavior

• Every provider admission rejects corrupt or incomplete accounting evidence; reconciliation never silently discards it.
• Failed persistence/finalization retains unknown liability across PID exit, restart and midnight; missing telemetry never proves free execution.
• Integrity stops are durable and enforced across canonical project callers; timers, observation edits and monetary acknowledgement cannot clear execution ownership or failure state.
• A supported, previewable recovery/cutover path preserves originals, current debt and later workflow results, rejects changed inputs/escaped paths, and resumes safely after interruption.
• Same-campaign continuation has one legitimate owner, preserving failed original attempts and real successor review provenance.
• Global historical blockers are distinguishable from current-campaign corruption, with explicit scoped decisions rather than silent forgiveness.

Remaining limitation: the existing acknowledgement API can commit state before its audit append; an injected crash exposed a missing-audit gap that its duplicate path does not repair. The recovery wrapper detects and stops on this condition rather than fabricating an audit. The recorded live batch had exactly one audit per target, but arbitrary intra-call crash recovery is not certified.

Historical billing completeness and broader campaign review/publication remain unclaimed. Our delayed updates were a separate assistant/OpenClaw coordination failure: silent observer success is not a notification receipt. This is not attributed to Argus; no parallel monitor-repair claim is made.

[Yushangjinghong2]

The runtime-integrity core of this report is addressed in #143 (commit eec8861, merged to dev). The remaining acceptance items are listed at the end; this issue stays open for them.

Implemented:

• Fail-closed accounting reader. argus/core/usage.py reads usage.jsonl through one strict physical-line reader and raises UsageJournalIntegrityError on malformed lines instead of skipping them, including a truncated record prefix spliced onto a complete later record on the same line. Blank lines stay benign; a partial final line without newline is tolerated only while another writer currently holds usage.lock; any unparsable line followed by more data is corruption. The complete suffix's call_id is exposed for the repair report only; the block stays until the bytes are repaired.
• Admission refuses before transport. reserve_call_budget and cost_admission_reason refuse with accounting_integrity: corrupt_accounting_journal: <file> line <n> (reason code also on the budget.reservation.denied audit row). The global read no longer treats one project's corruption as ignorable.
• Reconciliation preserves damaged bytes. Both reconciliation rewrites go through the strict reader and refuse to rewrite a journal that contains malformed lines; the file and no temp files are left behind.
• Failed finalization retains unknown liability. When an unknown-cost settlement fails (OSError such as ENOSPC, state error, busy lock, untracked reservation), a durable marker is written under <global_root>/cost-control.failed/<reservation_id>.json (O_EXCL, idempotent) and folded into unresolved cost on every admission, snapshot and acknowledgement read, so the block survives owner PID exit and restart. It is retired only by an operator acknowledgement or a later settled priced receipt. No zero-cost rows or synthetic completions are created.

Tests: tests/core/test_accounting_integrity.py (six cases covering the four scenarios above plus busy-lock retention and marker retirement).

Not addressed here, still open: the prepare/review/apply recovery transaction with manifest compare-and-swap; durable waiting (timer expiry and revision_changed wakes must not clear an integrity stop); same-campaign continuation ownership; the acknowledgement API committing state before its audit append. A reservation whose owner dies before any settlement attempt is still pruned as before.