Skip to content

fix(supervise): heal a released slot the sweep's process died before recording - #1265

Merged
drewstone merged 2 commits into
mainfrom
fix/heal-released-slot
Sep 16, 2026
Merged

drewstone merged 2 commits into
mainfrom
fix/heal-released-slot

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Closes the window 0.233.0's CHANGELOG stated and did not try to heal: a release sweep appends a retained child's environment-teardown receipt and then its terminal record, and a process that died between the two left an open cursor slot beside a destroyed environment. The next resume treated the node as interrupted, tried to recover an executor whose environment was gone or left it open forever, and every reader charged the ceiling the pool had already refunded.

Why a resume could not heal it before

The released record's seq is the cursor seq next() stamped on the delivery the driver saw, and it lived only in memory: no record for an open node carried it. A resume could invent one (and a reason, and a trace), which would place the record where the driver never saw it and claim things that never happened. So:

  • The reconciled record now carries the settlement the driver received (reason, infra, trace, outRef, providerModel, harnessTranscript), the withheld overspend, the cancellation source, and settledSeq, through the same field spread the terminal record uses.
  • On the root resume, healReleasedSlots walks the journal forest and, for a spawned node with no terminal record whose receipts after that record all read destroyed: true and name the environment its last admission named, writes the released record with the sweep's own builder at settledSeq and the reconciled at, then emits the sweep's agent.child event. The node is never interrupted; no recovery is attempted.
  • Fail closed: a destroyed: false receipt, an empty receipt set, a receipt for an unadmitted environment, or a reconciled record without settledSeq (every one written by 0.230.0–0.233.1) leaves the slot open exactly as before. A journal that already closes that seq fails the resume and writes nothing.
  • terminalDownEvent, settledNodeEvidence and the release payload move to supervise/terminal-record.ts so scope and resume share one builder without a runtime import cycle. The refund at the reconcile is unchanged.
  • Every writer that mints a cursor seq in a resumed process — the recorded-result settlements and the resumed scope's cursor — starts past one reservedCursorFloor (closed seqs and open nodes' settledSeq). The independent check found the recorded-result branch seeding past closed records only; a test pins the collision and fails on the old seed.

Process

Two independent designs → judge (from source, the seq rule above) → implement → independent check (sound, one defect, fixed in the second commit with a mutation-verified test).

Evidence

  • Targeted suites on this head: 234/234 (retained-environment-release 14, recover-released-slot 13 incl. replay/tree equality against an uncrashed control on a fixed clock, second-resume idempotency, cancelled + overspend variants, refused release untouched, pre-fix record untouched, nested manager, observer projection).
  • typecheck, biome, build, api-surface, docs:api (0 warnings), docs:freshness, check:version-bump (1 export change → 0.234.0, release:prepare applied), check:publish-workflow all exit 0. merge-tree clean against main.
  • A scan of all 385 retained Lab journals on the operator host found 0 crash-window nodes, so this is a correctness fix with no backlog; the not-healable pre-fix shape is stated in the CHANGELOG.
  • First release cut under ci(publish): one release at a time, and refuse a tag main has moved past #1263's publish lock.

🤖 Generated with Claude Code

drewstone and others added 2 commits September 16, 2026 04:43
…recording

A release sweep appends a retained child's environment-teardown receipt
and then its terminal record. A process that died between the two left
an open cursor slot beside a destroyed environment; the next resume
treated the node as interrupted, tried to recover an executor whose
environment was gone or left it open forever, and every reader charged
the ceiling the pool had already refunded.

The reconciled record now carries the settlement the driver received
(reason, infra, trace, outRef, providerModel, harnessTranscript), the
withheld overspend, the cancellation source, and settledSeq, the cursor
seq next() stamped on the delivery, through the same field spread the
terminal record uses. On the root resume, healReleasedSlots walks the
journal forest and, for a spawned node with no terminal record whose
receipts after that record all read destroyed: true and name the
environment its last admission named, writes the released record with
the sweep's own builder at settledSeq and the reconciled at, then emits
the sweep's agent.child event. The node is never interrupted and no
recovery is attempted. A destroyed: false receipt, an empty receipt set,
a receipt for an unadmitted environment, or a reconciled record without
settledSeq leaves the slot open as before. The resumed cursor starts
past every open node's settledSeq; a journal that already closes that
seq fails the resume and writes nothing.

terminalDownEvent, settledNodeEvidence and the release payload move to
supervise/terminal-record.ts so scope.ts and recover-executors.ts share
one builder without a runtime import cycle. The refund at the reconcile
is unchanged. Version 0.234.0: SpawnEvent's shape moved.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit c5b7b69c7f0dacbb805ce767e2a05598d117527e)
…q past the reserved floor

Independent check of the heal found the recorded-result branch of
prepareInterruptedExecutors seeding its seq one past the last CLOSED record
only, while the resumed scope's cursor already folded open nodes' reserved
settledSeq. A recorded result could therefore be journaled on the seq an
open retained node had reserved, exactly the collision the heal refuses.
One reservedCursorFloor(events) now seeds both; a test pins the shape
(open node at settledSeq 1, recorded result settles at 2) and fails on the
closed-only floor.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 00de2dd31a811330b2c9e700f2562b51ec81d8c4)
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@drewstone
drewstone merged commit 9bc5539 into main Sep 16, 2026
4 checks passed
@drewstone

Copy link
Copy Markdown
Contributor Author

Cutting v0.234.0 at the merge commit now (tip of main, no Publish run in flight, latest tag v0.233.1). First release under #1263's lock.

@drewstone

Copy link
Copy Markdown
Contributor Author

Published: @tangle-network/agent-runtime@0.234.0 is on the registry (npm accepted 11:57:20Z, visible 12:08Z; provenance logged). Served tarball sha1 6d8eaf76a27dd20cf102db86bbd613b94b7a75ad matches the publish log and carries healReleasedSlots, reservedCursorFloor, settledSeq, terminalDownEvent. Lab bump to 0.234.0 is in a fresh worktree with the suite running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant