Skip to content

fix(supervise): settle a released retained child with a terminal record and report fleet yield - #1259

Merged
drewstone merged 5 commits into
mainfrom
fix/reconciled-terminal
Sep 16, 2026
Merged

drewstone merged 5 commits into
mainfrom
fix/reconciled-terminal

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Closes the runtime half of #1247. Stacked on #1257 (base is fix/fleet-observability); retarget to main once #1257 merges. Lab half: tangle-network/discovery-lab#778.

The hole

A retained child (execution admitted durably, result read lost) settled down with its cursor slot open and its reservation reconciled at the streamed floor, so a later process could resume and reconcile the paid execution. On the pursuit path no later process ever comes: supervisePursuit refuses re-entry once result.json exists and forces retainedAtSettlement: 'release', so every such child is released at root settlement and none recovered. Measured 2026-09-15: on capability-per-parameter-cpp-glm-20260915c, 0 of 35 reconciled children ever settled; across 385 retained runs, 185 of 223 lost sandbox children stopped at reconciled. Every journal reader read them as never-settled (poisoning tokensKnown/usdKnown for the whole run), and Lab told them apart from ordinary downs by reason text.

The fix

  • The release sweep closes each released node's slot. After the environment-teardown receipt and the executor's confirmed teardown, it writes the settlement the driver received, under the seq the driver saw, marked retainedExecution: 'released', carrying harnessTranscript, the reconciled floor (never the reservation) and the overspend the pool committed. A cancelled retained child keeps kind cancelled and its source. A refused release (destroyed: false, or confirmTeardown throwing) writes nothing — unknown stays unknown. One builder serves both writers. The journal tree is read once per sweep.
  • retainedExecution: 'pending' | 'released' on Settled's down arm, the settled/cancelled records, NodeSnapshot, the agent.child payload (a second event per released node with releasedAt) and PursuitNodeProjection — so a reader splits the population on a recorded fact, not on the RetainedExecutionPendingError wording.
  • SupervisedResult.fleetYield on every arm: spawned / done / down / cancelled / neverSettled / releasedUnrecovered, counted by node id across the journal forest, with spawned === done + down + cancelled + neverSettled and released ≤ down + cancelled asserted. wrapJournalForBreaker skips the release-written record, so downCount is unchanged and an armed breaker cannot reclassify a settled run after it settled. The waterfall keeps a released child's bar at its settlement instant.
  • The pool half (fix(budget): a retained-execution failure commits its full reserved ceiling as spend, with no refund #1190) is unchanged. Version stays 0.232.0 (check-version-bump: 25 export changes paid by 0.231.1 → 0.232.0).

Why the release sweep and not finalizeSettlement

A process crash never reaches the sweep, so the journal keeps today's state (open slot + reconciled floor) and a resume: true process still recovers the child through prepareInterruptedExecutors. A terminal record written at settle time would make the node non-interrupted on resume while its environment is alive — an unreceipted leak. The record lands beside the receipt that proves the environment is gone.

Process

Plan judged from two independent designs, three adversarial verifiers (replay, accounting, anchors), implemented by one agent from the corrected plan, then independently checked (sound-with-corrections, four low findings, all addressed in the second commit). Known unhealed window stated in the CHANGELOG: a crash between the destroyed: true receipt and the released record leaves an open slot beside a destroyed box.

Evidence

  • Targeted suites: retained-environment-release 9/9 (5 extended + 4 new: breaker not tripped by a released record; overspend carried; refused release writes nothing; aborted run under explicit 'release' closes the slot as cancelled/source: 'signal' and a later resume recovers nothing), supervise, supervise-global-concurrency, supervise-retained-recovery, overspend-settlement, supervise-pursuit-record, observer-projection, waterfall, testing-fixture, delegate fixtures — all green on this head; 22-file targeted run 343/343 before the last commit.
  • typecheck, biome, build, api-surface, docs:api (0 warnings), docs:freshness, check:version-bump, check:testing-fixture all exit 0. merge-tree clean against both origin/main and fix/fleet-observability.
  • Full suite: CI on this PR.

🤖 Generated with Claude Code

drewstone and others added 3 commits September 15, 2026 22:26
…rd and report fleet yield

A retained child (execution admitted durably, result read lost) settled
down with its cursor slot open and its reservation reconciled at the
streamed floor, so a later process could resume and reconcile the paid
execution. On the pursuit path no later process ever comes: the settle
record refuses re-entry and forces retainedAtSettlement 'release', so
every such child is released at root settlement and none recovered.
Measured 2026-09-15: on capability-per-parameter-cpp-glm-20260915c,
0 of 35 reconciled children ever settled, and 185 of 223 lost sandbox
children across 385 runs stopped at 'reconciled'. Every journal reader
read them as never-settled and charged the ceiling the pool had
refunded, and Lab told them apart from ordinary downs by reason text.

The release sweep now closes each released node's slot: after the
environment-teardown receipt and the executor's confirmed teardown it
writes the settlement the driver received, under the seq the driver
saw, marked retainedExecution 'released', carrying harnessTranscript,
the reconciled floor (never the reservation) and the overspend the pool
committed. A cancelled retained child keeps kind 'cancelled' and its
source. A refused release writes nothing. One builder,
terminalDownEvent, serves both writers so no field lands on one path
only. The journal tree is read once per sweep.

retainedExecution 'pending' | 'released' is stated on Settled's down
arm, the settled/cancelled records, NodeSnapshot (live and
materialized), the agent.child payload (a second event per released
node with releasedAt) and PursuitNodeProjection. SupervisedResult gains
fleetYield on every arm: spawned, done, down, cancelled, neverSettled,
releasedUnrecovered, counted by node id across the journal forest with
spawned === done + down + cancelled + neverSettled asserted.
wrapJournalForBreaker skips released records, so downCount is unchanged
and an armed breaker cannot reclassify a settled run. The waterfall
keeps a released child's end at its settlement. The pool half (#1190)
is unchanged; tokensKnown/usdKnown stay false on such a run and the gap
changes kind from never-settled to unreported.

Closes the runtime half of #1247.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ction, and pin the waterfall guard

Independent check of f273010 (sound, four low findings): the released count is now
added only for a down or cancelled bucket and asserted against down + cancelled, so a
hand-built done record carrying the marker cannot count twice; FleetYield documents
that a nested manager whose own retained execution was released counts beside its
grandchildren; the waterfall guard that keeps a released child's bar at its
settlement instant has a test. Fixtures regenerated for 0.232.0 arrived with the
merge from fix/fleet-observability.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@drewstone

Copy link
Copy Markdown
Contributor Author

Retargeted to main after #1257 merged (merge-merged, so this diff is exactly the #1247 change). Head is level with main including #1258; typecheck, fixtures, freshness and the retained/supervise/pursuit-record suites green on the merged head. Review and merge are yours per the cut; nothing else pending from me on this one.

…BusyBox are a stated limit, not a fallback

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@drewstone

Copy link
Copy Markdown
Contributor Author

Coordination note: #1260 merge is now in progress; root owns publishing 0.232.0 immediately after the verified merge. Please preserve the #1259 source and do not tag or publish concurrently. After 0.232.0 is published, adoption of this retained-release settlement change should include a version-gate check against the registry. This behavior is important for later long-horizon evidence, but it is not a blocker for the r2 provider-create repair.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant