Skip to content

fix(supervise): fail closed when a provider refuses an environment create - #1273

Merged
drewstone merged 4 commits into
mainfrom
fix/terminal-create-refusal
Sep 16, 2026
Merged

drewstone merged 4 commits into
mainfrom
fix/terminal-create-refusal

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Closes #1272.

Problem

A provider SDK throws its own error classes. So an environment create that the platform refused reaches classifyDriverFailure as neither a BackendTransportError nor an AgentEvalError, falls past every branch, and takes the foreign-accident default: transient, retry. The refusal is deterministic, so every retry reproduces it until the attempt ceiling or the deadline.

Measured 2026-09-16 on discovery-lab, runtime 0.236.0, provider tangle-sandbox. Twelve roots pressed ~20 s apart; one got a Sandbox and eleven produced exactly this and nothing more:

{"kind":"execution-admitted","admission":{"phase":"intent","provider":"tangle-sandbox", …}}
{"kind":"metered","spend":{"tokens":{"input":0,"output":0},"ms":0,"tokensKnown":false}}  ×4

root-stream.jsonl stayed 0 bytes. The coordination servers bound and answered 401 unauthenticated, so the runs looked healthy from every local angle. They sat for 25 minutes until an operator cancelled them. The platform's side of the same window: 200 rejections across about a dozen sandbox ids at 14-22 attempts each, each carrying HTTP 400 {"error":"Platform key delegation failed …","code":"CONFIG_ERROR"}. The underlying cause was budget reservation — a budgeted platform key funds one live sandbox, because the first box's router child reserves the whole remaining budget.

The status was on the error the whole time. Nothing read it.

Change

classifyDriverFailure reads a plain HTTP status off any thrown Error after the existing branches, and applies the split the transport branch already promises consumers: 408, 429 and 5xx are the upstream having a bad moment; any other 4xx is a request that will fail identically forever.

Only a thrown Error is read, and only a plain integer in 400-599 decides anything. That bound is not cosmetic: status is an ordinary field name on settlements, run states and provider-model records. The first version of this patch tested typeof error === 'object' and broke bridge-executor.test.ts — a bridge child that aborts mid-turn rejects with such a value, and classifying it as an HTTP refusal skipped the retry that journals its paid usage, turning {input: 17, output: 3} into {input: 0, output: 0}. That case is now a regression test.

Checked

  • vitest run src/runtime/supervise tests/kernel — 1640 passed, 1 skipped, 0 failed.
  • vitest run src/runtime/supervise/driver-retry.test.ts — 58 passed, including the new cases: a provider SDK error at 400/403/409 is terminal, at 408/429/502 transient, and a non-Error value or a non-HTTP status keeps the historical default.
  • tsc --noEmit clean; biome check clean on both changed files.

What this does not fix

The refusal reaching the journal as a cause. A terminal create failure still settles driver-failed; carrying it through as retainedPendingCause: 'request-rejected' (the vocabulary from #1204/#1247) is the follow-up, and #1272 stays open for it. This PR stops the silence and the 25-minute wait.

Separately, hiding the real reason behind CONFIG_ERROR is the sandbox platform's own bug and its team owns it.

…eate

A provider SDK throws its own error classes, so a `create` the platform refused
reached classifyDriverFailure as neither a BackendTransportError nor an
AgentEvalError and took the foreign-accident default: retry. The status was on
the error the whole time.

classifyDriverFailure now reads a plain HTTP status off any thrown Error and
applies the split the transport branch already promises: 408, 429 and 5xx are
the upstream having a bad moment, any other 4xx will fail identically forever.

Measured 2026-09-16 on discovery-lab: a Tangle Sandbox create refused HTTP 400
CONFIG_ERROR, for a key whose budget was fully reserved by an existing box,
retried 14-22 times per node. Eleven of twelve roots showed a durable admission
intent and no other event for 25 minutes, with healthy coordination servers and
nothing for an operator to read.

Only a thrown Error is read. `status` is an ordinary field name on settlements
and run-state records, and reading one as an HTTP refusal stopped the retry that
journals a cancelled bridge child's paid usage.

Closes #1272.
The fixtures pin runtimeVersion against package.json, so a version bump without
`pnpm run generate:testing-fixture` fails tests/testing-fixture.test.ts.
…ssage quotes

error-message.ts already read an HTTP status off a thrown error, safely, to
build the persisted failure text; the retry verdict had grown a second reader
beside it. Export the original and delete the duplicate, so a failure is
classified from exactly the status an operator sees quoted back to them, and a
status behind a throwing getter cannot take the classifier down.
@drewstone

Copy link
Copy Markdown
Contributor Author

Two follow-ups pushed since the first review pass:

  • chore(testing): the proposal fixtures pin runtimeVersion against package.json, so the 0.237.1 bump needed pnpm run generate:testing-fixture. That was the CI failure.
  • refactor(supervise): error-message.ts already had a safe HTTP-status reader, used to build the persisted failure text, and this PR had grown a second one beside it. The original is now exported and the duplicate deleted, so a failure is classified from exactly the status an operator sees quoted back to them — and a status behind a throwing getter can no longer take the classifier down, which the duplicate did not guard. Covered by a new hostile-getter case.

vitest run src/runtime/supervise: 244 passed. tsc --noEmit and biome check clean.

Three checked-in artifacts embed the runtime version, and a hand bump reaches
only the first: the testing fixtures, docs/api/primitive-catalog.md, and
docs/canonical-api.md. scripts/prepare-release.mjs exists to move all of them
together, and this commit replaces the hand edits with its output.
@drewstone
drewstone merged commit a049ca3 into main Sep 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tangle-sandbox provider admits indefinitely when a Sandbox create is refused with a typed 4xx

1 participant