Skip to content

fix(acp): stop retrying a sandbox-init failure and name the failing layer - #12144

Merged
bolichen97 merged 1 commit into
mainfrom
fix/acp-sandbox-init-nonretryable
Sep 20, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/acp-sandbox-init-nonretryable

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

A broken OS sandbox now fails on the first try and tells you which layer to turn off, instead of crash-looping behind a generic "process exited".

What changed

Switch / condition Before After
Sandbox-init signature on the dead child's stderr unread classified, both ACP transports
Spawn attempts for that refusal (AcpClient.ensure_ready) 2 1
transient verdict on the error unclassified (None) → wording fallback retried it fixed False → every retry ladder stops
Error type AcpProcessDied / bare AcpError AcpSandboxInitFailed, carrying layer + remediation
Layer named in the error — read off the argv Kiro Crew's own wrap returned
Disable switch in the error — Crew's layer only, and only on a verdict from a trusted launcher run
Latch lifetime — one child's startup: armed until the first session handle, spent there
Shared-runtime startup sites reading it — all 3, through one translation
Automatic fallback to an unconfined spawn — still none, deliberately

Detection and the classified type live in the ACP layer (src/kiro_crew/acp/client.py, src/kiro_crew/acp/runtime.py); the shared-runtime startup paths that translate a failed spawn are in src/kiro_crew/providers/acp.py and read one shared verdict; and the facts a message needs — which layer wrapped a spawn, whether a trusted run agrees, and what each layer's switch is — are answered by src/kiro_crew/sandbox.py, which already declares both switches and the corroboration. The per-turn session-provider path is deliberately untouched: the latch is spent by the time it could read one.

Signatures matched: sandbox initialization failed, sandbox_apply, sandbox-exec:, and Crew's Linux launcher refusing after the spawn — the last classified by sandbox.launcher_refusal's own no_backend verdict rather than by its prefixes, because that function deliberately answers "not a sandbox failure" for launcher lines that are retryable (a failed readiness handshake) or that mean the sandbox worked (the hardlinked-credential guard).

Why

A host whose sandbox cannot be built refuses the same way every time. The reconnect budget exists for transport faults, so spending it here buys nothing and hides the cause: the operator saw the agent process exit while the reason sat one line earlier on the child's stderr, and every chat entry crash-looped. meshclaw diagnostics reported healthy.

The repo already had the right shape for this. AcpAuthRequired and AcpToolGateUnroutable are non-retryable because respawning hits the same wall; this is the third member of that family. Reported in Taskei Mesh-2998.

Why the layer is not derived from the signature

Identical stderr, opposite remedies. A harness's internal sandbox nested inside Crew's wrap fails with the harness's wording while the layer the operator must change is Crew's — which is how the field reports were actually resolved. So the layer comes from sandbox.wrapped_by_crew_sandbox(), reading the argv the wrap returned: the wrap's own record of the branch it took. A re-derivation from mode + platform + settings cannot match it, because the delegated branch still falls back to Crew's seatbelt for a masked spawn and the audit-or-deny step can refuse a delegation after it was chosen.

Each layer's remedy is read off the module that declares its switch, never spelled at the raise site, so a rename cannot leave a stale path in an operator-facing message.

Why the detector needs a sandbox token

The reported burst is three lines, and only the first is sandbox-specific; the other two are Failed to spawn child process and a bare Operation not permitted. Those are ordinary output for a missing binary, a bad interpreter or a denied credential file, and matching either alone would latch a permanent verdict onto a class of deaths a respawn legitimately fixes. Every alternative is anchored on a sandbox token, which costs no coverage because whichever layer refused prints one.

Why the disable switch needs a trusted run

The signature arrives on the dead child's stderr, and that child is the unverified binary the sandbox exists to contain. Answering it with kirocrew config set agent.sandbox off would let a planted binary print one line and have Kiro Crew instruct the operator to remove the isolation it is running under. sandbox.launcher_refusal states this hazard for its own callers and prescribes the answer, so that is the answer used: corroborate_launcher_refusal re-runs the real launcher around a trusted no-op under the refused spawn's own mode and masks, and only its stderr — text no child wrote — unlocks the switch, and only on its no_backend verdict: the same function reports transient for a failed readiness handshake that self-heals on the next spawn. Off the event loop, as the first-run gate runs it.

Uncorroborated, the message names the layer (that comes from our own argv, not from the child), says plainly that Kiro Crew has not confirmed the failure on this host, and stops — naming no switch and no check, on either layer. The harness branch never names its key at all, corroborated or not, and that is the sharper case for the rule rather than the looser one. Two reasons stack: the layer being blamed is the agent's own sandbox, so naming its key would let a planted binary talk the operator into removing the only isolation that child had — and corroboration re-runs Crew's launcher, so its verdict is evidence about Crew's layer and says nothing about the harness's. Letting it unlock the other layer's switch would be a cross-layer inference whose one confirmed outcome is that isolation is gone. And no check is named because corroboration covers Crew's Linux launcher only; on macOS the probe validates an (allow default) profile while the real wrap applies the strict one, so sending the operator there would false-green exactly the failure they are looking at. That gap closes with the real-wrap self-test, tracked separately.

One consequence worth stating, because it is easy to get wrong: the stderr an operator reads and the stderr a classifier matches are different shapes. The shared runtime's retained summary folds the tail onto one line behind a returncode prefix, and the launcher's refusal is recognised per line — so the two travel separately. Folding them together would leave corroboration silently never running on the very path the Linux launcher case arrives by.

Why the drain is settled twice, in two different places

The latch is written by the _drain_stderr task, while the death that brings a caller to the classifier is discovered on the stdout side — _mark_dead fails the pending initialize synchronously. Both are runnable at once, which is the ordinary shape of a real refusal: the child writes its signature and closes stdout together. So the read settles the drain first, bounded and swallowing, the same pattern AcpClient already applies at its own EOF.

That is necessary but not sufficient on the shared runtime, because its failed-handshake cleanup tears the runtime down and the teardown cancels the drain. A line still in the pipe then is a line nobody ever reads, and the caller's settle finds the task already done and learns nothing. So the cleanup settles before it tears down — the last moment the child's own account of why it could not start is still reachable. Without both, a genuine refusal reports as a generic repeatable crash on whichever fraction of runs loses the schedule.

Why the latch is a startup verdict, and where startup ends

The auth latch beside it is life-long, correctly: a rejected credential does not un-reject itself. This one is not that. The harness's own sandbox can refuse when it spawns a tool subprocess long after the agent started fine, and a life-long latch would let that make the next unrelated death permanently "your sandbox is broken".

Startup ends at the first session handle, not at the initialize handshake — the handshake is the middle of startup. session/new starts the child's own MCP servers, and a sandboxed launcher among them can refuse there; a window closing at the handshake would leave every such refusal unclassified on exactly the translation site (create_session) that exists to catch it. So the latch arms until the first handle exists and is cleared there. Clearing rather than merely un-arming is what makes every consumer correct by construction: the three startup translations read it while startup is unfinished, and any path reached after it can only ever see False — which is why the per-turn translation carries no sandbox branch at all.

Why no auto-downgrade

Falling back to an unconfined spawn would turn a broken host into a silently unsandboxed agent. The fix is classification, fail-fast, and an actionable remedy — stated with what turning the layer off costs.

Relation to #7396

Complementary, no overlap. #7396 makes the cron retry decision read a type-carried verdict (AcpProcessDied.resubmit_safe) instead of message wording; this adds a new type that any type-based retry treats as terminal, through the transient=False channel that already existed on AcpError. Neither touches the other's field or branch — AcpProcessDied is untouched here, and AcpSandboxInitFailed is a sibling of it, not a subclass — so the two merge in either order. On #7396's own terms this is a death nobody should resubmit at all, which its unclassified-falls-back-to-wording default already routes correctly.

Pattern harvest

Rule candidate: none — this is a missing classification, not a mechanical slip a matcher could have caught.

The generalizable lesson is about the shape of a deterministic failure reaching a retry ladder, and the repo already encodes it: AcpAuthRequired and AcpToolGateUnroutable exist because respawning re-reads the same answer. Each new deterministic refusal has to be named individually, so there is nothing to grep for.

Two patterns are worth naming for reviewers rather than for a linter. A MagicMock stands in for a collaborator that grew a predicate and answers truthy to whatever nobody stubbed — four test files here needed the new predicate answered explicitly, and a missed one would have silently asserted the new branch instead of the one it is about. And a classifier over untrusted subprocess output is not a verdict: this module already separates the two (launcher_refusal vs corroborate_launcher_refusal), and any new consumer of child stderr has to pick the right side of that line.

Tested

55 tests in test/test_acp_sandbox_init_classification.py: every signature classified and every control tail left retryable (bare errno, auth output, healthy startup, and each launcher line that carries a blocking prefix without being a sandbox failure); exactly one spawn attempt per refused-sandbox case, and a control death still getting its one retry; the switch withheld on an uncorroborated refusal and on a trusted run that clears the host, emitted on one that does not, and no trusted subprocess spent on a signature it could not speak to; the latch surviving a trimmed ring buffer, firing on a restricted session that retains no stderr, refusing a mid-life line, still catching one printed after the handshake, and spent at the first session handle so a live session's death is never a sandbox verdict; both layer attributions through the cold start; the losing schedule of the drain race reproduced deterministically and closed, a wedged drain still unable to hold the failure path open, the cleanup ordering that keeps a cancelled drain from discarding the evidence, and the display-vs-classify shapes staying separate so a folded summary cannot hide the launcher's line; and the shared retry predicate refusing the error on behalf of every consumer.

Twenty-two mutations verified caught, one per decision: dropping the sandbox anchor, disabling the fail-fast, always attributing to Crew's layer, leaving transient unclassified, matching launcher prefixes raw, latching for the runtime's whole life, leaving one startup site unconverted, handing out the switch without corroboration, accepting a transient trusted verdict as corroboration, ignoring the corroboration verdict entirely, never clearing the latch at the handshake, dropping the drain settle, making the settle not wait, making it wait without a bound, settling the drain after the teardown that cancels it instead of before, classifying the display summary instead of the lines, dropping the lines at the shared translation, naming the harness switch on any refusal, letting a Crew-launcher verdict unlock the harness switch, promising an unconfined spawn a governance floor can refuse, closing the startup window at the handshake, and never spending it at all.

isort, black, flake8, mypy clean. Neighbouring suites green: 1300 across the ACP client, runtime, provider, session-provider and stderr-latch files, plus the brand, comment-history, harness-parity, agent-sdk-boundary, black-baseline and docs-lint gates. Spec rationale documented under the OS-Level Sandbox section of docs/system-specs/modules/security.md.

no linked issue: reported through Taskei, which has no GitHub issue in this repository.

Doctor real-wrap parity — making diagnostics exercise the strict generated profile rather than the allow-default probe — is deliberately out of scope and tracked separately.

@iamwhatever
iamwhatever requested review from a team and pepmach September 20, 2026 01:37
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 30fce44c256144da79a1dc7ff352075cc7affdd2 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/acp/runtime.py:6605 -- “PRE-INITIALIZE” contradicts the _first_session_ready guard and post-session/new clear -> Fix: describe the window consistently as ending when the first session handle is ready, including the security spec.
[GPT-REVIEWED] 30fce44

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 30fce44c256144da79a1dc7ff352075cc7affdd2: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 30fce44c256144da79a1dc7ff352075cc7affdd2 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the classified error and corroborated switch on a real refusing Linux host — all 55 tests drive synthetic stderr, none a live launcher refusal.

[DESIGN-REVIEWED] 30fce44

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 30fce44c256144da79a1dc7ff352075cc7affdd2 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify one drift: the new spec hunk says the latch "is spent when initialize completes", but the code and its own test spend it at the first session handle.

What this change ships

Inventory (10 items) — 10 justified

Intent: stop crash-looping a deterministic OS-sandbox refusal and tell the operator which layer refused — a FIX (Taskei Mesh-2998; test_a_refused_sandbox_burns_exactly_one_spawn fails on base, which makes 2 attempts).

  1. A sandbox-refused spawn fails on the first attempt instead of feeding every retry ladder — justified
  2. The failure names the isolation layer, read from the argv Crew's own wrap returned — justified
  3. agent.sandbox off appears only after a trusted launcher run confirms the host — justified
  4. On macOS the message says "not confirmed" and names no switch — justified
  5. Restricted-memory AcpClient sessions keep today's retry — justified
  6. All 3 shared-runtime startup sites classify the refusal (counted: 3 saw_not_logged_in sites, 3 translations) — justified
  7. Refusals after the first session handle never latch; mid-life deaths stay generic — justified
  8. Failed-handshake cleanup reads stderr before teardown — justified
  9. security.md section, same commit — justified
  10. New runtime/sandbox accessors (settle_stderr, wrapped_by_crew_sandbox, sandbox_init_remediation, layer constants) — justified

Reuse verified: launcher_refusal / corroborate_launcher_refusal pre-exist in sandbox.py:9491,9572 and their docstrings prescribe exactly this corroboration; the non-retryable family (AcpAuthRequired, AcpToolGateUnroutable) pre-exists at client.py:2674,2740. No pins deleted.

[FIRST-PRINCIPLES-REVIEWED] 30fce44

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 30fce44c256144da79a1dc7ff352075cc7affdd2 — this comment is updated in place on each push.

Review details

Both candidates fail the falsification bar.

Candidate 1 (client.py:270 corroboration gate): The claimed defect is that launcher_refusal(to_classify) returns None on a tail where a hardlink BLOCKED line precedes a genuine unshare refusal (since launcher_refusal returns on its first prefix-matching line at sandbox.py:9547-9548), so corroboration is skipped. Verified the mechanism is real, but (a) the input does not occur in practice — a single Linux launcher run that blocks on a hardlinked credential exits at that check and never reaches the unshare step, so one child cannot emit both lines; and (c) the outcome is the safe direction (switch withheld, error still raised non-retryable), i.e. a weaker diagnostic, not a security hole, crash, or data loss. The candidate itself scores this low. Drop.

Candidate 2 (sandbox.py:838 governance wording): This is a "add a test that drives the governance path" request — a missing-tests finding, a category this pipeline rejects deterministically. The underlying claim ("governance can clamp off back up") the candidate concedes is true, and the cited rule is blocking: false. Drop.

No groundable new defect surfaced while falsifying.

No findings.

[OPUS-REVIEWED] 30fce44

Verdict parsed from the review's SHA-scoped output markers for commit 30fce44c256144da79a1dc7ff352075cc7affdd2.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 30fce44c256144da79a1dc7ff352075cc7affdd2: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the fix/acp-sandbox-init-nonretryable branch from a510ac3 to 5ddad3c Compare September 20, 2026 02:29
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@iamwhatever
iamwhatever force-pushed the fix/acp-sandbox-init-nonretryable branch from 5ddad3c to 3e5e516 Compare September 20, 2026 03:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@iamwhatever
iamwhatever force-pushed the fix/acp-sandbox-init-nonretryable branch from 3e5e516 to d422214 Compare September 20, 2026 04:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 20, 2026
@iamwhatever
iamwhatever force-pushed the fix/acp-sandbox-init-nonretryable branch from d422214 to 09888e3 Compare September 20, 2026 04:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@iamwhatever
iamwhatever force-pushed the fix/acp-sandbox-init-nonretryable branch from b8d1a60 to 0543990 Compare September 20, 2026 08:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@iamwhatever
iamwhatever force-pushed the fix/acp-sandbox-init-nonretryable branch from 0543990 to 431c3cf Compare September 20, 2026 09:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@iamwhatever
iamwhatever force-pushed the fix/acp-sandbox-init-nonretryable branch from 431c3cf to 8b928e4 Compare September 20, 2026 10:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@iamwhatever
iamwhatever force-pushed the fix/acp-sandbox-init-nonretryable branch from 8b928e4 to e61267a Compare September 20, 2026 11:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
…ayer

A host whose OS sandbox cannot be built refuses identically on every attempt,
but the ACP spawn/init path treated it as a transport fault: the reconnect
ladder respawned, the same refusal came back, and the operator saw a generic
process-exit card while the cause sat on the child's stderr.

Both ACP transports now detect the sandbox-init signature in the spawn/init
window and raise AcpSandboxInitFailed -- non-retryable for the same reason
AcpAuthRequired and AcpToolGateUnroutable are, with transient fixed False so
every retry ladder stops on the first one. The error carries the layer that
refused and that layer's own switch, read off the argv the wrap returned
rather than guessed from the signature: a harness sandbox nested inside Crew's
wrap fails with the harness's wording while the layer to change is Crew's.

No automatic downgrade to an unconfined spawn: the fix is classification plus
fail-fast plus an actionable remedy.
@iamwhatever
iamwhatever force-pushed the fix/acp-sandbox-init-nonretryable branch from e61267a to 30fce44 Compare September 20, 2026 11:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • accepted-and-deferred — span=081ce7af27e6 (src/kiro_crew/acp/runtime.py, stale "PRE-INITIALIZE" wording)

Correct, and it is my own stale wording. The guard reads _first_session_ready
and the clear sits after session/new, so the window ends at the first session
handle — the heading in _drain_stderr and one sentence in
docs/system-specs/modules/security.md still describe the narrower
pre-initialize window the earlier revision had.

Deferred rather than pushed, deliberately: the change is two comment lines with
no behaviour attached, and every push restarts a 95-check matrix that has cost
this PR four unrelated red rounds (a uv manifest-fetch timeout, an xdist worker
crash, a load-sensitive memory stress timeout, and a Windows regression from
#12188 that had to land its own fix on main first). Spending that to correct two
sentences is the wrong trade while the branch is green.

Nothing about the behaviour is ambiguous in the meantime: the docstring on
_first_session_ready states the window and why it is separate from
_initialized, the paragraph directly under the stale heading gives the
session/new reason in full, and
test_the_first_session_clears_the_latch_in_the_source fails if the clear ever
migrates back to the handshake.

self-added: yes
mechanism: the wording was accurate when written and was not re-read after the
window widened in the following round; the test above pins the behaviour the
comment describes, so the code cannot drift with it.

@bolichen97
bolichen97 enabled auto-merge (squash) September 20, 2026 17:07

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against the pr-triage-pipeline gates plus a full-diff security audit of client.py, runtime.py, providers/acp.py and sandbox.py.

fix(acp), 8 files. Classifies a host-level sandbox-init refusal as non-retryable (AcpSandboxInitFailed, transient=False) so ensure_ready stops re-attempting, and names which layer refused (kirocrew vs harness-internal) instead of reporting a generic spawn failure.

This PR hits the security-path denylist (src/kiro_crew/sandbox*) and answers three checklist questions YES (touches_sandbox_or_confinement, modifies_safety_override_or_gate, introduces_external_input_parsing), so it is a security-flagged review, not a routine fix.

[Security: the risk the gate exists to catch is not realized; the change is a net hardening. Enforcement is untouched, only reporting and guidance change. There is NO automatic downgrade to an unconfined spawn, and a sandbox that will not initialize still makes the agent fail fast rather than falling through unconfined. Retry removal is anchored on a sandbox-token signature, so a bare 'Operation not permitted' or 'Failed to spawn child process' does not match, and launcher refusals are classified only for the no_backend kind while readiness-handshake and hardlinked-credential refusals stay retryable. The latch is scoped to the pre-first-session startup window and cleared at _finish_create_session, so a mid-life tool-subprocess refusal cannot permanently mark a later unrelated death as sandbox-broken; a control death still gets its retry. The sandbox-disable guidance text is surfaced only on a corroborated verdict from corroborate_launcher_refusal, which re-runs the real launcher around a trusted no-op, so a planted binary printing a fake refusal cannot talk an operator out of isolation; an uncorroborated verdict names the layer and offers no switch, and the harness layer never gets one. Child stderr reaching the session card, security event log or cron last_error passes through redact_exfiltration_urls and redact_credentials, and the remediation text names only the layer plus at most a pre-existing config key, so no host paths or secrets leak.]

Spec ride-along: docs/system-specs/modules/security.md changed as a minority of the diff (1 file / 63 lines against 7 files / 1610 lines) and was not reviewed as a design decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants