Repository navigation
test(codex): let codex's own bubblewrap preflight decide its inner-sandbox branch - #37677
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the probe now asks the binary the test actually runs, and the CI logs back the change. Notes
[DESIGN-REVIEWED] be725f8 Description read[DESCRIPTION-READ] 2f451127676f9ff097b2f2902b40c87c11ea2bc19e799737f7fbcaaf60a77ef6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The test fix is aimed at the cause: the wrong program was asked. Two of the three new copies of the general rule, the shipped skill and the AUTOSDE item, add surface beyond the Goal. Notes
What this change shipsInventory (6 items) — 4 justifiedIntent: stop the codex inner-sandbox contract tests going red on
[FIRST-PRINCIPLES-REVIEWED] be725f8 Description read[DESCRIPTION-READ] 1576518f0fd8ea1445eafeac2cd716f3482ec05b99f58afcc10997fe337fc0f7 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Determinism proof for The 8 skips are the live Forced condition: the new |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
I checked head be725f8. Codex's own bubblewrap preflight now decides which result the inner-sandbox tests require: a pass means the step must run, a known loopback or uid-map refusal means it must fail closed, and only other outcomes fall back to Crew's probe. The live assertions still require either a real run or the fail-closed refusal, so the contract tests still measure something. The new unit test fails against the old function and passes here. The test file passes locally, and the Real Adapter lane on this head ran both codex sandbox contracts and passed. Nothing blocks.
Problem / Motivation
Goal: The two codex inner-sandbox contract tests stop failing on pull requests that run on the
ubuntu-26.04image ofkirocrew-linux-4-core, while every runner still measures what codex's sandbox does.test_real_codex_acp_git_status_runs_inside_its_own_sandboxandtest_real_codex_acp_network_runs_only_when_crew_allowsfail in Real Adapter Contract Tests on unrelated pull requests. Codex runs both steps fine (git statusexits 0 with## No commits yet on master; the network step exits 0 withNET-200), but the test expects codex'sbwrap:refusal. The captured log shows why:kiro_crew.sandbox ... userns probe failed (permanent): unshare(CLONE_NEWNS) failed with errno 1 (EPERM). Seen in job 132878298073, job 132910126534 and job 132913204103, all onImage: ubuntu-26.04.Why it matters
Every pull request routed to that runner shows a red Real Adapter Contract Tests check its diff cannot fix. The Linux flaky scan counts this one test blocking 4 open pull requests. Contributors chase it or learn to ignore the lane, which hides real adapter regressions.
Not a goal
What changed (motivation → approach → change)
The test asked the wrong program. It decided "can this host build codex's sandbox?" from Crew's own userns probe first, and ran codex's bubblewrap preflight only when Crew's probe said yes. AppArmor judges each binary by its own profile, so on
ubuntu-26.04Crew's Python probe is refused while codex's bubblewrap is not. The "no" short-circuited the check that would have said "yes".Now codex's own preflight, with the bubblewrap codex picks, decides whenever it gives a verdict; Crew's probe answers only when it cannot. The rule is written down where test authors and reviewers read it:
testing-conventions.md(new section), thewriting-testsskill (builtin_skills/kirocrew-dev/writing-tests/SKILL.md, Rule 3), anAUTOSDE.yamltests-are-deterministicitem, and thereal-adapter-contractrow ofdocs/ci/ci-and-reviews.md.Backwards compatibility
Compatible: test-only. A host whose preflight passes, or meets one of the two refusals, picks the same branch as before unless Crew's probe said no; the other outcomes are unchanged.
Removes nothing: no test, assertion, step or check is removed.
Tests
Unmarked tests in
test/test_codex_inner_sandbox_approval.py, with a fake runner in place of bubblewrap:test_a_passing_preflight_keeps_the_sandboxed_branch_when_crews_probe_says_no: Crew's probe says no, the preflight passes; the preflight runs, a step that ran passes, a step that printed a refusal still fails.test_a_preflight_host_refusal_takes_the_refusal_branchnow runs with Crew's probe both yes and no.test_no_bubblewrap_runs_no_preflight_and_crews_probe_answers: with no bubblewrap found, Crew's probe answers.Forced condition: the new test is red on the parent's probe (
assert [] == [['/opt/codex/bwrap', ...]], the preflight never ran) and green on this commit.Proof: not provable -- test-only change
Manual verification
Read the four failing jobs' logs; the three on
ubuntu-26.04all carry theunshare(CLONE_NEWNS) ... EPERMwarning beside codex exit 0. The live tests skip locally (no codex-acp). This pull request's own Real Adapter lane runs them for real.Related Issues
Part of #37647
Sibling sweep
Swept: git grep -n "userns_available" -- test/
Sibling: test/real_adapters/harness_probe/drive.py -- not affected -- it gates Crew's own sandbox, the program it drives
Sibling: test/test_kiro_cli_probe_sandbox.py -- not affected -- it gates Crew's own launcher sandbox, the program it drives
Pattern harvest
Rule candidate: review-prompt
Pattern: a host probe that picks a live test's expected branch asks a different program or binary than the one the test drives
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)