Skip to content

test(codex): let codex's own bubblewrap preflight decide its inner-sandbox branch - #37677

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/codex-sandbox-probe
Nov 15, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/codex-sandbox-probe

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The two codex inner-sandbox contract tests stop failing on pull requests that run on the ubuntu-26.04 image of kirocrew-linux-4-core, while every runner still measures what codex's sandbox does.

test_real_codex_acp_git_status_runs_inside_its_own_sandbox and test_real_codex_acp_network_runs_only_when_crew_allows fail in Real Adapter Contract Tests on unrelated pull requests. Codex runs both steps fine (git status exits 0 with ## No commits yet on master; the network step exits 0 with NET-200), but the test expects codex's bwrap: refusal. The captured log shows why: kiro_crew.sandbox ... userns probe failed (permanent): unshare(CLONE_NEWNS) failed with errno 1 (EPERM). Seen in job 132878298073, job 132910126534 and job 132913204103, all on Image: ubuntu-26.04.

Why it matters

Every pull request routed to that runner shows a red Real Adapter Contract Tests check its diff cannot fix. The Linux flaky scan counts this one test blocking 4 open pull requests. Contributors chase it or learn to ignore the lane, which hides real adapter regressions.

Not a goal

What changed (motivation → approach → change)

The test asked the wrong program. It decided "can this host build codex's sandbox?" from Crew's own userns probe first, and ran codex's bubblewrap preflight only when Crew's probe said yes. AppArmor judges each binary by its own profile, so on ubuntu-26.04 Crew's Python probe is refused while codex's bubblewrap is not. The "no" short-circuited the check that would have said "yes".

Crew's probe codex's preflight before after
yes pass runs runs
no pass must fail runs
any loopback / uid-map refusal must fail must fail
any other error, or no bwrap Crew's probe Crew's probe

Now codex's own preflight, with the bubblewrap codex picks, decides whenever it gives a verdict; Crew's probe answers only when it cannot. The rule is written down where test authors and reviewers read it: testing-conventions.md (new section), the writing-tests skill (builtin_skills/kirocrew-dev/writing-tests/SKILL.md, Rule 3), an AUTOSDE.yaml tests-are-deterministic item, and the real-adapter-contract row of docs/ci/ci-and-reviews.md.

Backwards compatibility

Compatible: test-only. A host whose preflight passes, or meets one of the two refusals, picks the same branch as before unless Crew's probe said no; the other outcomes are unchanged.

Removes nothing: no test, assertion, step or check is removed.

Tests

Unmarked tests in test/test_codex_inner_sandbox_approval.py, with a fake runner in place of bubblewrap:

  • test_a_passing_preflight_keeps_the_sandboxed_branch_when_crews_probe_says_no: Crew's probe says no, the preflight passes; the preflight runs, a step that ran passes, a step that printed a refusal still fails.
  • test_a_preflight_host_refusal_takes_the_refusal_branch now runs with Crew's probe both yes and no.
  • test_no_bubblewrap_runs_no_preflight_and_crews_probe_answers: with no bubblewrap found, Crew's probe answers.

Forced condition: the new test is red on the parent's probe (assert [] == [['/opt/codex/bwrap', ...]], the preflight never ran) and green on this commit.

Proof: not provable -- test-only change

Manual verification

Read the four failing jobs' logs; the three on ubuntu-26.04 all carry the unshare(CLONE_NEWNS) ... EPERM warning beside codex exit 0. The live tests skip locally (no codex-acp). This pull request's own Real Adapter lane runs them for real.

Related Issues

Part of #37647

Sibling sweep

Swept: git grep -n "userns_available" -- test/
Sibling: test/real_adapters/harness_probe/drive.py -- not affected -- it gates Crew's own sandbox, the program it drives
Sibling: test/test_kiro_cli_probe_sandbox.py -- not affected -- it gates Crew's own launcher sandbox, the program it drives

Pattern harvest

Rule candidate: review-prompt
Pattern: a host probe that picks a live test's expected branch asks a different program or binary than the one the test drives

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision be725f891642990dba03e245a32aa7056b5a0459 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of be725f891642990dba03e245a32aa7056b5a0459 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the probe now asks the binary the test actually runs, and the CI logs back the change.

Notes

  • The dropped userns=False, bwrap=… → no-preflight assertion was a pin. The reviewer could not read its history in this shallow checkout. The new test and the job logs argue it was an ordering artifact, not a decision.

[DESIGN-REVIEWED] be725f8

Description read

[DESCRIPTION-READ] 2f451127676f9ff097b2f2902b40c87c11ea2bc19e799737f7fbcaaf60a77ef6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of be725f891642990dba03e245a32aa7056b5a0459 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The test fix is aimed at the cause: the wrong program was asked. Two of the three new copies of the general rule, the shipped skill and the AUTOSDE item, add surface beyond the Goal.

Notes

  • Item 4 — oversized: one measured instance is restated as a standing rule in three places (testing-conventions, the shipped writing-tests SKILL.md and AUTOSDE item 13). The Goal is "the two codex inner-sandbox contract tests stop failing". Subtraction: delete the SKILL.md bullet and the AUTOSDE item 13, and keep the testing-conventions section as the single copy. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (6 items) — 4 justified

Intent: stop the codex inner-sandbox contract tests going red on ubuntu-26.04 runners while still measuring codex's sandbox. This is a FIX.

  1. Codex's own bubblewrap preflight now decides the expected branch, and a "no" from Crew's probe no longer skips it — justified
  2. Unit tests cover the case where Crew's probe says no and the preflight passes, plus both probe values on a refusal; the old pin was replaced, and the description's measured logs explain why — justified
  3. The real-adapter-contract row in ci-and-reviews now describes the new decision order — justified
  4. New general rule written into the writing-tests skill and AUTOSDE tests-are-deterministic — rides along, oversized
  5. New testing-conventions section with the MEASURED case — justified
  6. _unprivileged_userns docstring rewritten to describe the preflight-first order — rides along

[FIRST-PRINCIPLES-REVIEWED] be725f8

Description read

[DESCRIPTION-READ] 1576518f0fd8ea1445eafeac2cd716f3482ec05b99f58afcc10997fe337fc0f7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed be725f891642990dba03e245a32aa7056b5a0459 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] be725f8

Verdict parsed from the review's SHA-scoped output markers for commit be725f891642990dba03e245a32aa7056b5a0459.

False positive or not applicable? A repository writer can comment:
/ai-review override fable be725f891642990dba03e245a32aa7056b5a0459 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of be725f891642990dba03e245a32aa7056b5a0459 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] be725f8

False positive or not applicable? A repository writer can comment:
/ai-review override gpt be725f891642990dba03e245a32aa7056b5a0459 false-positive: <reason, 20 to 500 characters>

@bolichen97

Copy link
Copy Markdown
Contributor Author

Determinism proof for test/test_codex_inner_sandbox_approval.py (same tree as the PR head, run on a Linux DevSpace, Amazon Linux 2023, Python 3.12):

repeat -n0 (20 runs, all node ids):                     20/20   last: 12 passed, 8 skipped
repeat -n 4 --dist loadgroup --max-worker-restart=2
  with test_codex_session_mcp.py + test_sandbox_probe.py: 10/10   last: 74 passed, 18 skipped
shuffled order, seed=9147:                              12 passed, 8 skipped
test/test_flake_pattern_ratchet.py:                     68 passed
scripts/local-gate.py (related tests, both surfaces):   all selected gates passed; vitest 534 files / 15756 tests passed

The 8 skips are the live real_adapter tests (no codex-acp on that host); the Real Adapter Contract Tests lane runs them.

Forced condition: the new test_a_passing_preflight_keeps_the_sandboxed_branch_when_crews_probe_says_no run against the parent's _host_builds_codex_sandbox fails (assert [] == [['/opt/codex/bwrap', ...]]: the preflight never ran when Crew's probe said no) and passes on this head.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked head be725f8. Codex's own bubblewrap preflight now decides which result the inner-sandbox tests require: a pass means the step must run, a known loopback or uid-map refusal means it must fail closed, and only other outcomes fall back to Crew's probe. The live assertions still require either a real run or the fail-closed refusal, so the contract tests still measure something. The new unit test fails against the old function and passes here. The test file passes locally, and the Real Adapter lane on this head ran both codex sandbox contracts and passed. Nothing blocks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants