Skip to content

fix: check a review lane boundary before the model runs - #30440

Merged
bolichen97 merged 1 commit into
mainfrom
fix/13414-review-sandbox-check
Nov 5, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/13414-review-sandbox-check

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The fork review lane refuses to start its model when the runner's sandbox would let that model reach what other job processes hold, and runs exactly as before when it would not.

The fork review lane gives its model a shell on an untrusted fork diff. Before the model starts, the lane already proves two of the boundaries that shell must respect. A third one was assumed, never checked on the runner: if it failed there, nothing went red.

Why it matters

A prompt-injected reviewer on a fork PR could copy something another job process holds into a public review. The repository wants that case to stop the lane, not to pass silently.

Not a goal

  • No new restriction on what the review passes can read. Their configuration is unchanged.
  • Not the same-repository review lane, which reviews only branches pushed by people who already have write access.
  • No change to which runners the lane uses or how it is triggered.

What changed (motivation → approach → change)

Symptom The pre-model self-check could pass on a runner where the model's shell can reach other job processes.
Root cause The self-check tested two of the shell's boundaries and assumed the third.
Approach Check the property itself on every run, on the runner that runs the review. Adding a rule whose enforcement on that runner is unproven was the alternative, and the issue rules it out.
Change The existing .github self-check step gains one probe. It starts a marked sibling process and waits, outside the sandbox, until its marker is readable. It fails the job closed when the sandbox can read that marker, when the marker never shows outside the sandbox, or when the sandbox cannot read the checkout it must review (either would make the refusal meaningless).

The CI review lanes page in docs describes the probe. The test file covers all three outcomes.

Backwards compatibility

Compatible: on a runner where the boundary holds, every probe passes and the lane runs as before. The lane stops only on a runner where the model could reach what it must not.

Removes nothing: no input, key, record or capability is removed or tightened. The only new outcome is a refused run on a runner whose sandbox does not hold.

Tests

The new tests run the real step script against stand-in sandboxes. A sandbox that can reach other processes must fail the step, and one that cannot must pass it. A sibling whose marker never appears must fail the step too, so a sandboxed read can never pass just because it came too early. Both failing cases fail with the fix reverted.

Proof: PROVEN test/test_ai_review_workflows_credential_isolation.py::TestReviewSandboxFailsClosed::test_it_fails_closed_when_another_process_environment_is_readable[False], test/test_ai_review_workflows_credential_isolation.py::TestReviewSandboxFailsClosed::test_it_fails_closed_when_the_marked_sibling_never_shows_its_marker (head 8557115916c4)

Determinism: 15/15 runs of the three probe cases at -n 4; the related review-workflow test files, 2395 passed at -n 4.

Manual verification

The step's own script was run on a Linux host that can create user namespaces, against the pinned review CLI:

Sandbox Result
The pinned review CLI's sandbox, in both of its Linux modes every probe passes, exit 0
No sandbox at all (stand-in) fails closed on the new probe, exit 1

The hosted-runner check is the step itself: every fork review run measures the boundary on its own runner before the model starts.

Related Issues

Closes #13414

Sibling sweep

Siblings: none -- the only other lane that hands a model a shell reviews same-repository branches only, so no untrusted author can steer it.

Pattern harvest

Not generalizable: one review lane's pre-model check missed one boundary; the other lanes give their model no shell.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 5, 2026 13:17
@bolichen97
bolichen97 requested a review from buluoray November 5, 2026 13:17
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 356728cf6a4581982f98d106fbefd04dc4c0b20e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 356728cf6a4581982f98d106fbefd04dc4c0b20e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: one fail-closed probe in an existing step, aimed at a boundary that was assumed but never checked (#13414), with no new surface.

What this change ships

Inventory (4 items) — 4 justified

Intent: stop the fork review lane before its model starts on a runner where the model's sandboxed shell could read another job process's environment. This is a FIX.

  1. The fork GPT review job now fails before the model runs if the sandbox can read a marked sibling process's /proc/<pid>/environ — justified
  2. The job fails closed when the sibling's marker never becomes readable outside the sandbox (the positive control) — justified
  3. The job fails closed when the sandbox cannot read the checkout (README.md), so a sandbox that refuses everything cannot pass the probe — justified
  4. The CI review lanes page and the tests now cover the third probe — justified

[FIRST-PRINCIPLES-REVIEWED] 356728c

Description read

[DESCRIPTION-READ] 52fe5467deaeb5d9cfce4c7987dd4b7e9397b870cd896559cc9c44f0d7ab6adf

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 356728cf6a4581982f98d106fbefd04dc4c0b20e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: the step now tests the missing boundary directly, with positive controls on both sides, so a pass cannot come from a probe that ran too early or a broken sandbox.

Notes

  • Nobody has run this on a hosted runner yet. If Landlock's ptrace scoping or the bubblewrap /proc view acts differently there, every fork review fails closed. A revert undoes that, but expect the first fork run to be the real check.

[DESIGN-REVIEWED] 356728c

Description read

[DESCRIPTION-READ] 192f16f02c80529561375ee658620e8e3dadfb41c02836e1a0405e3213f933d9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 356728cf6a4581982f98d106fbefd04dc4c0b20e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 356728c

Verdict parsed from the review's SHA-scoped output markers for commit 356728cf6a4581982f98d106fbefd04dc4c0b20e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 356728cf6a4581982f98d106fbefd04dc4c0b20e false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 356728cf6a4581982f98d106fbefd04dc4c0b20e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 356728c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 356728cf6a4581982f98d106fbefd04dc4c0b20e false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 5, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 5, 2026 13:43
The fork review lane hands its model a shell on an untrusted diff. Before
the model starts, the lane now proves that this shell cannot reach what
other job processes hold, and refuses to review when it can, instead of
assuming the boundary holds on the runner it landed on.

On a runner where the boundary holds the lane runs exactly as before.
@bolichen97
bolichen97 force-pushed the fix/13414-review-sandbox-check branch from fee1ac7 to 356728c Compare November 5, 2026 18:15
@bolichen97

bolichen97 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: bounded unconfined poll for the sibling's marker before the sandboxed environ read

  • span=6b25d1da99cb process-environment probe races the sibling exec

fixed in 356728c: before the sandboxed read, the step polls /proc/$probe_pid/environ outside the sandbox (100 x 0.1s) until KC_SANDBOX_PROBE=sibling is there, and fails closed if it never appears.
The unconfined read is the positive control the finding asked for, so an early sandboxed read can no longer pass falsely.
New test test_it_fails_closed_when_the_marked_sibling_never_shows_its_marker covers the fail-closed branch; prove.py: PROVEN with the poll reverted.

@bolichen97

Copy link
Copy Markdown
Contributor Author

macOS Tests / Backend Tests (macOS) (6) on fee1ac7 failed in test/test_subagent_stop_reason_consistency.py::test_whatever_lands_after_the_claim_the_run_is_completed_once[deadline], which this PR does not touch. The captured log shows the claimed result write's own 5s drain bound expired on the loaded runner before the test checked it (result complete write ... did not finish in 5s), so the run was already done. That is the race open PR #28247 fixes in that test; nothing in this diff changes it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 5, 2026

@buluoray buluoray left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:

Approve. Before the fork review model gets a shell, the fork review lane now proves that the model's sandbox cannot read another job process's environment. If the sandbox can read it, the lane stops instead of passing quietly. Fork PR authors see no difference on a runner where the boundary holds.

Evidence at 356728c:

  • The probe sits in the existing "Verify the review sandbox fails closed" step (fork-gpt-review.yml:827-850), which runs before both GPT passes (:853, :1223). Every new branch exits 1: the marker never appears unconfined (:832-839), the checkout is unreadable under the sandbox (:842-845), or the sibling's environ is readable under the sandbox (:846-849). The sibling starts under env -i with only the marker, so it carries nothing from the job.
  • The diff changes no trigger, permission, secret reference or checkout ref. The step's env is unchanged.
  • test_ai_review_workflows_credential_isolation.py: 63 passed. Mutations were run one at a time, each restored before the next. Removing the environ refusal reddens test_it_fails_closed_when_another_process_environment_is_readable[False]. Replacing the unconfined marker wait reddens test_it_fails_closed_when_the_marked_sibling_never_shows_its_marker, which shows the earlier race finding is fixed. Moving the step after the discovery pass reddens test_the_step_exists_and_runs_before_both_codex_passes.
  • CI: 82 passed, 12 skipped, none failing. PR Readiness green. Merges cleanly onto main.

Not verified here: the probe has not yet run on a hosted runner, because this lane runs from the default branch. The claim that the pinned codex sandbox blocks the read in both Linux modes rests on the manual run in the description. Please watch the first fork review after merge. If it fails at the new probe, the runner's sandbox does not hold that boundary, and that needs a fix or a revert rather than a rerun.

Non-blocking follow-ups: nothing reddens if the checkout-read control (:842-845) is removed. The step name and the closing echo still mention only sudo and the credential file.

@bolichen97
bolichen97 merged commit 41abe00 into main Nov 5, 2026
98 checks passed
@bolichen97
bolichen97 deleted the fix/13414-review-sandbox-check branch November 5, 2026 20:02
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fork GPT review: the /proc and /sys read deny for the model shell is unverified and hard-fails the lane on the legacy sandbox backend

2 participants