Skip to content

ci(fork-review): verify the sandbox read restriction before the lane runs - #19812

Closed
cruisercohen wants to merge 1 commit into
mainfrom
t1tail-13414
Closed

cruisercohen wants to merge 1 commit into
mainfrom
t1tail-13414

Conversation

@cruisercohen

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make the fork review lane prove its sandbox read restriction is actually enforced before the model runs, and fail closed when it cannot.

The fork review lane hands the model a shell on an untrusted diff under a read-only sandbox. The step that checks the sandbox before the model runs only verified that privilege escalation and the credential file were blocked — it never exercised a denied read, so it could not tell the enforcing sandbox backend apart from a weaker fallback that leaves the read restriction unenforced.

Why it matters

A review that proceeds behind a read restriction that is configured but not actually in force gives false confidence. The lane runs on untrusted fork input, so the self-check should confirm the restriction holds — or refuse to run — rather than assume it.

Not a goal

  • No change to what the review passes themselves read; the passes keep their existing read-only configuration.
  • Not the same-repo review lane; this is scoped to the fork lane called out in the issue.

What changed (motivation → approach → change)

The pre-model self-check asserted sudo was blocked and the credential file was unreadable, but read-only mode permits every read, so no denied read was ever tested — the check could not distinguish the enforcing backend from a fallback that does not confine reads.

Measured against the pinned review CLI on a hosted runner: the enforcing backend reliably refuses a denied read of a regular path, while the restriction is simply not applied when that backend is unavailable (the sandbox then fails to start rather than silently downgrading).

So the self-check now adds a probe that applies a read-deny profile to a throwaway sentinel file and confirms, through the same confinement the passes use, that the sentinel read is refused while an ordinary read still works. If the enforcing backend cannot apply the restriction, the step fails closed with an actionable message instead of reviewing behind a fence that is not in force. The change is in .github/workflows/fork-gpt-review.yml (the verify step) plus a workflow-shape test.

Backwards compatibility

Compatible: adds a verification probe to an existing step; it changes no review-pass behavior and no configuration the passes rely on. On a healthy runner the probe passes (verified end to end); it only turns a silent weaker-fence case into a clear, failing one.

Tests

A new workflow-shape test asserts the verify step applies a read-deny profile, does not suppress it with a sandbox-mode override, and fails closed both when a denied read succeeds and when the enforcing backend cannot be established. It fails on the pre-change workflow and passes after.

Manual verification

The exact verify-step script was run on a hosted runner with the pinned CLI: all probes pass and the step exits 0 when the enforcing backend is available, and the sandbox fails to start (so the step fails closed) when it is not.

Issue link

Fixes #13414

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 4edb8e91aa0b9ca49afb4bc9bc9c9c9fd7a63f47 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 4edb8e91aa0b9ca49afb4bc9bc9c9c9fd7a63f47 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The probe checks a read-deny rule that no review pass uses, and "Fixes #13414" would close the issue while the /proc/<pid>/environ credential exposure stays open.

Watch

  • What the probe proves is not what the passes run under. Cause: the probe runs with env CODEX_HOME=$VERIFY_HOME and a throwaway profile, while the description says "the passes keep their existing read-only configuration" (sandbox_mode = "read-only", no default_permissions). Mechanism: the step proves bubblewrap can enforce a deny rule on a separate config. It does not prove anything about the config the passes load, which has no read restriction at all. Consequence: the goal line, "prove its sandbox read restriction is actually enforced", describes a restriction that does not exist yet. Meanwhile the lane now fails closed for every fork PR whenever bubblewrap is unavailable, even though none of its passes depend on bubblewrap.
    Clears when: the description changes "Fixes #13414" to "Refs #13414" and says this is the issue's step 2 prerequisite, or the PR adds the /proc and /sys deny to the staged config.toml so the probe guards a rule the passes actually load.
  • The probe tests a stand-in, not the path the issue is about. Cause: SENTINEL="$RUNNER_TEMP/sandbox-verify-sentinel" is an ordinary file. Mechanism: the issue's step 1 asks to show that a sandboxed command "cannot read another process's environ". procfs under a user namespace is a different mount and may not honour a path deny the way a temp file does. Consequence: when the /proc deny lands later, this probe can pass while /proc/<pid>/environ is still readable, which is exactly the false confidence the issue warns about.
    Clears when: the probe denies /proc and confirms that reading a same-UID process's /proc/<pid>/environ is refused, or a hosted-runner measurement showing it is refused is linked.

[DESIGN-REVIEWED] 4edb8e9

Description read

[DESCRIPTION-READ] 336339a022ebaa14b2a52e46f45635a17aeab760c57924f553f41281216152f8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🔴 BLOCK

Premise-level review of 4edb8e91aa0b9ca49afb4bc9bc9c9c9fd7a63f47 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: BLOCK

Probe 3 checks a read-deny rule the review passes never apply, and the author's own measurement says the existing $SANDBOX true control already catches a missing backend.

Not justified as shipped

  • Item 1 — zero consumers: the passes' config is sandbox_mode = "read-only" and has no [permissions.*] table. So the probe's "$SENTINEL" = "deny" checks a restriction no pass relies on, and the ## Not a goal keeps it that way ("No change to what the review passes themselves read").
  • Item 2 — rides along: the test pins Item 1's shape and nothing else.

What this change ships

Inventory (2 items) — 0 justified

Intent: make the fork review lane refuse to run when its sandbox's read confinement isn't really enforced. This is a FIX.

  1. The fork-review verify step now also fails when a profile-denied sentinel is readable or the profile sandbox won't start. — zero consumers
  2. A new workflow-shape test pins that probe. — rides along

Blockers

Probe verifies a fence nothing uses; its zero option costs nobody anything
The description says the probe works "through the same confinement the passes use". The hunk runs env CODEX_HOME=$VERIFY_HOME $CODEX_BIN sandbox --, which uses a different home, a profile mode and no read-only. The description also says that without the backend "the sandbox then fails to start rather than silently downgrading". The existing if ! $SANDBOX true already fails closed in exactly that case.
Subtraction: delete Probe 3 and TestForkGptLaneVerifiesTheReadRestrictionIsEnforced.
Clears when: the probe is deleted, or a pass actually applies a deny rule that this probe exercises.

[FIRST-PRINCIPLES-REVIEWED] 4edb8e9

Description read

[DESCRIPTION-READ] 57059db48b93edff199a0996e5f5fab451c2c0f86c9fa5e078c3660d16048d44

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4edb8e91aa0b9ca49afb4bc9bc9c9c9fd7a63f47 — this comment is updated in place on each push.

Review details

Finding: the new read-restriction probe never tests the configuration the review passes run under.

FINDING — .github/workflows/fork-gpt-review.yml:846 — Probe 3 runs env CODEX_HOME=$VERIFY_HOME $CODEX_BIN sandbox -- against a throwaway default_permissions = "verify" profile, but the codex exec passes (line 1140) load $HOME/.codex/config.toml. That file carries only sandbox_mode = "read-only" and no deny rule, so on every fork PR the step prints "read restriction enforced" and claims "the SAME confinement the passes use" even though the passes have no read restriction → Fix: put the deny profile in the real $HOME/.codex/config.toml and probe through that same CODEX_HOME, or reword the comment and success message so they claim only that the backend can enforce a deny rule.

[OPUS-REVIEWED] 4edb8e9

Verdict parsed from the review's SHA-scoped output markers for commit 4edb8e91aa0b9ca49afb4bc9bc9c9c9fd7a63f47.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4edb8e91aa0b9ca49afb4bc9bc9c9c9fd7a63f47: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4edb8e91aa0b9ca49afb4bc9bc9c9c9fd7a63f47 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4edb8e9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4edb8e91aa0b9ca49afb4bc9bc9c9c9fd7a63f47: <one-sentence reason>

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 15, 2026
@cruisercohen

Copy link
Copy Markdown
Contributor Author

Rebased onto main 83e97cd after the #20347 split; test/test_ai_review_workflows.py -> test/test_ai_review_workflows_credential_isolation.py where moved, no behaviour change.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 15, 2026
…runs

The fork review lane hands the model a shell on an untrusted diff under a read-only sandbox, but the pre-model self-check never exercised a denied read, so it could not tell the enforcing backend's filesystem confinement apart from a weaker fallback that leaves reads unrestricted.

Add a probe to the existing verify step that applies a permission profile denying a sentinel file and confirms, through the same confinement the passes use, that the denied read is refused while an ordinary read still works. When the enforcing backend cannot apply the restriction, the step now fails closed with an actionable message instead of reviewing behind an unenforced fence.

Fixes #13414
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 21, 2026
@cruisercohen

Copy link
Copy Markdown
Contributor Author

Closing this draft unmerged. The review lanes found that the probe checks a deny profile that no review pass actually loads, so this PR can't land as scoped. The real /proc deny for the fork lane waits on the hosted-runner measurement that #13414 asks for; #13414 stays open for that. The branch is kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fork GPT review: the /proc and /sys read deny for the model shell is unverified and hard-fails the lane on the legacy sandbox backend

1 participant