Skip to content

test: allow the sandbox tool marker and linux-gate the window pin test - #15680

Merged
iamwhatever merged 1 commit into
mainfrom
fix/macos-sandbox-test-markers
Oct 1, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/macos-sandbox-test-markers

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Main's macOS backend suite goes green again by fixing two test-side gaps that recent main merges left behind.

The macOS backend suite fails two tests on main. Neither failure is caused by the PR it surfaced on (job, macOS shard 4 of the #15309 run):

  • test/test_terminal_commands.py::TestRunProbe::test_the_environment_is_an_allowlist_not_a_filtered_inherit fails with Extra items in the left set: 'KIROCREW_SANDBOX_TOOL'.
  • test/test_sandbox_mount_pinned_target.py::test_the_directory_loop_records_every_window_it_binds fails with AttributeError: module 'os' has no attribute 'O_PATH'.

Why it matters

Every PR that runs the macOS suite (ci:macos) is red for a cause it did not introduce. A red that everyone has learned to ignore also hides real macOS regressions.

Not a goal

What changed (motivation → approach → change)

Test Why it fails Fix
test_the_environment_is_an_allowlist_not_a_filtered_inherit #15528 stamps KIROCREW_SANDBOX_TOOL=1 on every tree sandboxed_spawn_argv spawns, so the runtime reconciler can read it back from the child. The test's named allowlist predates it. Name it in sandbox_injected, and plant a leaked- value for it in the parent, the way GRADLE_OPTS already has one.
test_the_directory_loop_records_every_window_it_binds #15564's test drives the private-window walk, which opens each path component with bare os.O_PATH. Darwin has no O_PATH. Add the file's own @_LINUX_LINK_PIN skip marker, which the other cases that need O_PATH already carry.

The allowlist test runs only where a live sandbox backend exists. GitHub's Linux runners have none, so the Linux shards skip it, and only macOS showed the failure. The leaked- value keeps the check honest: the chokepoint overwrites the name with its own 1, so a parent value reaching the child would mean the stamp turned into an inherit. The _LINUX_LINK_PIN comment now names the window case too.

Backwards compatibility

Compatible: test-only change; no runtime behaviour, contract or default changes.

Tests

  • The allowlist test pins KIROCREW_SANDBOX_TOOL as an expected sandbox marker. Its new leaked- sentinel fails the test if the chokepoint ever copies a parent's value into the child under that name. Mutation check: when the stamp copies the parent's value, the test goes red on leaked-.
  • The window-bind test now skips off Linux, and still runs on Linux.

Manual verification

On a Linux host with a live namespace backend:

  • Main fails the allowlist test with the same extra name as macOS. With this change, test/test_terminal_commands.py passes in full.
  • A scratch plugin simulated Darwin: sys.platform reads darwin at module import, and os.O_PATH is removed for the run. test_sandbox_mount_pinned_target.py then gives 39 passed, 6 skipped, 0 failed, with the window test among the skips.
  • With only os.O_PATH removed on Linux, exactly the _LINUX_LINK_PIN tests and the window test fail. So no other test in the file needs the marker.

macOS itself is verified by this PR's ci:macos run.

Related Issues

no linked issue: main-red fix for #15528 and #15564; no issue tracks it.

Open #14008 carries an equivalent fix for both tests inside its feature diff. Whichever lands second resolves the conflict on these lines.

Pattern harvest

Not generalizable: each failure is a missing entry next to a convention the file already follows. The allowlist names every injected marker, and each O_PATH case carries _LINUX_LINK_PIN. A new marker or a new O_PATH case updates the list the way these two do.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 1, 2026 02:26
@bolichen97 bolichen97 added the ci:macos Run the on-demand macOS suite against this PR head label Oct 1, 2026
Two tests on main fail in the macOS backend suite, each for its own
reason.

- test_terminal_commands.py::TestRunProbe::
  test_the_environment_is_an_allowlist_not_a_filtered_inherit finds
  KIROCREW_SANDBOX_TOOL in the probe child's environment. #15528 stamps
  that marker on every tree spawned through sandboxed_spawn_argv on
  purpose, because the runtime reconciler reads it back from the child.
  So it joins the named sandbox_injected set, with a leaked- sentinel in
  the parent like GRADLE_OPTS has, which keeps the check honest: the
  chokepoint overwrites the name, and a parent value must never reach
  the child under it. The test runs only where a live sandbox backend
  exists. GitHub's Linux runners have none, so the Linux shards skip it.

- test_sandbox_mount_pinned_target.py::
  test_the_directory_loop_records_every_window_it_binds, added by
  #15564, drives the private-window walk. That walk opens each component
  with bare os.O_PATH, which only Linux has, so the test raised
  AttributeError on Darwin. It now carries the file's _LINUX_LINK_PIN
  skip marker, like the other cases that need O_PATH.

Failing job (macOS shard 4 of the PR #15309 run):
https://github.com/kirodotdev/KiroCrew/actions/runs/36798238474/job/110167010707
@bolichen97
bolichen97 force-pushed the fix/macos-sandbox-test-markers branch from d73ade0 to a3cf60c Compare October 1, 2026 02:41
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: onto main 7460284 (#15633, the Windows alias-lock fix) so Windows and macOS run on a fixed base. The patch is unchanged (git range-diff reports =). The new head is a3cf60c; CI is re-running.

The reds on the previous head d73ade0 were all infra, hit before any verdict was produced: API rate limit exceeded for installation (HTTP 403).

  • Build / Resolve desktop matrix, job 110189939505, request F5C4:2606E5:192CB97:533B06A:6ABDC513
  • Design Review, "Resolve human override", job 110189939558, request 4000:22845D:174DBF5:4D1928E:6ABDC562 (it could not read the PR's comments)
  • GPT 5.6, Opus 5, First Principles and UX failed in the same "Resolve human override" step at the same time

The old head's still-queued runs (CI, macOS on demand, CodeQL) were cancelled because the new push supersedes them.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of a3cf60c80a7c0ffd902a55917e167744ba786e9b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm this PR's ci:macos run is green before merge — that run is the only direct proof of the fix on the failing platform.

[DESIGN-REVIEWED] a3cf60c

Description read

[DESCRIPTION-READ] c738d8fc726760c14d4da9af0eee4abcae522e40e4d56ce5b73ebdf50789cbcb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed a3cf60c80a7c0ffd902a55917e167744ba786e9b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a3cf60c

Verdict parsed from the review's SHA-scoped output markers for commit a3cf60c80a7c0ffd902a55917e167744ba786e9b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a3cf60c80a7c0ffd902a55917e167744ba786e9b: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ human override accepted

Human judgment by @bolichen97 overrides the GPT 5.6 finding for a3cf60c80a7c0ffd902a55917e167744ba786e9b; the recorded reason is authoritative for this commit.

[GPT-OVERRIDE] a3cf60c

This comment is updated in place on each push.

The model was not re-run because an authorized human decision supersedes it.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a3cf60c80a7c0ffd902a55917e167744ba786e9b: <one-sentence reason>

@bolichen97

bolichen97 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=f865286abb86 newly added platform skip loosens the ratchet

Outcome: rebutted (not a defect). The skip removes no coverage that ever ran. Off Linux, the test could only raise AttributeError: the window walk opens every component with bare os.O_PATH (sandbox.py:9382, 9387) and stages from /proc/self/fd (sandbox.py:9448), and the namespace launcher runs only on Linux.
The exact-set assertion _BOUND_WINDOWS == {window} is unchanged and still runs on every Linux backend shard, where the launcher executes. It is the same gate the file's other O_PATH-dependent cases carry (L681, L737, L812, L1215).
The test was added on main by #15564 and has never passed on Darwin (macOS shard 4 of run 36798238474). This PR does not loosen it to fit new code; it scopes it to the only OS where its subject exists.
A harness O_PATH shim would let a Linux-only kernel path pretend to run on Darwin and add no real coverage, which the lane's own Opus adjudication also judged.
A security-fenced finding needs a human override; this writer does not post one.

@bolichen97

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt a3cf60c: The skip fires only off Linux, where the launcher under test cannot execute at all (bare os.O_PATH at sandbox.py:9382 and :9387, and a /proc/self/fd bind source at sandbox.py:9448), so the exact-set window-binding invariant loses no reachable coverage and still runs on every Linux CI job. The gate is the file's existing _LINUX_LINK_PIN, already carried by the sibling tests at L679/L735/L810/L1212; this test was added by #15564 and never passed on Darwin.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for a3cf60c80a7c0ffd902a55917e167744ba786e9b.

The skip fires only off Linux, where the launcher under test cannot execute at all (bare os.O_PATH at sandbox.py:9382 and :9387, and a /proc/self/fd bind source at sandbox.py:9448), so the exact-set window-binding invariant loses no reachable coverage and still runs on every Linux CI job. The gate is the file's existing _LINUX_LINK_PIN, already carried by the sibling tests at L679/L735/L810/L1212; this test was added by #15564 and never passed on Darwin.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 1, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Test-only: the allowlist assertion gains KIROCREW_SANDBOX_TOOL (and a planted parent value proves the chokepoint overwrites it), and the O_PATH window-pin test is Linux-gated. These are the two macOS shard reds currently inherited by other PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run the on-demand macOS suite against this PR head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants