Skip to content

ci: warn on network reach in changed test files - #24515

Merged
bolichen97 merged 1 commit into
mainfrom
ci/net-strict-changed-tests
Oct 27, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
ci/net-strict-changed-tests

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A test file that a pull request adds or changes is checked against the loopback-only rule (D10) before merge, and a file that breaks it is named in the job summary.

The rootdir network floor already sees a test that looks up a non-local name or connects past loopback. In CI it only reports, in a warnings summary at the end of a 20-40 minute shard, and nobody reads it. KIROCREW_NET_STRICT=1 would make the call raise, but no workflow sets it. In the 15-loop main-red cluster, the floor named test_governance_policy_viewer.py (name lookup 'governance.example.invalid') in the log and nobody saw it.

Why it matters

A lookup that is harmless on Monday can turn fatal after an unrelated merge, and then every open pull request pays a CI loop. Naming the test while its author is still on the PR is the cheapest point to fix it.

Not a goal

  • No strict mode for the whole suite. Six existing files (and others the floor has not shown) look up non-local names; they stay as they are until someone edits them.
  • No failing check in this release. The step is continue-on-error: true.
  • No change to the floor, the real_network marker or rule D10 itself.
  • No change to the main-red watch.

What changed (motivation → approach → change)

Strict mode already exists and works (conftest.py raises OffLoopbackNetworkRefused at the call and fails the test at teardown; a real_network mark and a UDP route probe are let through). What is missing is a place that turns it on without a baseline.

New scripts/net_strict_changed.py takes the changed test files from scripts/ratchet_scope.py (change_merge_base, change_files), the same base and file set the flake-pattern ratchet reads, and uses that ratchet's own FLAKE_RATCHET_BASE override. It has no --timeout flag and no NET_STRICT_BASE override of its own: nothing called either. It runs KIROCREW_NET_STRICT=1 pytest on those files, reads the junit file with defusedxml, and renders a report. It is a step of the existing flake-pattern-ratchet job (Linux only, one leg), with continue-on-error: true. The job cap goes 10 to 15 minutes so the extra steps cannot turn it red.

The step runs as the unprivileged ci-shell user, which cannot write $GITHUB_STEP_SUMMARY on CodeBuild (run-as-runner protects the file-command directory). So the script writes the report to $RUNNER_TEMP/net-strict-report.md, and a new step, Publish the network report to the job summary, appends it to the summary under the runner identity. That step runs no workspace file, starts Python with -I -S (so a sitecustomize.py a test left in the checkout cannot run as that identity), opens the report without following a link, takes a regular file only, and reads at most 16 KiB.

Edge cases: no test file changed, an unreadable base, and a deleted file skip with a written reason and exit 0. A report that is missing, unreadable or declares an XML entity is not called clean: the report says Nothing was measured and the step exits 1 (still warn-only). A file name comes from the pull request, so it must start under a test root, may not contain :: or a control character, and goes to pytest as an argument list after --; no shell sees it. Failures that are not a network refusal are counted, not judged. The job's header comment no longer says it only scans statically, since the last two steps now run the PR's test code.

Backwards compatibility

Compatible: nothing that passes today fails. The new step cannot change the job or check result, and no existing script, marker or setting is renamed or tightened.

Removes nothing: only a step, a script and two doc sentences are added.

Tests

test/test_net_strict_changed.py (31 tests):

  • which files are picked: name rules, a deleted file, a rename, a file not on disk, a change with no test file, an unreadable base (real scratch git repo);
  • a changed file with a non-local name lookup is reported, an unchanged file with the same lookup is not run and not named, loopback and @pytest.mark.real_network are not reported (a real pytest child with the real floor; the probes raise sys.audit lookup events, so no test makes a real lookup);
  • the report: both strict failure spellings are one finding, other failures are counted, a hostile detail cannot forge a log line, a missing, broken or entity-bearing junit file reads Nothing was measured and never "clean", and main() writes the report file, never $GITHUB_STEP_SUMMARY, and exits 1 on a finding;
  • the publish step: it starts python3 -I -S -, ignores a planted sitecustomize.py (a control run without the flags runs it), and publishes nothing for a symlink.

test/test_prepare_pr_profiles.py lists the script as a warn-only exemption from the prepare-pr floor (one comment and one line).

Mutation: with the child's KIROCREW_NET_STRICT set to 0 the changed-file test fails.

Proof: not provable -- new tooling with new tests, no production hunk to revert.

Manual verification

Ran python scripts/net_strict_changed.py on this branch with NET_STRICT_REPORT set: it picked the new test file, ran it strict, wrote No test reached past loopback. to the report file, exit 0. Ran the publish step's inline Python locally against an absent file (nothing written), a normal file (copied), a symlink and a FIFO (nothing), and a 40 KB file (cut to 16 KiB). Not run on a GitHub or CodeBuild runner: ci-shell, python3 in the publish step and the $RUNNER_TEMP and summary permissions are read from the neighbouring steps and the boundary check, not observed.

Related Issues

Design: main-red study, section 4 and plan row 3.

Sibling sweep

Swept: git grep -n KIROCREW_NET_STRICT -- .github scripts
Siblings: none -- no other workflow or script sets the strict switch; only the floor reads it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 26, 2026 18:03
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 26, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head bfa37a8.

The approach is sound. Turning on KIROCREW_NET_STRICT=1 only for the test files a change adds or changes avoids a baseline, and it reuses the same base and file set as the flake-pattern ratchet. I checked these points against main:

  • The two failure strings in scripts/net_strict_changed.py match the floor on main: off-loopback network refused in a test: (root conftest.py raise site) and this test reached past loopback under KIROCREW_NET_STRICT=1: ... . Stub the network (_determinism_floors_end).
  • change_merge_base(env_name, *, cwd=...) and change_files(base, *, cwd=...) in scripts/ratchet_scope.py have the signatures the script calls.
  • File names from the PR pass through is_runnable_name and go to pytest as an argument list after --. No shell sees them.
  • Workflow trust read of the .github/workflows/fast-gate.yml hunk: no secret or token use, no permissions: change, no new trigger, no remote download. The new step runs PR test code through ci-shell in a job that already runs PR test code (test/test_flake_pattern_ratchet.py and the PR's conftest), with persist-credentials: false. The job cap goes from 10 to 15 minutes, and the step has its own 5 minute cap above the script's 240 second pytest limit.

Two small items, neither blocks:

  1. A pytest run that collected nothing reads as clean. See the inline comment on scripts/net_strict_changed.py line 260.
  2. The job header comment in .github/workflows/fast-gate.yml line 606 still says Static scan only: a git diff plus the changed files. After this change the job also executes the changed test files. A reader who checks this job's trust boundary relies on that line. Suggest: Static scan plus one warn-only step that runs the changed test files with KIROCREW_NET_STRICT=1.

The PR changes .github/ and adds CI tooling, so a maintainer gives the final approval. The required checks were still running when I read it.

Comment thread scripts/net_strict_changed.py Outdated
return 0
with tempfile.TemporaryDirectory() as scratch:
junit = Path(scratch) / "junit.xml"
finished, _code = run_strict(files, root=ROOT, junit=junit, timeout=args.timeout)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 A pytest run that ran nothing is reported as "No test reached past loopback."

The exit code is discarded here (_code). When pytest stops before it writes a junit file (a usage error, exit 4, or an internal error, exit 3), parse_report returns ([], 0) and render prints No test reached past loopback. with exit 0. The step then looks like a clean check that checked nothing. This is the same outcome the ratchet step above turns into a red: "a gate that compared nothing is not a pass".

        finished, _code = run_strict(files, root=ROOT, junit=junit, timeout=args.timeout)
        findings, others = parse_report(junit)

Suggestion: keep the code and treat "no junit file" or an exit code other than 0, 1 or 5 as "did not finish", for example finished = finished and junit.is_file() and code in (0, 1, 5). Then the existing "the result below is partial" line and exit 1 cover it. A test that runs main with a run_strict stub returning (True, 4) and no junit file pins it.

If the step is meant to stay silent on a broken run during the warn-only release, say so in the docstring and I will withdraw this.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a001fa456e082eca4ffb3f698bcff77e8c744214 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a001fa456e082eca4ffb3f698bcff77e8c744214 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The step only warns, and its one surface without a consumer is two parameters that can be removed.

Notes

  • Item 7 — zero consumers: main() is the only production caller of run_strict, and it passes neither extra_args nor timeout. The one other caller is in the test (extra_args=("-p", "conftest"), timeout=240), and 240 is already DEFAULT_TIMEOUT. Subtraction: delete the timeout parameter, and either delete extra_args or have the test set the plugin through the environment it already controls.

What this change ships

Inventory (7 items) — 6 justified

Intent: a test file added or edited in a pull request that reaches past loopback (D10) gets named in the job summary before merge. ADDITION.

  1. Pull requests that touch test files run those files with KIROCREW_NET_STRICT=1, and a summary section names each test that reached past loopback — justified
  2. A new step copies the report file into the job summary under the runner identity, reading a regular file only, at most 16 KiB — justified
  3. The Flake Pattern Ratchet job cap goes from 10 to 15 minutes — justified
  4. A missing or unreadable junit report reads "Nothing was measured", and the step exits 1 (still warn-only) — justified
  5. The prepare-pr floor exempts net_strict_changed.py as warn-only — justified
  6. Two doc sentences (ci-and-reviews, testing-conventions D10) describe the step — justified
  7. run_strict takes extra_args and timeout parameters — zero consumers outside tests

[FIRST-PRINCIPLES-REVIEWED] a001fa4

Description read

[DESCRIPTION-READ] 7f32a8e68d878dac73bbd218d8cce880aedd977ec875eaa692baa0f8b5ba4b03

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a001fa456e082eca4ffb3f698bcff77e8c744214 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a warn-only CI step for one real, named miss, it reuses the ratchet's file-scoping, and it removes nothing.

Notes

  • Once the warning has run for a while, make the step a failing check. Until then it is still a report that a person has to open, which is the same visibility gap the PR describes.

[DESIGN-REVIEWED] a001fa4

Description read

[DESCRIPTION-READ] 3beb526ca3250d3b5675f6972da57668e03a798e29f4dd6b6fb254b069880729

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a001fa456e082eca4ffb3f698bcff77e8c744214 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a001fa4

Verdict parsed from the review's SHA-scoped output markers for commit a001fa456e082eca4ffb3f698bcff77e8c744214.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a001fa456e082eca4ffb3f698bcff77e8c744214 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a001fa456e082eca4ffb3f698bcff77e8c744214 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a001fa4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a001fa456e082eca4ffb3f698bcff77e8c744214 false-positive: <reason, 20 to 500 characters>

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head ffff183, as a delta from bfa37a8.

Both items from my last review are fixed:

  1. A run that measured nothing no longer reads as clean. parse_report now returns None for a missing, broken or entity-bearing junit file, render then says Nothing was measured with the exit code or the timeout, and main exits 1. A pytest usage error or internal error writes no junit file, so it now lands on that path. test_a_run_that_measured_nothing_never_says_it_was_clean and the last block of test_main_writes_the_report_file_and_never_the_step_summary pin it.
  2. The job header comment in .github/workflows/fast-gate.yml no longer says the job is a static scan only.

Other delta points I checked: change_merge_base(cwd=root) relies on the default env_name="FLAKE_RATCHET_BASE" in scripts/ratchet_scope.py at this head, so the shared override still applies. defusedxml is in install_requires in setup.cfg, which the job installs with -e ., so the import does not fall into the None path on CI. The probe fixtures now raise sys.audit events instead of real lookups.

One blocking item, in the new publish step. It agrees with the GPT lane finding on this head; I traced it myself:

  • On a root CodeBuild runner, run-as-runner chowns $GITHUB_WORKSPACE to the unprivileged runner user, so the strict step's test code can create src/sitecustomize.py.
  • setup.cfg maps the package to src and the job installs it editable, so src/ is on the site path.
  • Publish the network report to the job summary then runs a bare python3 - as root, outside ci-shell. Python's site imports sitecustomize at startup, before the heredoc's O_NOFOLLOW and regular-file checks run.

Details and a one-line fix are in the inline comment.

The PR changes .github/, so a maintainer gives the final approval after the fix.

Comment thread .github/workflows/fast-gate.yml Outdated
if: always()
continue-on-error: true
run: |
python3 - <<'PY'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The root publisher starts Python with site enabled, so test code can run as root here.

The step before this one runs the pull request's test files as the unprivileged user, and on a root CodeBuild runner that user owns the workspace. A changed test can write src/sitecustomize.py. The editable install puts src/ on the site path, and this step runs python3 - as root outside ci-shell, so site imports that module at startup. That happens before the O_NOFOLLOW and S_ISREG checks, so they do not help. The test code then runs with the runner identity this job set out to keep it away from, including the file-command directory in $RUNNER_TEMP.

        run: |
          python3 - <<'PY'

Suggestion: start the publisher isolated and without site, so it imports only the standard library already loaded at startup:

          python3 -I -S - <<'PY'

-I drops the working directory and the PYTHON* variables from the path, and -S skips site, so no .pth file and no sitecustomize runs. The script only needs os and stat. A plain shell form such as [ -f "$p" ] && [ ! -L "$p" ] && head -c 16384 "$p" >> "$GITHUB_STEP_SUMMARY" would also work, but it has a small check-then-read gap that the fd form does not.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 26, 2026
The rootdir network floor reports an off-loopback connect or name
lookup, but CI never sets KIROCREW_NET_STRICT, so the report sits at
the end of a shard. Add scripts/net_strict_changed.py: it runs only the
test files the change adds or changes (same diff scope as
ratchet_scope) with the strict switch on and renders a report.

It is a step of the flake-pattern-ratchet job with continue-on-error,
so nothing newly fails. The step runs as the unprivileged ci-shell
user, which cannot write the job summary, so it writes a report file
and a following step publishes it under the runner identity (python3
-I -S, no link followed, regular file only, 16 KiB cap). The prepare-pr
floor test lists the script as a warn-only exemption. No baseline is
touched.
@iamwhatever
iamwhatever force-pushed the ci/net-strict-changed-tests branch from ffff183 to a001fa4 Compare October 27, 2026 01:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head a001fa4, as a delta from ffff183.

The blocking item from my last review is fixed. Publish the network report to the job summary now starts python3 -I -S -, so the root step skips site (no .pth from the editable install, no sitecustomize) and drops the working directory and PYTHON* variables from the path. A sitecustomize.py that test code leaves in the workspace or in src/ no longer runs under the runner identity. The script only needs os and stat, which load without site.

The fix is pinned by two new tests in test/test_net_strict_changed.py: one asserts the step starts with python3 -I -S -, has no shell: override, and keeps continue-on-error and if: always(); the other runs the real step body with a planted sitecustomize.py, first proves a bare python3 - executes it (control), then proves the shipped form does not, and that a symlinked report publishes nothing.

Other root paths in the delta, checked at this head:

  • The publish step is the only new step outside ci-shell. It reads a fixed path in $RUNNER_TEMP (root-owned, sticky) with O_NOFOLLOW and O_NONBLOCK, checks S_ISREG on the open descriptor, and caps the read at 16 KiB, so a link, FIFO or swapped file cannot redirect it.
  • python3 resolves to the setup-python toolcache interpreter, which run-as-runner does not hand to the unprivileged user, so test code cannot replace the binary or its stdlib.
  • The strict step itself runs under ci-shell, and GITHUB_ENV, GITHUB_PATH and GITHUB_STEP_SUMMARY stay unwritable to it, so it cannot change the environment or PATH of the publish step.
  • No new secret use, permissions: change, trigger change, or download-and-execute.

I have no further findings. The PR changes .github/, so a maintainer gives the final approval.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 27, 2026

@CrysisDeu CrysisDeu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at a001fa45: Approve

All three of @bolichen97's points are fixed and pinned: "ran nothing reads as clean" (net_strict_changed.py:138-158,230-236,281), the stale "Static scan only" header (fast-gate.yml:606), and the root publisher now runs python3 -I -S - (fast-gate.yml:696, two tests incl. the planted-sitecustomize control). Workflow security holds: pull_request trigger, contents: read, persist-credentials: false, file names through is_runnable_name and passed as argv after --, no shell interpolation. Both steps are continue-on-error, so warn-only as stated. CI green, MERGEABLE.

Non-blocking:

  1. MINOR — net_strict_changed.py:278-281: a collection error (pytest exit 2, one <error> testcase) still renders "No test reached past loopback" and exits 0 — the same "clean check that checked nothing" shape as round 1. Treat code not in (0, 1, 5) as not-measured, or pass --continue-on-collection-errors; add a test with a (True, 2) stub.
  2. NIT — run_strict(extra_args=, timeout=) have no production caller (:177-184).
  3. NIT — the network step has no if: always() (fast-gate.yml:676), so a red ratchet skips the D10 warning; say so in the comment or add it.

Note: this approval does not clear @bolichen97's CHANGES_REQUESTED at ffff183f; they said "no further findings" at this head, so they need to approve or a maintainer dismisses.

Posted by code-reviewer (Kiro Crew member) on behalf of @CrysisDeu. Full report on file.

@bolichen97
bolichen97 enabled auto-merge (squash) October 27, 2026 06:45
@bolichen97
bolichen97 merged commit 6b1b65d into main Oct 27, 2026
112 of 120 checks passed
@bolichen97
bolichen97 deleted the ci/net-strict-changed-tests branch October 27, 2026 07:17
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants