Repository navigation
ci: warn on network reach in changed test files - #24515
Conversation
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed at head bfa37a8.
The approach is sound. Turning on KIROCREW_NET_STRICT=1 only for the test files a change adds or changes avoids a baseline, and it reuses the same base and file set as the flake-pattern ratchet. I checked these points against main:
- The two failure strings in
scripts/net_strict_changed.pymatch the floor on main:off-loopback network refused in a test:(rootconftest.pyraise site) andthis test reached past loopback under KIROCREW_NET_STRICT=1: ... . Stub the network(_determinism_floors_end). change_merge_base(env_name, *, cwd=...)andchange_files(base, *, cwd=...)inscripts/ratchet_scope.pyhave the signatures the script calls.- File names from the PR pass through
is_runnable_nameand go to pytest as an argument list after--. No shell sees them. - Workflow trust read of the
.github/workflows/fast-gate.ymlhunk: no secret or token use, nopermissions:change, no new trigger, no remote download. The new step runs PR test code throughci-shellin a job that already runs PR test code (test/test_flake_pattern_ratchet.pyand the PR's conftest), withpersist-credentials: false. The job cap goes from 10 to 15 minutes, and the step has its own 5 minute cap above the script's 240 second pytest limit.
Two small items, neither blocks:
- A pytest run that collected nothing reads as clean. See the inline comment on
scripts/net_strict_changed.pyline 260. - The job header comment in
.github/workflows/fast-gate.ymlline 606 still saysStatic scan only: a git diff plus the changed files.After this change the job also executes the changed test files. A reader who checks this job's trust boundary relies on that line. Suggest:Static scan plus one warn-only step that runs the changed test files with KIROCREW_NET_STRICT=1.
The PR changes .github/ and adds CI tooling, so a maintainer gives the final approval. The required checks were still running when I read it.
| return 0 | ||
| with tempfile.TemporaryDirectory() as scratch: | ||
| junit = Path(scratch) / "junit.xml" | ||
| finished, _code = run_strict(files, root=ROOT, junit=junit, timeout=args.timeout) |
There was a problem hiding this comment.
🟡 A pytest run that ran nothing is reported as "No test reached past loopback."
The exit code is discarded here (_code). When pytest stops before it writes a junit file (a usage error, exit 4, or an internal error, exit 3), parse_report returns ([], 0) and render prints No test reached past loopback. with exit 0. The step then looks like a clean check that checked nothing. This is the same outcome the ratchet step above turns into a red: "a gate that compared nothing is not a pass".
finished, _code = run_strict(files, root=ROOT, junit=junit, timeout=args.timeout)
findings, others = parse_report(junit)Suggestion: keep the code and treat "no junit file" or an exit code other than 0, 1 or 5 as "did not finish", for example finished = finished and junit.is_file() and code in (0, 1, 5). Then the existing "the result below is partial" line and exit 1 cover it. A test that runs main with a run_strict stub returning (True, 4) and no junit file pins it.
If the step is meant to stay silent on a broken run during the warn-only release, say so in the docstring and I will withdraw this.
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The step only warns, and its one surface without a consumer is two parameters that can be removed. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: a test file added or edited in a pull request that reaches past loopback (D10) gets named in the job summary before merge. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] a001fa4 Description read[DESCRIPTION-READ] 7f32a8e68d878dac73bbd218d8cce880aedd977ec875eaa692baa0f8b5ba4b03 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a warn-only CI step for one real, named miss, it reuses the ratchet's file-scoping, and it removes nothing. Notes
[DESIGN-REVIEWED] a001fa4 Description read[DESCRIPTION-READ] 3beb526ca3250d3b5675f6972da57668e03a798e29f4dd6b6fb254b069880729 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed at head ffff183, as a delta from bfa37a8.
Both items from my last review are fixed:
- A run that measured nothing no longer reads as clean.
parse_reportnow returns None for a missing, broken or entity-bearing junit file,renderthen saysNothing was measuredwith the exit code or the timeout, andmainexits 1. A pytest usage error or internal error writes no junit file, so it now lands on that path.test_a_run_that_measured_nothing_never_says_it_was_cleanand the last block oftest_main_writes_the_report_file_and_never_the_step_summarypin it. - The job header comment in
.github/workflows/fast-gate.ymlno longer says the job is a static scan only.
Other delta points I checked: change_merge_base(cwd=root) relies on the default env_name="FLAKE_RATCHET_BASE" in scripts/ratchet_scope.py at this head, so the shared override still applies. defusedxml is in install_requires in setup.cfg, which the job installs with -e ., so the import does not fall into the None path on CI. The probe fixtures now raise sys.audit events instead of real lookups.
One blocking item, in the new publish step. It agrees with the GPT lane finding on this head; I traced it myself:
- On a root CodeBuild runner,
run-as-runnerchowns$GITHUB_WORKSPACEto the unprivilegedrunneruser, so the strict step's test code can createsrc/sitecustomize.py. setup.cfgmaps the package tosrcand the job installs it editable, sosrc/is on the site path.Publish the network report to the job summarythen runs a barepython3 -as root, outsideci-shell. Python'ssiteimportssitecustomizeat startup, before the heredoc'sO_NOFOLLOWand regular-file checks run.
Details and a one-line fix are in the inline comment.
The PR changes .github/, so a maintainer gives the final approval after the fix.
| if: always() | ||
| continue-on-error: true | ||
| run: | | ||
| python3 - <<'PY' |
There was a problem hiding this comment.
🔴 The root publisher starts Python with site enabled, so test code can run as root here.
The step before this one runs the pull request's test files as the unprivileged user, and on a root CodeBuild runner that user owns the workspace. A changed test can write src/sitecustomize.py. The editable install puts src/ on the site path, and this step runs python3 - as root outside ci-shell, so site imports that module at startup. That happens before the O_NOFOLLOW and S_ISREG checks, so they do not help. The test code then runs with the runner identity this job set out to keep it away from, including the file-command directory in $RUNNER_TEMP.
run: |
python3 - <<'PY'Suggestion: start the publisher isolated and without site, so it imports only the standard library already loaded at startup:
python3 -I -S - <<'PY'-I drops the working directory and the PYTHON* variables from the path, and -S skips site, so no .pth file and no sitecustomize runs. The script only needs os and stat. A plain shell form such as [ -f "$p" ] && [ ! -L "$p" ] && head -c 16384 "$p" >> "$GITHUB_STEP_SUMMARY" would also work, but it has a small check-then-read gap that the fd form does not.
The rootdir network floor reports an off-loopback connect or name lookup, but CI never sets KIROCREW_NET_STRICT, so the report sits at the end of a shard. Add scripts/net_strict_changed.py: it runs only the test files the change adds or changes (same diff scope as ratchet_scope) with the strict switch on and renders a report. It is a step of the flake-pattern-ratchet job with continue-on-error, so nothing newly fails. The step runs as the unprivileged ci-shell user, which cannot write the job summary, so it writes a report file and a following step publishes it under the runner identity (python3 -I -S, no link followed, regular file only, 16 KiB cap). The prepare-pr floor test lists the script as a warn-only exemption. No baseline is touched.
ffff183 to
a001fa4
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed at head a001fa4, as a delta from ffff183.
The blocking item from my last review is fixed. Publish the network report to the job summary now starts python3 -I -S -, so the root step skips site (no .pth from the editable install, no sitecustomize) and drops the working directory and PYTHON* variables from the path. A sitecustomize.py that test code leaves in the workspace or in src/ no longer runs under the runner identity. The script only needs os and stat, which load without site.
The fix is pinned by two new tests in test/test_net_strict_changed.py: one asserts the step starts with python3 -I -S -, has no shell: override, and keeps continue-on-error and if: always(); the other runs the real step body with a planted sitecustomize.py, first proves a bare python3 - executes it (control), then proves the shipped form does not, and that a symlinked report publishes nothing.
Other root paths in the delta, checked at this head:
- The publish step is the only new step outside
ci-shell. It reads a fixed path in$RUNNER_TEMP(root-owned, sticky) withO_NOFOLLOWandO_NONBLOCK, checksS_ISREGon the open descriptor, and caps the read at 16 KiB, so a link, FIFO or swapped file cannot redirect it. python3resolves to the setup-python toolcache interpreter, whichrun-as-runnerdoes not hand to the unprivileged user, so test code cannot replace the binary or its stdlib.- The strict step itself runs under
ci-shell, andGITHUB_ENV,GITHUB_PATHandGITHUB_STEP_SUMMARYstay unwritable to it, so it cannot change the environment or PATH of the publish step. - No new secret use,
permissions:change, trigger change, or download-and-execute.
I have no further findings. The PR changes .github/, so a maintainer gives the final approval.
CrysisDeu
left a comment
There was a problem hiding this comment.
Review at a001fa45: Approve
All three of @bolichen97's points are fixed and pinned: "ran nothing reads as clean" (net_strict_changed.py:138-158,230-236,281), the stale "Static scan only" header (fast-gate.yml:606), and the root publisher now runs python3 -I -S - (fast-gate.yml:696, two tests incl. the planted-sitecustomize control). Workflow security holds: pull_request trigger, contents: read, persist-credentials: false, file names through is_runnable_name and passed as argv after --, no shell interpolation. Both steps are continue-on-error, so warn-only as stated. CI green, MERGEABLE.
Non-blocking:
- MINOR —
net_strict_changed.py:278-281: a collection error (pytest exit 2, one<error>testcase) still renders "No test reached past loopback" and exits 0 — the same "clean check that checked nothing" shape as round 1. Treatcode not in (0, 1, 5)as not-measured, or pass--continue-on-collection-errors; add a test with a(True, 2)stub. - NIT —
run_strict(extra_args=, timeout=)have no production caller (:177-184). - NIT — the network step has no
if: always()(fast-gate.yml:676), so a red ratchet skips the D10 warning; say so in the comment or add it.
Note: this approval does not clear @bolichen97's CHANGES_REQUESTED at ffff183f; they said "no further findings" at this head, so they need to approve or a maintainer dismisses.
Posted by code-reviewer (Kiro Crew member) on behalf of @CrysisDeu. Full report on file.
Problem / Motivation
Goal: A test file that a pull request adds or changes is checked against the loopback-only rule (D10) before merge, and a file that breaks it is named in the job summary.
The rootdir network floor already sees a test that looks up a non-local name or connects past loopback. In CI it only reports, in a warnings summary at the end of a 20-40 minute shard, and nobody reads it.
KIROCREW_NET_STRICT=1would make the call raise, but no workflow sets it. In the 15-loop main-red cluster, the floor namedtest_governance_policy_viewer.py(name lookup 'governance.example.invalid') in the log and nobody saw it.Why it matters
A lookup that is harmless on Monday can turn fatal after an unrelated merge, and then every open pull request pays a CI loop. Naming the test while its author is still on the PR is the cheapest point to fix it.
Not a goal
continue-on-error: true.real_networkmarker or rule D10 itself.What changed (motivation → approach → change)
Strict mode already exists and works (
conftest.pyraisesOffLoopbackNetworkRefusedat the call and fails the test at teardown; areal_networkmark and a UDP route probe are let through). What is missing is a place that turns it on without a baseline.New
scripts/net_strict_changed.pytakes the changed test files fromscripts/ratchet_scope.py(change_merge_base,change_files), the same base and file set the flake-pattern ratchet reads, and uses that ratchet's ownFLAKE_RATCHET_BASEoverride. It has no--timeoutflag and noNET_STRICT_BASEoverride of its own: nothing called either. It runsKIROCREW_NET_STRICT=1 pyteston those files, reads the junit file withdefusedxml, and renders a report. It is a step of the existingflake-pattern-ratchetjob (Linux only, one leg), withcontinue-on-error: true. The job cap goes 10 to 15 minutes so the extra steps cannot turn it red.The step runs as the unprivileged
ci-shelluser, which cannot write$GITHUB_STEP_SUMMARYon CodeBuild (run-as-runnerprotects the file-command directory). So the script writes the report to$RUNNER_TEMP/net-strict-report.md, and a new step,Publish the network report to the job summary, appends it to the summary under the runner identity. That step runs no workspace file, starts Python with-I -S(so asitecustomize.pya test left in the checkout cannot run as that identity), opens the report without following a link, takes a regular file only, and reads at most 16 KiB.Edge cases: no test file changed, an unreadable base, and a deleted file skip with a written reason and exit 0. A report that is missing, unreadable or declares an XML entity is not called clean: the report says
Nothing was measuredand the step exits 1 (still warn-only). A file name comes from the pull request, so it must start under a test root, may not contain::or a control character, and goes to pytest as an argument list after--; no shell sees it. Failures that are not a network refusal are counted, not judged. The job's header comment no longer says it only scans statically, since the last two steps now run the PR's test code.Backwards compatibility
Compatible: nothing that passes today fails. The new step cannot change the job or check result, and no existing script, marker or setting is renamed or tightened.
Removes nothing: only a step, a script and two doc sentences are added.
Tests
test/test_net_strict_changed.py(31 tests):@pytest.mark.real_networkare not reported (a real pytest child with the real floor; the probes raisesys.auditlookup events, so no test makes a real lookup);Nothing was measuredand never "clean", andmain()writes the report file, never$GITHUB_STEP_SUMMARY, and exits 1 on a finding;python3 -I -S -, ignores a plantedsitecustomize.py(a control run without the flags runs it), and publishes nothing for a symlink.test/test_prepare_pr_profiles.pylists the script as a warn-only exemption from the prepare-pr floor (one comment and one line).Mutation: with the child's
KIROCREW_NET_STRICTset to0the changed-file test fails.Proof: not provable -- new tooling with new tests, no production hunk to revert.
Manual verification
Ran
python scripts/net_strict_changed.pyon this branch withNET_STRICT_REPORTset: it picked the new test file, ran it strict, wroteNo test reached past loopback.to the report file, exit 0. Ran the publish step's inline Python locally against an absent file (nothing written), a normal file (copied), a symlink and a FIFO (nothing), and a 40 KB file (cut to 16 KiB). Not run on a GitHub or CodeBuild runner:ci-shell,python3in the publish step and the$RUNNER_TEMPand summary permissions are read from the neighbouring steps and the boundary check, not observed.Related Issues
Design: main-red study, section 4 and plan row 3.
Sibling sweep
Swept: git grep -n KIROCREW_NET_STRICT -- .github scripts
Siblings: none -- no other workflow or script sets the strict switch; only the floor reads it.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)