Repository navigation
test(gates): refuse a test that reaches the repository through the CWD - #13885
Conversation
A structural test that asserts something about the repository's own source has to name a file. Naming it as a bare relative path binds the test to whatever directory the process started in, and that only holds while pytest is launched from the root: from a subdirectory, an editor's runner, or any wrapper that changes directory first, the literal resolves to nothing and the read raises FileNotFoundError before a single assertion runs. scripts/check_cwd_relative_repo_reads.py enumerates the condition from source rather than pinning one site: every filesystem access over the collected test trees whose path expression bottoms out at a bare relative literal naming one of the repository's own top-level entries, read from the repository itself so the rule cannot drift from the layout. A relative literal that is only compared, or joined onto a resolved root, is not a violation -- the allowlists in this suite are exactly that shape. A file that changes directory itself is skipped, since a relative path there names the tree the test built, and one access can carry a `# cwd-ok: <reason>`. The remedy the gate's message names is the module under test asked where it lives, not a better relative path: a module that moves or is renamed then fails at import, which no stale literal survives. Measured: 3207 collected test files, zero violations, so the gate lands clean and carries no backlog. Planting the one known instance -- the file added by pull request #11794, byte-identical at blob 7d79d03 -- turns it red on that line alone and removes it again on restore. Refs #13809
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the 54 chdir-skipped files hold no repo-relative reads today — the whole-file skip means the "zero violations" measurement never looked inside them. [DESIGN-REVIEWED] a69b0cd |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The gate's headline harm is disproved by its own measurement: CI runs from the root, the one instance sits on an unmerged PR, zero on base. Not justified as shipped
What this change shipsInventory (9 items) — 7 justifiedIntent: ADDITION — a new CI gate refusing tests that read repository files via bare CWD-relative paths, so structural tests survive non-root pytest invocations.
[FIRST-PRINCIPLES-REVIEWED] a69b0cd |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNothing blocks; three advisory findings on the new gate's exemption matcher, its printed remedy, and its unregistered fast-gate row. FINDING — scripts/check_cwd_relative_repo_reads.py:267 — FINDING — scripts/check_cwd_relative_repo_reads.py:436 — the remedy the gate prints (and FINDING — .github/workflows/fast-gate.yml:377 — the new [OPUS-REVIEWED] a69b0cd Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
The gate landed in fast-gate.yml without the three registrations this repo requires of a blocking gate, so three of its own guards caught it. Pin the git ls-tree read to UTF-8. The scanner reads path names out of the repository, which are UTF-8, so locale decoding is not what is wanted here. Add the gate to the prepare-pr floor in its CI-exact form, both invocations, so an installed copy runs the same two commands CI does. Mirror the step into the main ratchet audit's ratchet-gates job. The gate takes no base ref and no diff: it derives its scan roots from setup.cfg testpaths and its top-level entry set from the repository itself, so the whole tree is the only question it answers and main is where the answer belongs. Recording it as PR-only would be a false claim, and deferring it would leave a zero-backlog gate unmeasured on main until its drift surfaced on an unrelated pull request. Refs #13809
Round on
|
|
Board is green on |
What is the problem?
A structural test that asserts something about the repository's own source text has
to name a file. Naming it as a bare relative path binds the test to whatever
directory the process happens to start in:
That binding holds only while pytest is launched from the repository root. Launched
from anywhere else -- a subdirectory, an editor's runner, a wrapper that changes
directory first -- the same literal resolves to a path that does not exist and the
read raises
FileNotFoundErrorbefore a single assertion runs.Two things were measured before writing anything, because the worse failure for a
structural test is not a red but a silent pass against a file that is not there:
working directory the read raises rather than skipping: resolved to
<...>/test/src/kiro_crew/dashboard/ws.py,read_textgivesFileNotFoundError. And if the assignment it pins were renamed or moved, the ASTenumeration returns
[], soassert assigned == ["owner_request"]fails. Neitherdirection passes vacuously. The severity is therefore the one the report states,
not higher.
docs/system-specs/common/testing-conventions.mdpins the working directory under pytest to the repository root, the rootdir
conftest's
pytest_runtest_teardownrestores_SESSION_CWDbefore any fixtureunwinds, and CI invokes pytest from the root. An xdist worker therefore inherits
the root and the read succeeds. The live trigger is the invocation directory,
not the worker.
Why this issue matters to the user
A test that fails for a reason unrelated to the behaviour it pins costs a CI round
and, worse, trains readers to re-run past a red. The assertion itself is worth
keeping: it checks that a flag read three modules away is assigned from the
predicate that is supposed to produce it, which no behavioural test can see.
The class is also one line wide and 3207 test files large. Fixing one site leaves
the next one to be written.
How our fix solves it
The chain from symptom to root cause:
FileNotFoundErrorin a structural test -> the path was resolved against theprocess working directory -> nothing refuses a relative literal naming the
repository, so the next test written reintroduces it -> the rule is documented for
os.chdirleaks but was never written down or enforced for the read itself.So this pins the condition, enumerated from source, rather than a site:
scripts/check_cwd_relative_repo_reads.pyflags every filesystem access over thecollected test trees whose path expression bottoms out at a bare relative string
literal naming one of the repository's own top-level entries. The top-level set is
read from the repository (
git ls-tree, falling back to the filesystem), so therule cannot drift from the layout.
Both halves are necessary. A relative literal that is only compared, or joined
onto a resolved root, is not a defect: this suite keeps allowlists of relative
paths and checks them with
path.relative_to(root), and four such sites existtoday. Equally, a relative read of something that is not a repository entry --
Path("out.json")inside a test that changed directory into its owntmp_path--is the working directory being used on purpose.
The remedy the failure message names is not a better relative path, which would
only defer the next move. It is the module under test asked where it lives:
A module that moves or is renamed then fails at import, which no stale literal
survives, and the path cannot name a different file than the one the rest of the
test exercises. The weaker
Path(__file__).resolve().parents[1] / <relative>formstays accepted for a file no module owns -- a workflow, a script, a spec.
Exemptions are narrow and stated. A file containing any
chdircall is skippedwhole: it has taken the working directory into its own hands, and a relative path
there names the tree the test itself built. That is 54 of 2973 files, and it
removes the only false-positive class. A single access can carry
# cwd-ok: <reason>.Wired into
fast-gate.ymlbeside its siblings, self-test first, and registered inboth explicit gate inventories (
test_fast_gate_barrier.py,test_ci_fleet_routing_expression_parity.py) so a dropped job fails rather thandisappearing.
The rule is written into
docs/system-specs/common/testing-conventions.mdnext tothe existing working-directory rule, so the gate enforces documented policy.
What tests we did
test files, 0 violations. Whole-tree rather than diff-scoped for that reason --
there is nothing to charge to whoever pushes next.
that carries the known instance -- blob
7d79d03f713b8a23b1e6ca2dc59d047d954523c7,byte-identical to the one on pull request feat(notifications): route notifications to chat transports (bridge B1) #11794 at head
e9fb2ae-- takes thegate from
rc=0torc=1and names exactly one line:test/test_notification_bridge_wiring.py:1119: .read_text() on relative 'src/kiro_crew/dashboard/ws.py'.Control before the mutation: clean. Removing the file again: clean. So the red is
the plant, and the hashes prove the plant is the real thing.
flagged (bare read, module-qualified spelling, builtin
open, a literal bound to aname first, a join onto the literal, a non-reading verb, a relative write) and seven
that must stay clean (compared-only, repository-rooted, module-rooted, a
non-repository name, absolute, a chdir file, a pragma). A gate that only proves it
catches things can be weakened into catching everything.
test/test_cwd_relative_repo_reads_gate.py. The self-testlives in the same file as the rules it probes, so a commit could weaken both in one
edit. This file asserts the same judgments from outside, parametrised over every
entry in the access set so an entry matching nothing is caught as a hole, plus the
fail-closed paths and that the workflow really runs both invocations.
pytest -q -n0over the three owned files --test_cwd_relative_repo_reads_gate.py,test_fast_gate_barrier.py,test_ci_fleet_routing_expression_parity.py-- 119 passed in 5.3s.black,isort,flake8clean on the changed Python; the comment-history gate passesdiff-scoped against the base;
docs-lint.shpasses.Any other suggestions on the work
main, and this change deliberately does notclaim to have fixed it. Both the test and the production flag it pins are added
by pull request feat(notifications): route notifications to chat transports (bridge B1) #11794, which is open;
test/test_notification_bridge_wiring.pyand the
ws["_is_owner"]assignment are both absent frommain. HenceRefsrather than
Fixes.on
mainand this change carries the one-line fix on its rebase, which is exactlywhat the report asks for. If this lands first, the gate reddens feat(notifications): route notifications to chat transports (bridge B1) #11794 and it needs
one more commit, which would void the head-pinned dispositions already filed
against
e9fb2ae. Either order ends with the same source; only one of them spendsthat re-disposition.
Path(__file__).resolve().parents[1] / "src/..."readers tothe module-rooted form is a separate, mechanical change across many files. Left
alone here: they are already independent of the working directory, so they are not
this defect.
Refs #13809