Skip to content

test(gates): refuse a test that reaches the repository through the CWD - #13885

Merged
iamwhatever merged 2 commits into
mainfrom
fix/structural-source-path-13809
Sep 25, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/structural-source-path-13809

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

What is the problem?

A structural test that asserts something about the repository's own source text has
to name a file. Naming it as a bare relative path binds the test to whatever
directory the process happens to start in:

source = pathlib.Path("src/kiro_crew/dashboard/ws.py").read_text(encoding="utf-8")

That binding holds only while pytest is launched from the repository root. Launched
from anywhere else -- a subdirectory, an editor's runner, a wrapper that changes
directory first -- the same literal resolves to a path that does not exist and the
read raises FileNotFoundError before a single assertion runs.

Two things were measured before writing anything, because the worse failure for a
structural test is not a red but a silent pass against a file that is not there:

  • The named instance is brittle, not hollow, in both directions. From a wrong
    working directory the read raises rather than skipping: resolved to
    <...>/test/src/kiro_crew/dashboard/ws.py, read_text gives
    FileNotFoundError. And if the assignment it pins were renamed or moved, the AST
    enumeration returns [], so assert assigned == ["owner_request"] fails. Neither
    direction passes vacuously. The severity is therefore the one the report states,
    not higher.
  • The stated trigger is not the one that fires. docs/system-specs/common/testing-conventions.md
    pins the working directory under pytest to the repository root, the rootdir
    conftest's pytest_runtest_teardown restores _SESSION_CWD before any fixture
    unwinds, and CI invokes pytest from the root. An xdist worker therefore inherits
    the root and the read succeeds. The live trigger is the invocation directory,
    not the worker.

Why this issue matters to the user

A test that fails for a reason unrelated to the behaviour it pins costs a CI round
and, worse, trains readers to re-run past a red. The assertion itself is worth
keeping: it checks that a flag read three modules away is assigned from the
predicate that is supposed to produce it, which no behavioural test can see.

The class is also one line wide and 3207 test files large. Fixing one site leaves
the next one to be written.

How our fix solves it

The chain from symptom to root cause:

FileNotFoundError in a structural test -> the path was resolved against the
process working directory -> nothing refuses a relative literal naming the
repository, so the next test written reintroduces it -> the rule is documented for
os.chdir leaks but was never written down or enforced for the read itself.

So this pins the condition, enumerated from source, rather than a site:

  • scripts/check_cwd_relative_repo_reads.py flags every filesystem access over the
    collected test trees whose path expression bottoms out at a bare relative string
    literal naming one of the repository's own top-level entries. The top-level set is
    read from the repository (git ls-tree, falling back to the filesystem), so the
    rule cannot drift from the layout.

  • Both halves are necessary. A relative literal that is only compared, or joined
    onto a resolved root, is not a defect: this suite keeps allowlists of relative
    paths and checks them with path.relative_to(root), and four such sites exist
    today. Equally, a relative read of something that is not a repository entry --
    Path("out.json") inside a test that changed directory into its own tmp_path --
    is the working directory being used on purpose.

  • The remedy the failure message names is not a better relative path, which would
    only defer the next move. It is the module under test asked where it lives:

    from kiro_crew.dashboard import ws
    source = Path(ws.__file__).read_text(encoding="utf-8")

    A module that moves or is renamed then fails at import, which no stale literal
    survives, and the path cannot name a different file than the one the rest of the
    test exercises. The weaker Path(__file__).resolve().parents[1] / <relative> form
    stays accepted for a file no module owns -- a workflow, a script, a spec.

  • Exemptions are narrow and stated. A file containing any chdir call is skipped
    whole: it has taken the working directory into its own hands, and a relative path
    there names the tree the test itself built. That is 54 of 2973 files, and it
    removes the only false-positive class. A single access can carry
    # cwd-ok: <reason>.

  • Wired into fast-gate.yml beside its siblings, self-test first, and registered in
    both explicit gate inventories (test_fast_gate_barrier.py,
    test_ci_fleet_routing_expression_parity.py) so a dropped job fails rather than
    disappearing.

  • The rule is written into docs/system-specs/common/testing-conventions.md next to
    the existing working-directory rule, so the gate enforces documented policy.

What tests we did

  • The gate lands clean and carries no backlog. Whole-tree scan: 3207 collected
    test files, 0 violations. Whole-tree rather than diff-scoped for that reason --
    there is nothing to charge to whoever pushes next.
  • Mutation, against the real defect rather than a look-alike. Planting the file
    that carries the known instance -- blob 7d79d03f713b8a23b1e6ca2dc59d047d954523c7,
    byte-identical to the one on pull request feat(notifications): route notifications to chat transports (bridge B1) #11794 at head e9fb2ae -- takes the
    gate from rc=0 to rc=1 and names exactly one line:
    test/test_notification_bridge_wiring.py:1119: .read_text() on relative 'src/kiro_crew/dashboard/ws.py'.
    Control before the mutation: clean. Removing the file again: clean. So the red is
    the plant, and the hashes prove the plant is the real thing.
  • The gate's own self-test: 14 probes, both directions. Seven shapes that must be
    flagged (bare read, module-qualified spelling, builtin open, a literal bound to a
    name first, a join onto the literal, a non-reading verb, a relative write) and seven
    that must stay clean (compared-only, repository-rooted, module-rooted, a
    non-repository name, absolute, a chdir file, a pragma). A gate that only proves it
    catches things can be weakened into catching everything.
  • External exerciser, test/test_cwd_relative_repo_reads_gate.py. The self-test
    lives in the same file as the rules it probes, so a commit could weaken both in one
    edit. This file asserts the same judgments from outside, parametrised over every
    entry in the access set so an entry matching nothing is caught as a hole, plus the
    fail-closed paths and that the workflow really runs both invocations.
  • Local run: pytest -q -n0 over the three owned files --
    test_cwd_relative_repo_reads_gate.py, test_fast_gate_barrier.py,
    test_ci_fleet_routing_expression_parity.py -- 119 passed in 5.3s. black,
    isort, flake8 clean on the changed Python; the comment-history gate passes
    diff-scoped against the base; docs-lint.sh passes.

Any other suggestions on the work

  • The one named instance is not on main, and this change deliberately does not
    claim to have fixed it.
    Both the test and the production flag it pins are added
    by pull request feat(notifications): route notifications to chat transports (bridge B1) #11794, which is open; test/test_notification_bridge_wiring.py
    and the ws["_is_owner"] assignment are both absent from main. Hence Refs
    rather than Fixes.
  • The cheaper merge order is feat(notifications): route notifications to chat transports (bridge B1) #11794 first. If it lands before this, the line is
    on main and this change carries the one-line fix on its rebase, which is exactly
    what the report asks for. If this lands first, the gate reddens feat(notifications): route notifications to chat transports (bridge B1) #11794 and it needs
    one more commit, which would void the head-pinned dispositions already filed
    against e9fb2ae. Either order ends with the same source; only one of them spends
    that re-disposition.
  • Converting the existing Path(__file__).resolve().parents[1] / "src/..." readers to
    the module-rooted form is a separate, mechanical change across many files. Left
    alone here: they are already independent of the working directory, so they are not
    this defect.

Refs #13809

A structural test that asserts something about the repository's own source has
to name a file. Naming it as a bare relative path binds the test to whatever
directory the process started in, and that only holds while pytest is launched
from the root: from a subdirectory, an editor's runner, or any wrapper that
changes directory first, the literal resolves to nothing and the read raises
FileNotFoundError before a single assertion runs.

scripts/check_cwd_relative_repo_reads.py enumerates the condition from source
rather than pinning one site: every filesystem access over the collected test
trees whose path expression bottoms out at a bare relative literal naming one of
the repository's own top-level entries, read from the repository itself so the
rule cannot drift from the layout. A relative literal that is only compared, or
joined onto a resolved root, is not a violation -- the allowlists in this suite
are exactly that shape. A file that changes directory itself is skipped, since a
relative path there names the tree the test built, and one access can carry a
`# cwd-ok: <reason>`.

The remedy the gate's message names is the module under test asked where it
lives, not a better relative path: a module that moves or is renamed then fails
at import, which no stale literal survives.

Measured: 3207 collected test files, zero violations, so the gate lands clean
and carries no backlog. Planting the one known instance -- the file added by
pull request #11794, byte-identical at blob 7d79d03 -- turns it red on that
line alone and removes it again on restore.

Refs #13809
@chenmingwei23
chenmingwei23 requested review from a team and CrysisDeu September 25, 2026 17:16
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of a69b0cd6d0b3e2fe5fade10fd4a02536ba86ca4a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the 54 chdir-skipped files hold no repo-relative reads today — the whole-file skip means the "zero violations" measurement never looked inside them.

[DESIGN-REVIEWED] a69b0cd

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of a69b0cd6d0b3e2fe5fade10fd4a02536ba86ca4a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a69b0cd

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a69b0cd6d0b3e2fe5fade10fd4a02536ba86ca4a: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of a69b0cd6d0b3e2fe5fade10fd4a02536ba86ca4a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The gate's headline harm is disproved by its own measurement: CI runs from the root, the one instance sits on an unmerged PR, zero on base.

Not justified as shipped

What this change ships

Inventory (9 items) — 7 justified

Intent: ADDITION — a new CI gate refusing tests that read repository files via bare CWD-relative paths, so structural tests survive non-root pytest invocations.

  1. PRs now fail a fast-gate job on a bare relative repo read in a test — oversized: named harm disproved by the description's own measurement; zero instances on base
  2. The same scan runs in the main-ratchet audit on main — justified
  3. Developers get a # cwd-ok: <reason> escape hatch per access — justified
  4. Files that call chdir are exempt whole — justified
  5. testing-conventions.md documents the rooted-path rule beside the existing CWD rule — justified
  6. The gate is registered in both CI job inventories — justified
  7. prepare-pr's local gate profile now runs the check — justified
  8. External exerciser test pins the gate's classifications from outside its file — justified
  9. RFC solo-spawn-gate status flipped superseded→in-progress (+ README row) — rides along, undeclared: identical hunk already on main via docs(rfc): move solo-spawn gate RFC to in-progress while its removal is in flight #13705

[FIRST-PRINCIPLES-REVIEWED] a69b0cd

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed a69b0cd6d0b3e2fe5fade10fd4a02536ba86ca4a — this comment is updated in place on each push.

Review details

Nothing blocks; three advisory findings on the new gate's exemption matcher, its printed remedy, and its unregistered fast-gate row.

FINDING — scripts/check_cwd_relative_repo_reads.py:267 — if PRAGMA in line: is an unanchored substring test over raw source, so # cwd-ok: inside a string literal exempts lines nobody annotated (it already exempts lines 156–159 and 167–168 of the test/test_cwd_relative_repo_reads_gate.py this PR adds) and a bare # cwd-ok: with no reason exempts despite the docstring's "The reason is required" → Fix: mirror check_testpaths_coverage.has_marker — require line.lstrip().startswith("#") or a tokenize.COMMENT token, and a non-empty reason after the colon; add both directions to _PROBES.

FINDING — scripts/check_cwd_relative_repo_reads.py:436 — the remedy the gate prints (and docs/system-specs/common/testing-conventions.md:676) prescribes Path(__file__).resolve().parents[1] / <relative path> unconditionally, but parents[1] is the repo root only for a module directly under test/; followed from one of the 78 nested test modules (test/e2e/**, src/kiro_crew/apps/builtins/**/test_*.py) it resolves to test/ or builtins/ and the read raises FileNotFoundError — the exact failure the gate exists to prevent, and a shape the gate itself reports clean → Fix: state the root as Path(__file__).resolve().parents[N] with N matching the file's depth (or direct readers to an existing repo-root helper) in both the message and the convention bullet. (origin: validation)

FINDING — .github/workflows/fast-gate.yml:377 — the new cwd-relative-repo-reads job is the only one of fast-gate.yml's 14 jobs with no row in docs/ci/ci-and-reviews.md's per-job | Job | What it enforces | table, so the enumeration contributors read to learn what blocks their PR is now incomplete → Fix: add the row to docs/ci/ci-and-reviews.md (an untouched file, hence advisory rather than blocking), alongside testpaths-coverage.

[OPUS-REVIEWED] a69b0cd

Verdict parsed from the review's SHA-scoped output markers for commit a69b0cd6d0b3e2fe5fade10fd4a02536ba86ca4a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a69b0cd6d0b3e2fe5fade10fd4a02536ba86ca4a: <one-sentence reason>

The gate landed in fast-gate.yml without the three registrations this repo
requires of a blocking gate, so three of its own guards caught it.

Pin the git ls-tree read to UTF-8. The scanner reads path names out of the
repository, which are UTF-8, so locale decoding is not what is wanted here.

Add the gate to the prepare-pr floor in its CI-exact form, both invocations,
so an installed copy runs the same two commands CI does.

Mirror the step into the main ratchet audit's ratchet-gates job. The gate takes
no base ref and no diff: it derives its scan roots from setup.cfg testpaths and
its top-level entry set from the repository itself, so the whole tree is the
only question it answers and main is where the answer belongs. Recording it as
PR-only would be a false claim, and deferring it would leave a zero-backlog gate
unmeasured on main until its drift surfaced on an unrelated pull request.

Refs #13809
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Round on a69b0cd6d0 -- gate registration, and dispositions for the two First Principles items

The gate was wired into fast-gate.yml without the three registrations this repo requires of a blocking gate, and three of its own guards caught that. All three are now registered in one commit.

UTF-8 on the scanner's own subprocess. repo_top_level() reads git ls-tree --name-only HEAD in text mode; it now passes encoding="utf-8". Path names out of a repository are UTF-8, so locale decoding is not what is wanted here and the # subprocess-encoding: locale escape would have been a false claim.

prepare-pr floor, CI-exact. Both invocations, the same two commands the workflow runs:

"python3 scripts/check_cwd_relative_repo_reads.py --test",
"python3 scripts/check_cwd_relative_repo_reads.py",

Main ratchet audit: MIRRORED, not recorded as an exemption. The gate takes no base ref and no diff -- its only flag is --test, its root is Path(__file__).resolve().parents[1], its scan roots come from setup.cfg testpaths and its top-level entry set from the repository itself. The whole tree is therefore the only question it can answer, so _PR_ONLY_BY_CONSTRUCTION would have been untrue and _DEFERRED_WHOLE_TREE_GATES would have left a zero-backlog gate unmeasured on main until its drift surfaced on an unrelated pull request. The step is mirrored into ratchet-gates and the job's own failure-report line now names the added gate family.

The four lanes that were red are green on this head: Backend Lint & Type Check (3.12), Backend Tests (3.12, 1), Backend Tests (Windows) (1), Backend Tests (3.12, 5). The last of those was never this change's: its annotation named test/test_source_providers.py::test_local_token_uses_local_owner_subject_without_configured_owner, which passes locally on this branch and is unreachable from the three files this round touched. It re-measured green on a fresh round.

First Principles item 9 -- not present on this branch

The RFC superseded->in-progress flip and its README row are not in this diff. Measured against the merge base, the branch changes eight files:

.github/workflows/fast-gate.yml
.github/workflows/main-ratchet-audit.yml
docs/system-specs/common/testing-conventions.md
scripts/check_cwd_relative_repo_reads.py
src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/profiles/kirocrew.json
test/test_ci_fleet_routing_expression_parity.py
test/test_cwd_relative_repo_reads_gate.py
test/test_fast_gate_barrier.py

No RFC file and no README among them, so there is no hunk for a rebase to drop. The branch is eight commits behind main with zero overlap against those eight files, which is the shape that produces this reading when a head is diffed against a main that has since advanced.

First Principles item 1 -- accepted as a real objection, deferred to the owner

The reviewer's reading is correct on the facts and I am not disputing them: the whole-tree backlog is zero, the one named instance lives on open #11794 rather than on base, and # cwd-ok: has no uses. Its subtraction -- hold the blocking fast-gate and prepare-pr wiring until an instance exists on base -- is a scope decision about whether this gate blocks now or after #11794, and it runs opposite to the registration this round was asked for. Items 2, 6 and 7 of the same inventory, which are exactly what this round changed, are marked justified. Flagged for the owner rather than acted on unilaterally; nothing in this round depends on which way it goes.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Board is green on a69b0cd6d0: 92 check-runs, 85 success and 7 skipped, zero failures and nothing in flight. PR Readiness published success at 18:35Z -- its earlier pending came from a publisher run that read the board at 18:05Z while the backend shards were still going, and the readiness sweep republished it without a new run on this head. Both AWS CodeBuild us-east-1 contexts are success, and no [BLOCK-MERGE] marker is anchored on this head.

@iamwhatever
iamwhatever enabled auto-merge (squash) September 25, 2026 23:11
@iamwhatever
iamwhatever merged commit 9dbce55 into main Sep 25, 2026
95 checks passed
@iamwhatever
iamwhatever deleted the fix/structural-source-path-13809 branch September 25, 2026 23:11
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants