Skip to content

ci: run the tests that read a changed test file by name on a leaf diff - #23707

Merged
iamwhatever merged 2 commits into
kirodotdev:mainfrom
leonlaiyc:claude/leaf-named-test-file-gates
Nov 2, 2026
Merged

iamwhatever merged 2 commits into
kirodotdev:mainfrom
leonlaiyc:claude/leaf-named-test-file-gates

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a test-only PR's leaf run includes every test that reads a changed test file by its file name, so it can't pass a change the full suite rejects.

scripts/leaf_test_scope.py keeps a modified test file a leaf unless another file imports it or quotes its module name, and it adds as "corpus gates" the tests that scan a directory reaching test/ or join / "test".

test/test_history_locking_remediation.py reads one sibling file by name and passes neither check:

source = (Path(__file__).resolve().parent / "test_member_memory_runtime.py").read_text(encoding="utf-8")

It then asserts that test_http_resume_cannot_authorize_private_transcript is still in that file. It scans no directory and has no / "test" join, so it is not a gate. mentions_of misses it too: its quoted-word index needs the closing quote right after the stem, and .py sits in between.

Reproduced on main at d5b8c2099c: with that function renamed in test_member_memory_runtime.py, classify({"test/test_member_memory_runtime.py": "M"}) accepts a leaf plan (plus 702 corpus gate(s), the reader not among them), and pytest test/test_history_locking_remediation.py fails (1 failed, 43 passed).

Why it matters

The leaf run reports green, and the next full run on main goes red for whoever pushes next. The leaf reduction exists on the promise that it never skips a check the diff can break.

Not a goal

  • A test file read through a COMPUTED name (f"{stem}.py"). A quoted bare stem is already caught by mentions_of; a name assembled from pieces is not, and closing that needs an import graph, which this script deliberately does not build.
  • Non-test readers. The scripts/ and src/ literals naming a test/ file on main are skip lists (check_brand_name, check_harness_parity, docs_lint), a hand-edited-files map (regen_shared), or self-test fixtures (leaf_test_scope, run_scoped_tests), not reads whose result a test asserts.
  • Leaf-mode runtime. This adds 10 run-once gates; running the gate set once across the matrix is ci: run leaf corpus gates once across the matrix, not per job #23461.

What changed (motivation → approach → change)

Symptom: a leaf run skips a test that reads the changed file by name. Root cause: the selector knows three ways another test depends on a changed leaf (it imports it, quotes its module name, or scans the test/ tree), and a test that reads one named file does none of them.

  • scripts/leaf_test_scope.py: new readers_of(). For the changed modules it returns every top-level test that names one in its own source ("test_x.py", "test/test_x.py", or a node id "test/test_x.py::test_y"), or that names a .txt list under test/ which lists one. classify() adds those readers to that diff's run-once set. The tree index records each test module's quoted file names in the pass it already makes, so the lookup adds no tree scan; it reads only the .txt lists under test/.
  • The list case is the reader the Opus note on the first head pointed at: test_windows_gap_list parses every module windows-expected-failures.txt lists and fails when a listed test is renamed. That list names 120 tests in 51 modules that nothing imports or mentions. Reproduced on d5b8c2099c: after renaming the listed TestOffloadRunsOnSubprocessPool::test_kill_process_runs_scan_and_sweep_off_loop in test_acp_offload.py, the leaf plan carries 702 gates and no test_windows_gap_list, and test_windows_gap_list then fails (test_every_entry_names_a_test_that_still_exists, 1 failed, 4 passed).
  • Per diff, not as corpus gates. The first head of this PR made the 10 readers it found static gates (702 -> 712), so every leaf run paid for them on all 8 Backend Tests jobs, where the gate group already takes about 59 of the 60 minutes (CI: leaf-mode Backend Tests jobs run every corpus gate and hit the 60-minute limit on hosted runners #23450). Run once on this host with 4 workers, those 10 files are 405 tests and 4 min 34 s. 200 s of that is test_local_gate::test_dry_run_against_the_real_repo_never_plans_a_full_suite, and test_local_gate names test_gatewayd_diag.py only as a sample path. 5 of the 10 use the name only as data like that. This head leaves corpus_gates() and its 702 gates as they are on main, and a reader runs only when the diff modifies a file it names. On main, 67 of 5,320 test modules have a reader that is not already a gate: an edit to one of them adds 1 to 3 files to its run (62 add one), and an edit to any other module adds none.
  • Not escalated, and not a wider mention index. A reader's outcome depends on the files it read and on nothing else, so running it beside the edit covers it; the full suite would only add files that do not read the edit. Widening mentions_of to .py literals would escalate every module named by an existing gate such as test_source_pin_budget (about 50 names) to the full suite.
  • .txt lists only. test/README.md names a module in passing, and its bare name is quoted by 65 tests that read some other README.
  • The runtime bullet under Not a goal describes the first head (10 run-once gates). This head adds no gate.
  • Overlap with ci: count tests that read test/ through a helper as leaf corpus gates #23677 and ci: run leaf corpus gates once across the matrix, not per job #23461: this head no longer touches corpus_gates(), which ci: count tests that read test/ through a helper as leaf corpus gates #23677 changes, and neither PR touches _TreeIndex, readers_of or classify(). The clean_leaf fixture change and the new tests sit on lines neither PR edits, and a trial merge of this head with each of theirs (2b6f1d79cd, ac8efbe057) is clean. Readers are part of the gate list CI receives, so ci: run leaf corpus gates once across the matrix, not per job #23461 shards them like any gate.
  • readers_of()'s docstring states the limit the Design note on 2a06ff9af4 asked for: only a file name spelled out in a source or a list is found, and a test that builds the name at run time (f"{stem}.py", path.with_name(var)) is not.

Backwards compatibility

Compatible: only the set of tests a leaf run executes grows, and only for a diff that modifies a file some test reads by name. Exit codes, output shape and the full-suite path are unchanged.

Removes nothing: corpus_gates() still derives the same 702 gates, and readers are only added.

Tests

test/test_leaf_test_scope.py:

  • test_an_edit_to_a_file_another_test_reads_by_name_runs_that_test: the reproduced case. Modifying test_member_memory_runtime.py alone is still a leaf diff, and test_history_locking_remediation runs with it. It also checks that the reader still names the file, does not scan the tree itself, and is not a static gate, so it stays a guard on this rule.
  • test_an_edit_to_a_listed_test_runs_the_audit_of_that_list: picks a leaf from windows-expected-failures.txt that no import, mention or name in the audit's source reaches, and checks that test_windows_gap_list runs with it.
  • test_a_reader_runs_only_for_a_diff_that_touches_a_file_it_names: on a synthetic test/ tree, a bare name, a test/-relative name, a node id and a .txt list each find their reader, and an edit to an unrelated leaf runs the corpus gate alone.
  • The clean_leaf fixture now also skips a file that has a reader, since test_accepted_run_always_includes_the_corpus_gates asserts the exact gate set.

Proof: PROVEN test/test_leaf_test_scope.py::test_an_edit_to_a_file_another_test_reads_by_name_runs_that_test, test/test_leaf_test_scope.py::test_an_edit_to_a_listed_test_runs_the_audit_of_that_list, test/test_leaf_test_scope.py::test_a_reader_runs_only_for_a_diff_that_touches_a_file_it_names (head 2a06ff9)

Negative control: prove.py puts main's scripts/leaf_test_scope.py back under the new tests, and all three fail on an assertion. With the fix, the whole file passes: 49 passed.

scripts/run_scoped_tests.py --surface backend (related set, 36 files): 2980 passed, 2 failed. Both failures are the ones the first head recorded on clean main on this host, and neither reads leaf_test_scope: test_runtime_bindings::test_the_constructor_keeps_its_type_hints and test_sandbox_launcher_program_parts::test_every_function_of_the_program_is_the_one_the_module_binds.

Lint: black, isort, flake8, check_subprocess_encoding, check_comment_history, check_brand_name, check_file_line_cap and test_flake_pattern_ratchet (60 passed) pass on the diff.

Manual verification

Ran the selector on d5b8c2099c against both broken dependencies, then ran the reader. (1) test_http_resume_cannot_authorize_private_transcript renamed in test_member_memory_runtime.py: this head plans 703 gates with test_history_locking_remediation, which then fails (1 failed, 43 passed). Main's selector leaves it out, as the negative control above shows. (2) The listed test_acp_offload test renamed: main plans 702 gates without test_windows_gap_list, which fails; this head plans 703 with it.

Related Issues

Closes #23706

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

@leonlaiyc
leonlaiyc requested review from a team and KiroCrewCaptain as code owners October 24, 2026 08:58
@leonlaiyc
leonlaiyc requested a review from CrysisDeu October 24, 2026 08:58
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 24, 2026
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 03fb3b92d3978506d161ebb853fa1bfb44c11dc6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 03fb3b92d3978506d161ebb853fa1bfb44c11dc6 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (5 items) — 5 justified

Intent: stop a test-only PR's leaf run from passing an edit that breaks another test reading that file by name. This is a FIX.

  1. A leaf run now also runs every test that names the changed test file in a quoted string (bare, test/-relative or a node id) — justified
  2. A leaf run now also runs a test that names a .txt list under test/ that lists the changed module — justified
  3. The clean_leaf fixture now skips a test file that something reads by name — justified
  4. Three new tests pin the reader lookup: the real reader, the list audit and a synthetic tree — justified
  5. The module docstring gains an entry for tests that read a file by name — justified

[FIRST-PRINCIPLES-REVIEWED] 03fb3b9

Description read

[DESCRIPTION-READ] 7868c3845687c489ed0ac6806d491e6213a54c62930ad8a125f999fbe607b5be

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 03fb3b92d3978506d161ebb853fa1bfb44c11dc6 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only adds tests to a leaf run, scoped to each diff, and takes nothing away.

Notes

  • A reader's quoted name is matched by its basename only, so "other_dir/test_x.py" or a fixture string will also match. That costs at most one extra file in the run of the diff that edits test_x.py, and the docstring already says so.

[DESIGN-REVIEWED] 03fb3b9

Description read

[DESCRIPTION-READ] 0ed3ea1ee53a5567a690f8eb30c11e928196fe13f91932799abf7ebed6e26a6b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 03fb3b92d3978506d161ebb853fa1bfb44c11dc6 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 03fb3b9

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 03fb3b92d3978506d161ebb853fa1bfb44c11dc6 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 03fb3b9

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 24, 2026
@leonlaiyc
leonlaiyc force-pushed the claude/leaf-named-test-file-gates branch from 610772a to 2a06ff9 Compare October 24, 2026 14:15
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 24, 2026
@leonlaiyc leonlaiyc changed the title ci: count tests that read a sibling test file by name as leaf gates ci: run the tests that read a changed test file by name on a leaf diff Oct 24, 2026
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Revised: a reader now runs only for a diff that touches a file it reads (head 2a06ff9af4f8)

The First Principles and Design notes on 610772ad2f said the 10 readers this PR made static corpus gates ran on every leaf diff, though each depends only on the files it names. Measured before changing it:

  • Those 10 files, run once on this Windows host with 4 workers: 405 tests, 4 min 34 s. 200 s of that is one test, test_local_gate::test_dry_run_against_the_real_repo_never_plans_a_full_suite, and test_local_gate names test_gatewayd_diag.py only as a sample path. 5 of the 10 use the name only as data: an exclusion check, a sample argv, a ledger row, a classification case, a workflow string.
  • CI runs the whole gate group on each of the 8 Backend Tests jobs, where it already takes about 59 of the 60 minutes (CI: leaf-mode Backend Tests jobs run every corpus gate and hit the 60-minute limit on hosted runners #23450).

This head leaves corpus_gates() as it is on main (702 gates). classify() instead looks the changed modules up with readers_of() and runs their readers once beside the edit. On main, 67 of 5,320 test modules have a reader that is not already a gate: an edit to one of them adds 1 to 3 files to its run (62 add one), and an edit to any other module adds none.

The same lookup takes the node-id spelling from the Opus note, and the list behind it. test_windows_gap_list parses every module windows-expected-failures.txt lists, and nothing in its source names them. Reproduced on d5b8c2099c: after renaming the listed TestOffloadRunsOnSubprocessPool::test_kill_process_runs_scan_and_sweep_off_loop in test_acp_offload.py, the leaf plan carried 702 gates and no test_windows_gap_list, and test_windows_gap_list::test_every_entry_names_a_test_that_still_exists then failed. With this head, that plan carries 703, the audit among them.

The goal sections are left as written. The runtime bullet under Not a goal describes the first head; this head adds no gate.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 24, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 2a06ff9.

The change looks sound: readers_of() only adds tests to a leaf run, per diff, and leaves the 702 corpus gates and the full-suite path unchanged (leaf_test_scope.py#L383-L406). All checks are green and the GPT and Opus lanes have no findings.

One current-head AI item has no reply. The Design Review note asks that readers_of's docstring say a test which builds the file name (f"{stem}.py", with_name(var)) is still not found, so nobody assumes the coverage is complete. The PR body scopes this out under Not a goal, but the code does not say so, and the reply on this head addresses the earlier gate-cost notes only. To clear: add that one sentence to the docstring, or reply to the note explaining why it is not needed.

— crew-pr-reviewer (r124)

@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Re the HOLD at 2a06ff9af4 (r124): e707389894 adds the sentence the Design note asked for to readers_of()'s docstring:

Only a name spelled out in a source or a list is found. A test that builds the name at run time (f"{stem}.py", path.with_name(var)) is not, so the readers returned are not every test that can read the module.

Docstring only, no code change. test/test_leaf_test_scope.py on this head: 49 passed. The description has a matching bullet under What changed.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention labels Oct 25, 2026
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Re the Design note on e707389894 (readers counted under "corpus gate(s)" in the plan summary): kept as one count on purpose.

  • The count is every file that runs once beside the edit. classify() returns readers in the same gates list that --targets hands CI, and ci.yml runs that list as one group, "leaf corpus gates (N file(s), once)", sharded with the rest by ci: run leaf corpus gates once across the matrix, not per job #23461. Labelling readers separately only here would make this line and the CI group name the same files differently. Labelling them in CI too means editing ci.yml, which this PR does not touch.
  • The overcount is small and only on diffs that have a reader. On main, 67 of 5,320 test modules have one, and an edit to one of them adds 1 to 3 files to the 702 static gates. Every other diff prints exactly the static set.

If a maintainer prefers the split, it belongs with a ci.yml group label, and I can do both in a follow-up.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 25, 2026
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

CI on e707389894 is red from one known flake, since fixed on main; the diff (leaf-scope CI tooling) has no path to it.

These are the only two failing checks on this head. Could a maintainer re-run the failed jobs of run 40201615335? I am not pushing to reroll it, so the reviewed head stays as it is.

@iamwhatever
iamwhatever force-pushed the claude/leaf-named-test-file-gates branch from e707389 to 1cf2a5e Compare November 1, 2026 13:25
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 1, 2026
leonlaiyc and others added 2 commits November 2, 2026 02:53
A leaf run kept a modified test file a leaf, but a test that reads that
file by name is neither an importer, a mention nor a corpus gate:
test_history_locking_remediation parses test_member_memory_runtime.py,
and test_windows_gap_list parses every module that
windows-expected-failures.txt lists. A rename either one depends on
passed the leaf run and failed the next full run.

classify() now looks the changed modules up and runs each test that
names one, once, beside the edit: named in its own source (bare,
test/-relative, or as a node id) or in a list under test/ that the test
names. The tree index records each test module's quoted file names in
the pass it already makes.

Readers are added per diff, not as corpus gates. Each depends only on
the modules it names, and as gates the ten found on main would add 405
tests to every leaf job. The 702 corpus gates are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lookup matches only a file name spelled out in a test's source or
in a test/*.txt list, so a test that builds the name (f"{stem}.py",
path.with_name(var)) is not returned as a reader. Say so in the
docstring, so the result is not taken for every test that can read
the module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamwhatever
iamwhatever force-pushed the claude/leaf-named-test-file-gates branch from 1cf2a5e to 03fb3b9 Compare November 2, 2026 02:53
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Security Scope Review (fork) — 🔴 BLOCK (blocking)

Which legitimate operations does 03fb3b92d3978506d161ebb853fa1bfb44c11dc6 newly refuse? Model proposes, scripts/deny_diff.py decides — updated in place on each push. A script-confirmed regression blocks PR readiness.

no leg reported, and this was not the nothing-to-adjudicate short circuit

No [SCOPE-REVIEWED] marker for this head was produced, so the model's text is withheld: it cannot be shown to describe this revision. See the job logs.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 2, 2026
@iamwhatever iamwhatever self-assigned this Nov 2, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 03fb3b92d3978506d161ebb853fa1bfb44c11dc6.

readers_of() finds tests that name a changed test module by file name, or through a .txt list under test/, and classify() runs them once beside the edit (leaf_test_scope.py L544). A leaf run can only grow; the 702 corpus gates and the full-suite path are unchanged. The r124 hold is cleared: the docstring now states that computed names are not found (L396-L398). Checks are green on this head.

AI findings judged non-blocking

  • Design Review: a quoted name in a fixture string can count as a reader. The cost is one extra test file in a leaf run, and the docstring says so (L387-L414).
  • Security Scope Review (fork) BLOCK comment, "no leg reported": stale. The newer check run on this same head concluded success with no security-surface path touched (check run).

— crew-pr-reviewer (r141)

@iamwhatever
iamwhatever enabled auto-merge (squash) November 2, 2026 18:54
@iamwhatever
iamwhatever merged commit 2439bb7 into kirodotdev:main Nov 2, 2026
93 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

leaf_test_scope: a test that reads a sibling test file by name is not a corpus gate

2 participants