Repository navigation
ci: run the tests that read a changed test file by name on a leaf diff - #23707
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (5 items) — 5 justifiedIntent: stop a test-only PR's leaf run from passing an edit that breaks another test reading that file by name. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 03fb3b9 Description read[DESCRIPTION-READ] 7868c3845687c489ed0ac6806d491e6213a54c62930ad8a125f999fbe607b5be That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change only adds tests to a leaf run, scoped to each diff, and takes nothing away. Notes
[DESIGN-REVIEWED] 03fb3b9 Description read[DESCRIPTION-READ] 0ed3ea1ee53a5567a690f8eb30c11e928196fe13f91932799abf7ebed6e26a6b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
610772a to
2a06ff9
Compare
|
Revised: a reader now runs only for a diff that touches a file it reads (head The First Principles and Design notes on
This head leaves The same lookup takes the node-id spelling from the Opus note, and the list behind it. The goal sections are left as written. The runtime bullet under Not a goal describes the first head; this head adds no gate. |
|
HOLD at 2a06ff9. The change looks sound: One current-head AI item has no reply. The Design Review note asks that — crew-pr-reviewer (r124) |
|
Re the HOLD at
Docstring only, no code change. |
|
Re the Design note on
If a maintainer prefers the split, it belongs with a |
|
CI on
These are the only two failing checks on this head. Could a maintainer re-run the failed jobs of run 40201615335? I am not pushing to reroll it, so the reviewed head stays as it is. |
e707389 to
1cf2a5e
Compare
A leaf run kept a modified test file a leaf, but a test that reads that file by name is neither an importer, a mention nor a corpus gate: test_history_locking_remediation parses test_member_memory_runtime.py, and test_windows_gap_list parses every module that windows-expected-failures.txt lists. A rename either one depends on passed the leaf run and failed the next full run. classify() now looks the changed modules up and runs each test that names one, once, beside the edit: named in its own source (bare, test/-relative, or as a node id) or in a list under test/ that the test names. The tree index records each test module's quoted file names in the pass it already makes. Readers are added per diff, not as corpus gates. Each depends only on the modules it names, and as gates the ten found on main would add 405 tests to every leaf job. The 702 corpus gates are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lookup matches only a file name spelled out in a test's source or
in a test/*.txt list, so a test that builds the name (f"{stem}.py",
path.with_name(var)) is not returned as a reader. Say so in the
docstring, so the result is not taken for every test that can read
the module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1cf2a5e to
03fb3b9
Compare
Security Scope Review (fork) — 🔴 BLOCK (blocking)Which legitimate operations does no leg reported, and this was not the nothing-to-adjudicate short circuit No |
iamwhatever
left a comment
There was a problem hiding this comment.
Approve at 03fb3b92d3978506d161ebb853fa1bfb44c11dc6.
readers_of() finds tests that name a changed test module by file name, or through a .txt list under test/, and classify() runs them once beside the edit (leaf_test_scope.py L544). A leaf run can only grow; the 702 corpus gates and the full-suite path are unchanged. The r124 hold is cleared: the docstring now states that computed names are not found (L396-L398). Checks are green on this head.
AI findings judged non-blocking
- Design Review: a quoted name in a fixture string can count as a reader. The cost is one extra test file in a leaf run, and the docstring says so (L387-L414).
- Security Scope Review (fork) BLOCK comment, "no leg reported": stale. The newer check run on this same head concluded success with no security-surface path touched (check run).
— crew-pr-reviewer (r141)
Problem / Motivation
Goal: a test-only PR's leaf run includes every test that reads a changed test file by its file name, so it can't pass a change the full suite rejects.
scripts/leaf_test_scope.pykeeps a modified test file a leaf unless another file imports it or quotes its module name, and it adds as "corpus gates" the tests that scan a directory reachingtest/or join/ "test".test/test_history_locking_remediation.pyreads one sibling file by name and passes neither check:It then asserts that
test_http_resume_cannot_authorize_private_transcriptis still in that file. It scans no directory and has no/ "test"join, so it is not a gate.mentions_ofmisses it too: its quoted-word index needs the closing quote right after the stem, and.pysits in between.Reproduced on main at
d5b8c2099c: with that function renamed intest_member_memory_runtime.py,classify({"test/test_member_memory_runtime.py": "M"})accepts a leaf plan (plus 702 corpus gate(s), the reader not among them), andpytest test/test_history_locking_remediation.pyfails (1 failed, 43 passed).Why it matters
The leaf run reports green, and the next full run on
maingoes red for whoever pushes next. The leaf reduction exists on the promise that it never skips a check the diff can break.Not a goal
f"{stem}.py"). A quoted bare stem is already caught bymentions_of; a name assembled from pieces is not, and closing that needs an import graph, which this script deliberately does not build.scripts/andsrc/literals naming atest/file on main are skip lists (check_brand_name,check_harness_parity,docs_lint), a hand-edited-files map (regen_shared), or self-test fixtures (leaf_test_scope,run_scoped_tests), not reads whose result a test asserts.What changed (motivation → approach → change)
Symptom: a leaf run skips a test that reads the changed file by name. Root cause: the selector knows three ways another test depends on a changed leaf (it imports it, quotes its module name, or scans the
test/tree), and a test that reads one named file does none of them.scripts/leaf_test_scope.py: newreaders_of(). For the changed modules it returns every top-level test that names one in its own source ("test_x.py","test/test_x.py", or a node id"test/test_x.py::test_y"), or that names a.txtlist undertest/which lists one.classify()adds those readers to that diff's run-once set. The tree index records each test module's quoted file names in the pass it already makes, so the lookup adds no tree scan; it reads only the.txtlists undertest/.test_windows_gap_listparses every modulewindows-expected-failures.txtlists and fails when a listed test is renamed. That list names 120 tests in 51 modules that nothing imports or mentions. Reproduced ond5b8c2099c: after renaming the listedTestOffloadRunsOnSubprocessPool::test_kill_process_runs_scan_and_sweep_off_loopintest_acp_offload.py, the leaf plan carries 702 gates and notest_windows_gap_list, andtest_windows_gap_listthen fails (test_every_entry_names_a_test_that_still_exists, 1 failed, 4 passed).test_local_gate::test_dry_run_against_the_real_repo_never_plans_a_full_suite, andtest_local_gatenamestest_gatewayd_diag.pyonly as a sample path. 5 of the 10 use the name only as data like that. This head leavescorpus_gates()and its 702 gates as they are on main, and a reader runs only when the diff modifies a file it names. On main, 67 of 5,320 test modules have a reader that is not already a gate: an edit to one of them adds 1 to 3 files to its run (62 add one), and an edit to any other module adds none.mentions_ofto.pyliterals would escalate every module named by an existing gate such astest_source_pin_budget(about 50 names) to the full suite..txtlists only.test/README.mdnames a module in passing, and its bare name is quoted by 65 tests that read some other README.corpus_gates(), which ci: count tests that read test/ through a helper as leaf corpus gates #23677 changes, and neither PR touches_TreeIndex,readers_oforclassify(). Theclean_leaffixture change and the new tests sit on lines neither PR edits, and a trial merge of this head with each of theirs (2b6f1d79cd,ac8efbe057) is clean. Readers are part of the gate list CI receives, so ci: run leaf corpus gates once across the matrix, not per job #23461 shards them like any gate.readers_of()'s docstring states the limit the Design note on2a06ff9af4asked for: only a file name spelled out in a source or a list is found, and a test that builds the name at run time (f"{stem}.py",path.with_name(var)) is not.Backwards compatibility
Compatible: only the set of tests a leaf run executes grows, and only for a diff that modifies a file some test reads by name. Exit codes, output shape and the full-suite path are unchanged.
Removes nothing:
corpus_gates()still derives the same 702 gates, and readers are only added.Tests
test/test_leaf_test_scope.py:test_an_edit_to_a_file_another_test_reads_by_name_runs_that_test: the reproduced case. Modifyingtest_member_memory_runtime.pyalone is still a leaf diff, andtest_history_locking_remediationruns with it. It also checks that the reader still names the file, does not scan the tree itself, and is not a static gate, so it stays a guard on this rule.test_an_edit_to_a_listed_test_runs_the_audit_of_that_list: picks a leaf fromwindows-expected-failures.txtthat no import, mention or name in the audit's source reaches, and checks thattest_windows_gap_listruns with it.test_a_reader_runs_only_for_a_diff_that_touches_a_file_it_names: on a synthetictest/tree, a bare name, atest/-relative name, a node id and a.txtlist each find their reader, and an edit to an unrelated leaf runs the corpus gate alone.clean_leaffixture now also skips a file that has a reader, sincetest_accepted_run_always_includes_the_corpus_gatesasserts the exact gate set.Proof: PROVEN test/test_leaf_test_scope.py::test_an_edit_to_a_file_another_test_reads_by_name_runs_that_test, test/test_leaf_test_scope.py::test_an_edit_to_a_listed_test_runs_the_audit_of_that_list, test/test_leaf_test_scope.py::test_a_reader_runs_only_for_a_diff_that_touches_a_file_it_names (head 2a06ff9)
Negative control:
prove.pyputs main'sscripts/leaf_test_scope.pyback under the new tests, and all three fail on an assertion. With the fix, the whole file passes: 49 passed.scripts/run_scoped_tests.py --surface backend(related set, 36 files): 2980 passed, 2 failed. Both failures are the ones the first head recorded on clean main on this host, and neither readsleaf_test_scope:test_runtime_bindings::test_the_constructor_keeps_its_type_hintsandtest_sandbox_launcher_program_parts::test_every_function_of_the_program_is_the_one_the_module_binds.Lint: black, isort, flake8,
check_subprocess_encoding,check_comment_history,check_brand_name,check_file_line_capandtest_flake_pattern_ratchet(60 passed) pass on the diff.Manual verification
Ran the selector on
d5b8c2099cagainst both broken dependencies, then ran the reader. (1)test_http_resume_cannot_authorize_private_transcriptrenamed intest_member_memory_runtime.py: this head plans 703 gates withtest_history_locking_remediation, which then fails (1 failed, 43 passed). Main's selector leaves it out, as the negative control above shows. (2) The listedtest_acp_offloadtest renamed: main plans 702 gates withouttest_windows_gap_list, which fails; this head plans 703 with it.Related Issues
Closes #23706
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
🤖 Generated with Claude Code