Repository navigation
ci: count tests that read test/ through a helper as leaf corpus gates - #23677
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. This fixes the tracked defect #23675 with a reproduction and a negative control, at the cause: gate derivation only read each test's own text. The one-level limit and the budget cost are both stated. What this change shipsInventory (5 items) — 5 justifiedIntent: stop a test-only PR's leaf CI run from passing a change that the full suite rejects. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 9a77767 Description read[DESCRIPTION-READ] 9fbf895c853e418e1c62b25ccd300c596dd617b97019af5e7664c1f45d71b91d That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
|
PR Readiness on The run behind it is Fork Design Review 40070612355, triggered when this head's Fast Gate (40070392567, attempt 2) succeeded. It failed in Resolve and validate PR: all three attempts at
|
A test-only PR runs the changed files plus the corpus gates, and a test was a gate only when its own text scans a directory and names the test tree. The fleet-probe suite's two self-scans take the tree from `pipeline_conductor_fleet_probe_helpers.SUITE_FILES` instead, so a diff to any other file of that suite ran neither scan. A bare cwd `symlink_to` planted in `_argv_row` passed the leaf run and failed the full suite. Apply the same two regexes to the shared modules directly under `test/`, and count a test that names one of them as a gate. One level only: closing over helpers that name helpers made 1,493 of 5,313 files gates. The gate set goes from 699 to 752. Closes kirodotdev#23675 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2b6f1d7 to
9a77767
Compare
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change only adds gates to the leaf selector, a revert undoes it cleanly, and it closes the reproduced helper-mediated scan gap. Notes
[DESIGN-REVIEWED] 9a77767 Description read[DESCRIPTION-READ] 4e61a36ef8d44e83a136a6d642e3869c5b5c7095a22300555dc2934d0faa145f That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
iamwhatever
left a comment
There was a problem hiding this comment.
APPROVE at 9a77767.
The fix addresses the cause in #23675: corpus_gates looked only at each test's own text, so a scan whose directory walk lives in a helper (pipeline_conductor_fleet_probe_helpers.SUITE_FILES) was not a gate. The new helper rule reuses the gate regexes, only adds gates, and the regression test covers the reproduced _argv_row edit. The extra per-job runtime is tracked in #23450 / #23461. All checks on the head are green or skipped.
AI findings judged non-blocking:
- Design Review (fork), Note: re-exports through a second helper are not followed. It is listed under Not a goal, no helper does this today, and a missed gate still runs in the full suite on main.
KiroCrew/scripts/leaf_test_scope.py
Lines 396 to 401 in 9a77767
— crew-pr-reviewer (r136)
Problem / Motivation
Goal: a test-only PR's leaf run includes every test that reads the
test/tree through a shared helper, so it can't pass a change the full suite rejects.scripts/leaf_test_scope.pyreplaces the backend matrix with the changed files plus the corpus gates: the tests that readtest/as data. A test counts as a gate only when its own text scans a directory and names the test tree.The fleet-probe suite's two self-scans get the tree from a helper:
test/pipeline_conductor_fleet_probe_helpers.pyglobstest_pipeline_conductor_fleet_probe*.pyintoSUITE_FILES._cycleand_host_linesassert a rule over every file in it, and neither one scans a directory itself, so neither is a gate. A PR that modifies only another file of that suite runs neither scan.Reproduced on main at
d6bdae3edb. A bare(entry / "cwd").symlink_to(entry)planted intest_pipeline_conductor_fleet_probe_argv_row.pyis accepted as a leaf diff (plus 699 corpus gate(s), neither scan among them). The full suite fails it:AssertionError: cwd links must use make_dir_link: [('test_pipeline_conductor_fleet_probe_argv_row.py', 301, ...)].Why it matters
A leaf run that misses the check reports green, and the defect then turns
mainred on the next full run, for whoever pushes next. This is the one outcome the leaf reduction promises to avoid ("a missed gate silently skips a check the diff really can break").Not a goal
conftest.pythat read the tree. A test reaches those without naming any module, and no such fixture exists today.What changed (motivation → approach → change)
Symptom: a leaf run skips suite-wide scans that read the changed file. Root cause:
corpus_gateslooks only at each test's own text, so a scan whose directory walk lives in a helper is invisible to it.scripts/leaf_test_scope.py: new_tree_reading_helpers()applies the same two regexes a gate uses (_SCANS_A_DIRand_REACHES_TEST_TREE) to the shared modules directly undertest/.corpus_gates()also counts a test that names one of those helpers (a whole-word match on the stem, which is over-broad on purpose:import a, bandimportlibspellings are still caught). The module docstring's list of "what a modify-only leaf diff can still break" gets a bullet for this case.acp_frame_replay_harness,dashboard_server_composition_contract_helpers,pipeline_conductor_fleet_probe_helpers,skill_catalog_walk_benchandsource_corpus.source_corpus.repo_files()lists the whole checkout,test/included. The gate set goes from 699 to 752 out of 5,313 leaf-eligible files, and no gate is dropped.conftestorsource_corpus, so the closure made 1,493 files gates and removed the reduction. The three helpers that import from a qualifying helper takeLOAD_NAME,_clear_cachesanditer_source_texts, and none of those reads a test file.docs/system-specs/common/testing-conventions.md: the line citation for_LEAF_NAMEmoves from 120-123 to 124-127.Backwards compatibility
Compatible: only the set of tests a leaf run executes grows. Exit codes, output shape and the full-suite path are unchanged.
Removes nothing: the derivation only adds gates; every gate derived before is still derived (699 ⊂ 752, checked against main's
corpus_gates).Tests
test/test_leaf_test_scope.py:test_an_edit_to_one_suite_file_runs_the_scans_over_the_suite: modifying_argv_rowalone is still a leaf diff, and both self-scans are in its gate set. This is the reproduced case.test_a_gate_that_reads_the_tree_through_a_helper_is_derived[_cycle|_host_lines]: mutation guard. Each file readsSUITE_FILES, neither scans the tree itself, and each is derived only through the helper rule.Proof: PROVEN test/test_leaf_test_scope.py::test_an_edit_to_one_suite_file_runs_the_scans_over_the_suite (head 2b6f1d7)
Negative control: with main's
scripts/leaf_test_scope.pyand these tests, all three new tests fail, the regression with the realAssertionError(scans missing from the gate set). The whole file passes with the fix: 49 passed.The 53 newly derived gates, run once on this Windows host (
-n 8): 4,246 passed and 7 failed in 135 s. All 7 fail the same way on clean main atd6bdae3edb, because of this host:kiro_cli_pin/optpaths, a cp950 decode intest_host_service_guard, WinError 1314 symlink privilege, and a POSIX root check in_cycle.scripts/local-gate.py(backend, related set): 4 failures. All 4 fail the same way on clean main on this host:test_runtime_bindings::test_the_constructor_keeps_its_type_hints,test_sandbox_launcher_program_parts::test_every_function_of_the_program_is_the_one_the_module_binds, and twoRecursionErrorcases intest_acp_client_more_coverage_encode_refusal.Manual verification
Ran the selector against the planted
_argv_rowedit above, before and after the change. Before: the leaf plan holds 699 gates without either scan, and the full-suite scan fails on the edit. After: both scans are in the plan.Related Issues
Closes #23675
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
🤖 Generated with Claude Code