Skip to content

ci: count tests that read test/ through a helper as leaf corpus gates - #23677

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/leaf-helper-corpus-gates
Oct 30, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/leaf-helper-corpus-gates

Conversation

@leonlaiyc

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a test-only PR's leaf run includes every test that reads the test/ tree through a shared helper, so it can't pass a change the full suite rejects.

scripts/leaf_test_scope.py replaces the backend matrix with the changed files plus the corpus gates: the tests that read test/ as data. A test counts as a gate only when its own text scans a directory and names the test tree.

The fleet-probe suite's two self-scans get the tree from a helper: test/pipeline_conductor_fleet_probe_helpers.py globs test_pipeline_conductor_fleet_probe*.py into SUITE_FILES. _cycle and _host_lines assert a rule over every file in it, and neither one scans a directory itself, so neither is a gate. A PR that modifies only another file of that suite runs neither scan.

Reproduced on main at d6bdae3edb. A bare (entry / "cwd").symlink_to(entry) planted in test_pipeline_conductor_fleet_probe_argv_row.py is accepted as a leaf diff (plus 699 corpus gate(s), neither scan among them). The full suite fails it: AssertionError: cwd links must use make_dir_link: [('test_pipeline_conductor_fleet_probe_argv_row.py', 301, ...)].

Why it matters

A leaf run that misses the check reports green, and the defect then turns main red on the next full run, for whoever pushes next. This is the one outcome the leaf reduction promises to avoid ("a missed gate silently skips a check the diff really can break").

Not a goal

What changed (motivation → approach → change)

Symptom: a leaf run skips suite-wide scans that read the changed file. Root cause: corpus_gates looks only at each test's own text, so a scan whose directory walk lives in a helper is invisible to it.

  • scripts/leaf_test_scope.py: new _tree_reading_helpers() applies the same two regexes a gate uses (_SCANS_A_DIR and _REACHES_TEST_TREE) to the shared modules directly under test/. corpus_gates() also counts a test that names one of those helpers (a whole-word match on the stem, which is over-broad on purpose: import a, b and importlib spellings are still caught). The module docstring's list of "what a modify-only leaf diff can still break" gets a bullet for this case.
  • On main, five helpers qualify: acp_frame_replay_harness, dashboard_server_composition_contract_helpers, pipeline_conductor_fleet_probe_helpers, skill_catalog_walk_bench and source_corpus. source_corpus.repo_files() lists the whole checkout, test/ included. The gate set goes from 699 to 752 out of 5,313 leaf-eligible files, and no gate is dropped.
  • One level only. I measured the fixed point over helpers that name helpers first: nearly every helper names conftest or source_corpus, so the closure made 1,493 files gates and removed the reduction. The three helpers that import from a qualifying helper take LOAD_NAME, _clear_caches and iter_source_texts, and none of those reads a test file.
  • docs/system-specs/common/testing-conventions.md: the line citation for _LEAF_NAME moves from 120-123 to 124-127.

Backwards compatibility

Compatible: only the set of tests a leaf run executes grows. Exit codes, output shape and the full-suite path are unchanged.

Removes nothing: the derivation only adds gates; every gate derived before is still derived (699 ⊂ 752, checked against main's corpus_gates).

Tests

test/test_leaf_test_scope.py:

  • test_an_edit_to_one_suite_file_runs_the_scans_over_the_suite: modifying _argv_row alone is still a leaf diff, and both self-scans are in its gate set. This is the reproduced case.
  • test_a_gate_that_reads_the_tree_through_a_helper_is_derived[_cycle|_host_lines]: mutation guard. Each file reads SUITE_FILES, neither scans the tree itself, and each is derived only through the helper rule.

Proof: PROVEN test/test_leaf_test_scope.py::test_an_edit_to_one_suite_file_runs_the_scans_over_the_suite (head 2b6f1d7)

Negative control: with main's scripts/leaf_test_scope.py and these tests, all three new tests fail, the regression with the real AssertionError (scans missing from the gate set). The whole file passes with the fix: 49 passed.

The 53 newly derived gates, run once on this Windows host (-n 8): 4,246 passed and 7 failed in 135 s. All 7 fail the same way on clean main at d6bdae3edb, because of this host: kiro_cli_pin /opt paths, a cp950 decode in test_host_service_guard, WinError 1314 symlink privilege, and a POSIX root check in _cycle.

scripts/local-gate.py (backend, related set): 4 failures. All 4 fail the same way on clean main on this host: test_runtime_bindings::test_the_constructor_keeps_its_type_hints, test_sandbox_launcher_program_parts::test_every_function_of_the_program_is_the_one_the_module_binds, and two RecursionError cases in test_acp_client_more_coverage_encode_refusal.

Manual verification

Ran the selector against the planted _argv_row edit above, before and after the change. Before: the leaf plan holds 699 gates without either scan, and the full-suite scan fails on the edit. After: both scans are in the plan.

Related Issues

Closes #23675

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

@leonlaiyc
leonlaiyc requested review from a team and KiroCrewCaptain as code owners October 24, 2026 06:29
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 24, 2026
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 9a77767dbbe333670588ad73a16d85e50a13f05f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 9a77767dbbe333670588ad73a16d85e50a13f05f via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. This fixes the tracked defect #23675 with a reproduction and a negative control, at the cause: gate derivation only read each test's own text. The one-level limit and the budget cost are both stated.

What this change ships

Inventory (5 items) — 5 justified

Intent: stop a test-only PR's leaf CI run from passing a change that the full suite rejects. This is a FIX.

  1. A leaf run now also runs tests that read test/ through a shared helper (gates go from 699 to 752) — justified
  2. Every leaf matrix job runs 53 more gates until ci: run leaf corpus gates once across the matrix, not per job #23461 lands — justified
  3. The leaf_test_scope.py docstring gets a bullet for gates that read the tree through a helper — justified
  4. The _LEAF_NAME line citation in testing-conventions.md is updated — justified
  5. A regression test for the _argv_row edit, plus a mutation guard for _cycle/_host_lines — justified

[FIRST-PRINCIPLES-REVIEWED] 9a77767

Description read

[DESCRIPTION-READ] 9fbf895c853e418e1c62b25ccd300c596dd617b97019af5e7664c1f45d71b91d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 9a77767dbbe333670588ad73a16d85e50a13f05f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9a77767

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 9a77767dbbe333670588ad73a16d85e50a13f05f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9a77767

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 24, 2026
@leonlaiyc

leonlaiyc commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

PR Readiness on 2b6f1d79cd has been pending since 06:32 UTC, "waiting on Design Review (not started)". The fork Design lane never opened its check-run for this head, though nothing about the code was judged.

The run behind it is Fork Design Review 40070612355, triggered when this head's Fast Gate (40070392567, attempt 2) succeeded. It failed in Resolve and validate PR: all three attempts at GET /repos/kirodotdev/KiroCrew/pulls?state=open got dial tcp 140.82.116.5:443: connect: connection refused (06:32:44 to 06:32:59 UTC). The job then ended cancelled. This is the failure #23596 describes, which #23705 is fixing. The First Principles, Opus and GPT fork lanes from the same trigger completed and posted above.

pr-readiness-sweep.yml says a lane with no run on a head clears only by a push or by re-running that lane, and a fork author can't re-run upstream runs. Could a maintainer re-run run 40070612355? I'm not pushing a new commit just to retrigger it.

A test-only PR runs the changed files plus the corpus gates, and a test
was a gate only when its own text scans a directory and names the test
tree. The fleet-probe suite's two self-scans take the tree from
`pipeline_conductor_fleet_probe_helpers.SUITE_FILES` instead, so a diff
to any other file of that suite ran neither scan. A bare cwd
`symlink_to` planted in `_argv_row` passed the leaf run and failed the
full suite.

Apply the same two regexes to the shared modules directly under
`test/`, and count a test that names one of them as a gate. One level
only: closing over helpers that name helpers made 1,493 of 5,313 files
gates. The gate set goes from 699 to 752.

Closes kirodotdev#23675

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamwhatever
iamwhatever force-pushed the claude/leaf-helper-corpus-gates branch from 2b6f1d7 to 9a77767 Compare October 30, 2026 04:49
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 9a77767dbbe333670588ad73a16d85e50a13f05f via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only adds gates to the leaf selector, a revert undoes it cleanly, and it closes the reproduced helper-mediated scan gap.

Notes

  • The one-level helper rule misses a helper that re-exports a tree-reading object from another helper. The docstring says so; a pinning test that fails when a helper imports a qualifying helper's tree object would turn that silent gap into a red test.

[DESIGN-REVIEWED] 9a77767

Description read

[DESCRIPTION-READ] 4e61a36ef8d44e83a136a6d642e3869c5b5c7095a22300555dc2934d0faa145f

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 30, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at 9a77767.

The fix addresses the cause in #23675: corpus_gates looked only at each test's own text, so a scan whose directory walk lives in a helper (pipeline_conductor_fleet_probe_helpers.SUITE_FILES) was not a gate. The new helper rule reuses the gate regexes, only adds gates, and the regression test covers the reproduced _argv_row edit. The extra per-job runtime is tracked in #23450 / #23461. All checks on the head are green or skipped.

AI findings judged non-blocking:

  • Design Review (fork), Note: re-exports through a second helper are not followed. It is listed under Not a goal, no helper does this today, and a missed gate still runs in the full suite on main.
    One level, not a closure over helpers that name helpers: nearly every helper
    names `conftest` or `source_corpus`, and the closure made 1,493 files gates,
    which is no reduction at all. A helper that RE-EXPORTS a tree-reading object
    from another helper is therefore not seen. None does today: the three helpers
    that import from one take `LOAD_NAME`, `_clear_caches` and `iter_source_texts`,

— crew-pr-reviewer (r136)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 30, 2026 17:45
@iamwhatever
iamwhatever merged commit f6cab90 into kirodotdev:main Oct 30, 2026
93 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: leaf-test runs skip corpus gates that read test/ through a helper module

2 participants