Repository navigation
test: keep ratchet and count tests green when two PRs merge - #37660
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns. The shared total that kept turning Notes
[DESIGN-REVIEWED] 858baec Description read[DESCRIPTION-READ] 829250a3337877823aa81ddf19033e08e964f69d334a84543a8323b0eeaa5cf7 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the dropped total is redundant with the exact table the test still compares against, the budget message fix sits at mechanism level, and K21 guards a defect with linked runs. Notes
What this change shipsInventory (6 items) — 4 justifiedIntent: keep
[FIRST-PRINCIPLES-REVIEWED] 858baec Description read[DESCRIPTION-READ] e12c20cc4a45f7a03993de2992650e9b54b6b904c340af75ddbb5c94d4f00ec7 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 446 lines, more than the 400 it may approve on its own. Test/docs-only change that removes a redundant shared total ( An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
59467a8 to
04e79c3
Compare
|
Rebased: main moved in 8 commits (e58284e -> 56f3e84), one of which re-pinned test/test_dashboard_server_composition_contract.py's lazy-import total to 156 while this PR removes that total. Resolved by keeping the removal; the exact per-module mapping still decides. Re-ran test_dashboard_server_composition_contract.py, test_security_gate_liveness.py, test_flake_pattern_ratchet.py and test_file_line_cap.py on the new base: 264 passed. Also adds the K21 shared-total-pin guard commit; body updated. New head: 04e79c3. |
9dc36c0 to
1c61ae7
Compare
|
Rebased: main moved in (56f3e84 -> 4dec84c), touching AUTOSDE.yaml, docs/system-specs/common/testing-conventions.md and src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md that this PR also edits; rebased cleanly. Re-ran the composition contract, line budget, flake-pattern ratchet, file line cap and the tests that read those three files on the new base: 416 passed; docs-lint passed. New head: 29ea3be. |
1c61ae7 to
29ea3be
Compare
29ea3be to
858baec
Compare
|
Rebased: main moved in 8 commits (4dec84c -> 7d8aa4e), one editing docs/system-specs/common/testing-conventions.md, which this PR also edits; rebased cleanly. Re-ran the composition contract, line budget, flake-pattern ratchet, file line cap and the tests that read the edited docs on the new base: 416 passed; docs-lint passed. New head: 858baec. |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approving. On this head the review confirmed three things. First, dropping assert len(expected) == 156 loosens nothing: the exact local == {name: {module}} mapping already implies it, and the new duplicate and empty checks pass on the current 156-name table. Second, the line-budget change rewords only the failure message, and total <= budget is unchanged. Third, the touched tests and the K21 detector cases pass. Over every test file the detector finds the sites the body lists, it gains no site in any open PR, and putting the old pin back would fail the ratchet.
Done well: K21 lives in its own module and counts each file against the merge-base, so new shared-total pins are stopped and the existing ones fail no one.
Optional:
- test/test_security_gate_liveness.py:913: the example
git diff --numstat ... -- src/kiro_crew/security/also countsredaction_structured/, which the budget does not measure.':(glob)src/kiro_crew/security/*.py'matches what is counted. - test/flake_pattern_shared_total.py:249: no case covers the rule that a literal pin counts only inside an
assert. A non-assertif len(found) == 154:in the K21 near-miss would pin it. - K21's thresholds are written out in AUTOSDE.yaml, testing-conventions.md and SKILL.md. A pointer to the doc section would keep them in step with
MIN_PINNED_TOTALand_TOTAL_WORDS.
Problem / Motivation
Goal: Ratchet and count tests stay green on
mainwhen two pull requests that were each green on the same base both merge.Each PR's CI runs on its own base and the repository merges without testing the combined tree. A ratchet whose record is one shared number breaks there: two PRs that each add one site both rewrite that number to the same value, git merges the identical edit cleanly, and
mainholds one more site than the record says.test_dashboard_server_composition_contract.py'sassert len(expected) == 154did this four times in 2026-11 (14 redmainruns, e.g. run 42668352886,assert 144 == 143), each fixed by a re-pin PR.Why it matters
A red
mainfails every open PR's CI until someone re-pins the number, and the re-pin is itself racy. Writing the rule down stops new ratchets from repeating the shape.Not a goal
test/test_pid_reader_ratchet.pymoves to a per-file record in its own PR (Part of test: ratchet and count tests turn main red when two green PRs both merge #37656).len(_BASE_NAMES) == 317,91 + 22,== 101) stay: no later change edits them.What changed (motivation → approach → change)
test/test_dashboard_server_composition_contract.pylazy importstest/test_security_gate_liveness.pyline budget+{total - budget}, the amount OVER budget: growth minus whatever headroom the base had, so two rows written that way under-count once both mergetotal <= budget) is unchangedtest/test_flake_pattern_ratchet.pyshapeK21-shared-total-pin(detector intest/flake_pattern_shared_total.py)==an integer literal of 10 or more, or==/!=an ALL-CAPS_TOTAL/_BASELINE-style constant, is counted per file against the merge-base like the other shapes. A changed file fails only when its K21 count rises;# flake-ok: <reason>excuses a deliberate site. Existing sites are the baseline, not failuresThe rule goes where authors and reviewers read it: a new Merge-safe ratchets and counts section in
docs/system-specs/common/testing-conventions.md, a checklist line in the writing-tests guide (src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md), and a paragraph inAUTOSDE.yaml'sa-ratchet-may-only-tightenthat flags a new single-total record and says moving to a keyed record is tightening.The section also names checks that must stay exact because a red after a merge is a real defect there:
test_updates_handlers_facade.py(facade manifest vs owner imports) and the docs-index reachability gate.Backwards compatibility
Compatible: test, guide and docs text only; no product code or contract changes.
Removes nothing: the dropped
len(expected) == 156(re-pinned onmainwhile this PR was open, which is the race this fixes) is implied by the exactlocal == expectedcomparison that stays.Tests
Two-branch merge simulation (scratch clone; branches A and B cut from one base, each adds one lazy import in a different owner module and updates the record exactly as the test asks, then
git merge):git mergetest_the_lazy_imports_stay_inside_the_functions_that_need_themorigin/main61bef4eassert 156 == 155Determinism: both changed files 3× serial (
-n0) and once in reversed file order with-n 4: 59 passed each time. Related backend set (scripts/run_scoped_tests.py): 1927 passed.After rebasing onto 56f3e84: the composition contract, line budget, flake-pattern ratchet (with the K21 fixture cases) and file line cap tests, 264 passed. The K21 detector is a heuristic; its fixture cases cover a non-walking module (0 counted) and a walking one (2 counted).
K21 false-positive check
Ran the detector (driver: the ratchet's own
PY_DETECTORS["K21-shared-total-pin"]and its per-file class-total rule) over every Python test file onorigin/main56f3e84, and over the diffs of the 30 most recent open PRs that change a Python test file (632 changed test files in total: #37673 #37670 #37668 #37667 #37665 #37662 #37659 #37654 #37653 #37652 #37646 #37643 #37633 #37629 #37627 #37626 #37619 #37616 #37615 #37609 #37604 #37601 #37598 #37596 #37572 #37568 #37567 #37565 #37559 #37558).Open PRs: 0 blocked. #37633 and #37598 re-pin
len(expected)in the composition contract; a re-pin keeps the file's K21 count, so the ratchet passes them.main(existing sites become the baseline and do not fail; listed to judge what K21 matches):test_dashboard_server_composition_contract.pylen(expected) == 156test_dashboard_agents_composition_contract.pysum(missing_code) == 37test_dashboard_files_composition_contract.pysum(missing_code) == 80test_handlers_messaging_composition_contract.pysum(_CENSUS_HOMES.values()) == 12test_handlers_messaging_composition_contract.pylen(reexports) == 53test_http_server_bind_ratchet.pylen(sites) == _BASELINE_SITEStest_kill_chokepoint_ratchet.pytotal == BYPASS_BASELINEtest_pid_reader_ratchet.pyscan.owner_marked == _BASELINE_OWNER_MARKEDtest_transient_notice_parity.pylen(sites) == 36test_crew_log_types.pylen(SESSION_ENTRY_TYPES) == 53test_codex_stt.pylen(files) == 13(locale files)test_fixture_sessions_long_history.py... == ROTATED_ROWS(sessions / "archive").globon a seeded fixture home, and the module only matched the product-tree gate through an unrelated"src"pathFix for the false positive, in the detector rather than by a
flake-ok: a helper whose walk is rooted at its own parameter is a tree walk only where a caller passes it a module-level root (_scan_tree(_SRC)), not a local or its own parameter (_segments(home / "sessions")). After the change themainscan counts the 11 real sites and not the fixture one; both shapes are inK21_CASES. Remaining known miss: a parameter-rooted helper called with a local that holds the product root.Proof: not provable -- test-only change
Manual verification
N/A — test and docs only; the simulation above is the behavioral check.
Related Issues
Part of #37656
Checklist