Skip to content

test: keep ratchet and count tests green when two PRs merge - #37660

Merged
KiroCrewCaptain merged 2 commits into
mainfrom
crew/linuxflackyguard/merge-safe-ratchets
Nov 15, 2026
Merged

KiroCrewCaptain merged 2 commits into
mainfrom
crew/linuxflackyguard/merge-safe-ratchets

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Ratchet and count tests stay green on main when two pull requests that were each green on the same base both merge.

Each PR's CI runs on its own base and the repository merges without testing the combined tree. A ratchet whose record is one shared number breaks there: two PRs that each add one site both rewrite that number to the same value, git merges the identical edit cleanly, and main holds one more site than the record says. test_dashboard_server_composition_contract.py's assert len(expected) == 154 did this four times in 2026-11 (14 red main runs, e.g. run 42668352886, assert 144 == 143), each fixed by a re-pin PR.

Why it matters

A red main fails every open PR's CI until someone re-pins the number, and the re-pin is itself racy. Writing the rule down stops new ratchets from repeating the shape.

Not a goal

  • No merge queue, no workflow, ruleset or repository-setting change.
  • test/test_pid_reader_ratchet.py moves to a per-file record in its own PR (Part of test: ratchet and count tests turn main red when two green PRs both merge #37656).
  • No loosening: every check that was exact stays exact.
  • Frozen snapshots of one-time events in the composition contract (len(_BASE_NAMES) == 317, 91 + 22, == 101) stay: no later change edits them.

What changed (motivation → approach → change)

Test Why it went red after a merge Change
test/test_dashboard_server_composition_contract.py lazy imports A hand-kept total sat beside an exact mapping that already decides it; the total is the one line two PRs edit to the same number Drop the total. Keep the exact mapping (one line per module). Add checks that no name is listed under two modules and that the table is not empty
test/test_security_gate_liveness.py line budget Rows are already one file per change, but the failure message asked for +{total - budget}, the amount OVER budget: growth minus whatever headroom the base had, so two rows written that way under-count once both merge The message asks for the diff's own net line change and says why the overflow is wrong. The check (total <= budget) is unchanged
New guard: test/test_flake_pattern_ratchet.py shape K21-shared-total-pin (detector in test/flake_pattern_shared_total.py) Nothing stopped the next test from pinning one shared total again In a module that walks the product tree, a count pinned == an integer literal of 10 or more, or ==/!= an ALL-CAPS _TOTAL/_BASELINE-style constant, is counted per file against the merge-base like the other shapes. A changed file fails only when its K21 count rises; # flake-ok: <reason> excuses a deliberate site. Existing sites are the baseline, not failures

The rule goes where authors and reviewers read it: a new Merge-safe ratchets and counts section in docs/system-specs/common/testing-conventions.md, a checklist line in the writing-tests guide (src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md), and a paragraph in AUTOSDE.yaml's a-ratchet-may-only-tighten that flags a new single-total record and says moving to a keyed record is tightening.

The section also names checks that must stay exact because a red after a merge is a real defect there: test_updates_handlers_facade.py (facade manifest vs owner imports) and the docs-index reachability gate.

Backwards compatibility

Compatible: test, guide and docs text only; no product code or contract changes.

Removes nothing: the dropped len(expected) == 156 (re-pinned on main while this PR was open, which is the race this fixes) is implied by the exact local == expected comparison that stays.

Tests

Two-branch merge simulation (scratch clone; branches A and B cut from one base, each adds one lazy import in a different owner module and updates the record exactly as the test asks, then git merge):

Base git merge test_the_lazy_imports_stay_inside_the_functions_that_need_them
origin/main 61bef4e clean FAILED assert 156 == 155
this branch clean passed

Determinism: both changed files 3× serial (-n0) and once in reversed file order with -n 4: 59 passed each time. Related backend set (scripts/run_scoped_tests.py): 1927 passed.

After rebasing onto 56f3e84: the composition contract, line budget, flake-pattern ratchet (with the K21 fixture cases) and file line cap tests, 264 passed. The K21 detector is a heuristic; its fixture cases cover a non-walking module (0 counted) and a walking one (2 counted).

K21 false-positive check

Ran the detector (driver: the ratchet's own PY_DETECTORS["K21-shared-total-pin"] and its per-file class-total rule) over every Python test file on origin/main 56f3e84, and over the diffs of the 30 most recent open PRs that change a Python test file (632 changed test files in total: #37673 #37670 #37668 #37667 #37665 #37662 #37659 #37654 #37653 #37652 #37646 #37643 #37633 #37629 #37627 #37626 #37619 #37616 #37615 #37609 #37604 #37601 #37598 #37596 #37572 #37568 #37567 #37565 #37559 #37558).

Open PRs: 0 blocked. #37633 and #37598 re-pin len(expected) in the composition contract; a re-pin keeps the file's K21 count, so the ratchet passes them.

main (existing sites become the baseline and do not fail; listed to judge what K21 matches):

Site Verdict
test_dashboard_server_composition_contract.py len(expected) == 156 real: the total this PR removes
test_dashboard_agents_composition_contract.py sum(missing_code) == 37 real: shared total beside exact per-file rows
test_dashboard_files_composition_contract.py sum(missing_code) == 80 real: same
test_handlers_messaging_composition_contract.py sum(_CENSUS_HOMES.values()) == 12 real: same
test_handlers_messaging_composition_contract.py len(reexports) == 53 real: re-pinned 6 times in 30 days
test_http_server_bind_ratchet.py len(sites) == _BASELINE_SITES real: one shared baseline
test_kill_chokepoint_ratchet.py total == BYPASS_BASELINE real: one shared baseline
test_pid_reader_ratchet.py scan.owner_marked == _BASELINE_OWNER_MARKED real: #37661
test_transient_notice_parity.py len(sites) == 36 real: shared total
test_crew_log_types.py len(SESSION_ENTRY_TYPES) == 53 real: re-pinned 7 times in 30 days
test_codex_stt.py len(files) == 13 (locale files) real, low churn: two PRs each adding a locale would merge red
test_fixture_sessions_long_history.py ... == ROTATED_ROWS false positive: the walk is (sessions / "archive").glob on a seeded fixture home, and the module only matched the product-tree gate through an unrelated "src" path

Fix for the false positive, in the detector rather than by a flake-ok: a helper whose walk is rooted at its own parameter is a tree walk only where a caller passes it a module-level root (_scan_tree(_SRC)), not a local or its own parameter (_segments(home / "sessions")). After the change the main scan counts the 11 real sites and not the fixture one; both shapes are in K21_CASES. Remaining known miss: a parameter-rooted helper called with a local that holds the product root.

Proof: not provable -- test-only change

Manual verification

N/A — test and docs only; the simulation above is the behavioral check.

Related Issues

Part of #37656

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 18:33
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 858baec253be61e2691e646b087db68f1133eaf4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 858baec253be61e2691e646b087db68f1133eaf4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns. The shared total that kept turning main red is removed, the exact mapping beside it already covers it, and the new K21 guard is a reversible per-file heuristic.

Notes

  • K21 detects a shared total from what the code looks like, not what it does, and the description lists a known miss: a helper rooted at its own parameter but called with a local holding the product root. The two-branch merge simulation is the real proof. If K21 starts producing flake-ok noise, tighten the detector rather than collecting more excuses.

[DESIGN-REVIEWED] 858baec

Description read

[DESCRIPTION-READ] 829250a3337877823aa81ddf19033e08e964f69d334a84543a8323b0eeaa5cf7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 858baec253be61e2691e646b087db68f1133eaf4 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the dropped total is redundant with the exact table the test still compares against, the budget message fix sits at mechanism level, and K21 guards a defect with linked runs.

Notes

  • Item 4 — symptom-level (siblings deferred): the grep == _BASELINE_SITES|== BYPASS_BASELINE|== _BASELINE_OWNER_MARKED|len\(reexports\) == \d+ finds 8 unfixed sites of the same shared-total shape in 7 files, including test_fd_census_ratchet.py:77, test_http_server_bind_ratchet.py:91, test_kill_chokepoint_ratchet.py:174 and test_handlers_messaging_composition_contract.py:418. Only test_pid_reader_ratchet.py has a follow-up PR named (test: ratchet and count tests turn main red when two green PRs both merge #37656). The others become K21 baseline, which means they are tolerated, not removed, so they can still turn main red after a merge.
  • Item 6 — oversized: the same rule is written out in full three times, in testing-conventions.md, the writing-tests checklist and the AUTOSDE.yaml paragraph, and the copies will drift apart. Subtraction: cut the AUTOSDE and SKILL.md additions to one line each that points at § Merge-safe ratchets and counts.

What this change ships

Inventory (6 items) — 4 justified

Intent: keep main green when two PRs that each passed on the same base both merge. This is a FIX.

  1. The lazy-import test no longer pins a total of 156; the exact table stays the record — justified
  2. The lazy-import test now fails when a name is listed under two modules or the table is empty — justified
  3. The security line-budget failure message asks for the diff's own net line change, not the amount over budget — justified
  4. A new flake-ratchet shape, K21, refuses new shared-total pins in tests that walk the product tree — symptom-level, siblings deferred
  5. test_k21_reads_only_a_module_that_walks_the_product_tree checks that a fixture walk is not counted — justified
  6. The merge-safe ratchet rule is written into the testing conventions, the writing-tests skill and AUTOSDE — oversized, three copies

[FIRST-PRINCIPLES-REVIEWED] 858baec

Description read

[DESCRIPTION-READ] e12c20cc4a45f7a03993de2992650e9b54b6b904c340af75ddbb5c94d4f00ec7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 858baec253be61e2691e646b087db68f1133eaf4 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 858baec

Verdict parsed from the review's SHA-scoped output markers for commit 858baec253be61e2691e646b087db68f1133eaf4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 858baec253be61e2691e646b087db68f1133eaf4 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 858baec253be61e2691e646b087db68f1133eaf4 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 858baec

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 858baec253be61e2691e646b087db68f1133eaf4 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 446 lines, more than the 400 it may approve on its own.

Test/docs-only change that removes a redundant shared total (len(expected) == 156), which the exact local == expected mapping already decides. A new duplicate-name check covers the one thing the total implicitly checked. The line-budget failure message is fixed to match the README's own-delta rule; the check itself is unchanged. A K21 heuristic is added to the flake ratchet, which counts per file against the merge-base, so existing sites become the baseline and main stays green. The detector cases trace correctly and CI, including the flake-pattern ratchet, is green. No trust boundary or product code is touched.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 858baec253 · 446 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 15, 2026
@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/merge-safe-ratchets branch from 59467a8 to 04e79c3 Compare November 15, 2026 19:21
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 8 commits (e58284e -> 56f3e84), one of which re-pinned test/test_dashboard_server_composition_contract.py's lazy-import total to 156 while this PR removes that total. Resolved by keeping the removal; the exact per-module mapping still decides. Re-ran test_dashboard_server_composition_contract.py, test_security_gate_liveness.py, test_flake_pattern_ratchet.py and test_file_line_cap.py on the new base: 264 passed. Also adds the K21 shared-total-pin guard commit; body updated. New head: 04e79c3.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in (56f3e84 -> 4dec84c), touching AUTOSDE.yaml, docs/system-specs/common/testing-conventions.md and src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md that this PR also edits; rebased cleanly. Re-ran the composition contract, line budget, flake-pattern ratchet, file line cap and the tests that read those three files on the new base: 416 passed; docs-lint passed. New head: 29ea3be.

@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/merge-safe-ratchets branch from 1c61ae7 to 29ea3be Compare November 15, 2026 21:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 15, 2026
@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/merge-safe-ratchets branch from 29ea3be to 858baec Compare November 15, 2026 22:16
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 8 commits (4dec84c -> 7d8aa4e), one editing docs/system-specs/common/testing-conventions.md, which this PR also edits; rebased cleanly. Re-ran the composition contract, line budget, flake-pattern ratchet, file line cap and the tests that read the edited docs on the new base: 416 passed; docs-lint passed. New head: 858baec.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 15, 2026 23:58

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. On this head the review confirmed three things. First, dropping assert len(expected) == 156 loosens nothing: the exact local == {name: {module}} mapping already implies it, and the new duplicate and empty checks pass on the current 156-name table. Second, the line-budget change rewords only the failure message, and total <= budget is unchanged. Third, the touched tests and the K21 detector cases pass. Over every test file the detector finds the sites the body lists, it gains no site in any open PR, and putting the old pin back would fail the ratchet.
Done well: K21 lives in its own module and counts each file against the merge-base, so new shared-total pins are stopped and the existing ones fail no one.
Optional:

  • test/test_security_gate_liveness.py:913: the example git diff --numstat ... -- src/kiro_crew/security/ also counts redaction_structured/, which the budget does not measure. ':(glob)src/kiro_crew/security/*.py' matches what is counted.
  • test/flake_pattern_shared_total.py:249: no case covers the rule that a literal pin counts only inside an assert. A non-assert if len(found) == 154: in the K21 near-miss would pin it.
  • K21's thresholds are written out in AUTOSDE.yaml, testing-conventions.md and SKILL.md. A pointer to the doc section would keep them in step with MIN_PINNED_TOTAL and _TOTAL_WORDS.

@KiroCrewCaptain
KiroCrewCaptain merged commit 5957315 into main Nov 15, 2026
99 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/linuxflackyguard/merge-safe-ratchets branch November 15, 2026 23:58
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants