Repository navigation
test: derive notice, session-tool, crew-log and provider counts instead of pinning totals - #37685
Conversation
…ad of pinning totals Two PRs that each add one item edit a stored total to the same new number, git merges them cleanly, and main is red. Replace each total with the exact per-item record or set the test already compares, a looser parse of the same source, or prose that names no number.
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns. Each pinned total is replaced by an exact set or per-site record, which is stricter than the count and safe when two PRs merge. Notes
[DESIGN-REVIEWED] 42734bf Description read[DESCRIPTION-READ] 77e95326f495b9f750d7bb6eee3230d97b75f16c8d843eb0041fd7ae233b27b4 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. The diff changes tests and doc wording only, and adds or changes no dashboard control, so no screenshot is needed. [UX-REVIEWED] 42734bf Description read[DESCRIPTION-READ] 098b2598e2f9e740c7c9181cbb29429a2ab1ae106907427a818469de2a5f67fc That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Every replaced total either becomes an exact set check or is dropped beside one that already exists. No new product surface comes with it. Not checked: Notes
What this change shipsInventory (8 items) — 6 justifiedIntent: keep
[FIRST-PRINCIPLES-REVIEWED] 42734bf Description read[DESCRIPTION-READ] 0ca677e038bf16f69cfeda66c56ea088780e874225787e6916f1eca25fd5c0ea That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 7% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Test-only change (plus doc wording). Fixed totals in four test areas become exact set or per-line checks, so two PRs that each add an entry no longer merge into a red main. Checked: _backend_tokens() returns a name→token dict, so comparing its keys to the new loose scan is valid. The session-control bullets list 26 + 15 = 41 tools, matching the code's two groups. Connections' Connect and Set-up rows must now equal get_visible_providers(). The crew-log summary rows must equal the documented types. as_posix() keeps the site paths portable on Windows. No other reader of the removed doc counts exists in the repo. All CI and AI review lanes passed on this head. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Reviewed head 42734bf. The changed backend test files pass on this head and again after merging current main. I also confirmed that the frontend token scan and the new loose assignment scan agree on the same 91 tokens. Every total that was removed now has an exact set or per-site check against a non-empty source in its place, so none of the new assertions is vacuous. Coverage holds where a count was simply dropped, because the remaining checks still pin those sets. This is a test and doc wording change with all CI green, and I found no bug that blocks it.
Problem / Motivation
Goal: The notice-parity, session-tool, crew-log type and provider-catalogue tests stay green on
mainwhen two pull requests that were each green on the same base both merge.Each pinned one shared total, in a test or in the doc a test reads. Two PRs that each add one entry rewrite that number to the same value, git merges the identical edit cleanly, and
mainholds one entry more than the number says. Onmainin 30 days: the transient-notice token totals were re-pinned 11, 11, 5, 11 and 10 times; "41" tools insession-control.md7 times;EXPECTED_LIVE_TYPES = 536 times; the "38" providers inconnections.md4 times.Why it matters
A red
mainfails every open PR's CI until someone re-pins the number, and the re-pin races the next one.Not a goal
src/kiro_crew/docs/ui-index.generated.json'scoverageline (84 re-pins in 30 days) is a generated file and gets its own change.What changed (motivation → approach → change)
test/test_transient_notice_parity.py,website/src/test/gatewayNotice.test.tsx91 / 91 / 81,toHaveLength(91),len(copy) == 107,len(texts) == 13 / 16;len(sites) == 36re:occurrences in the table;_TOKENIZED_SITESlists one site per linetest/test_session_control_doc.py,src/kiro_crew/docs/session-control.md,src/kiro_crew/docs/index.mdlen(SESSION_CONTROL_TOOLS) == 26test/test_docs_crew_log_reference.py,test/test_crew_log_types.pyEXPECTED_LIVE_TYPES = 53,len(SESSION_ENTRY_TYPES) == 53test/test_connections_doc.py,src/kiro_crew/docs/connections.md38-providerin index.md, "eleven" aliasesget_visible_providers(). The old count-word test only knew words up to eleven, so the twelfth aliased provider would have failed its own PRThe rule is the one in
docs/system-specs/common/testing-conventions.md§ Merge-safe ratchets and counts (PR #37660). Two of these totals (len(sites) == 36,len(SESSION_ENTRY_TYPES) == 53) are among the sites that PR's K21 guard counts.Backwards compatibility
Compatible: tests and doc wording only. The docs still name every tool and provider; they no longer state a count that drifts.
Tests
Two-branch merge simulation (scratch clone; A and B cut from one base, each adds one entry and updates the record exactly as the test asks, then
git merge):origin/mainassert 93 == 92,assert 109 == 108, vitestto have a length of 92 but got 93'all 43 of its tools' in …,assert 28 == 27summary rows parsed: 55does not state the catalogue size (40)Negative controls on this branch (entry added, record not updated): all 7 FAILED. Two of them pass on
mainonce the count alone is bumped: a constant written in a form the strict regex does not read, and a strict tool written into the "Sidebar shape" bullet.On this branch rebased onto current
main: the changed Python test files plustest/test_flake_pattern_ratchet.py, 827 passed, 50 skipped;gatewayNotice.test.tsx147 passed;./scripts/docs-lint.sh, black and flake8 clean.Proof: not provable -- test-only change
Manual verification
N/A — tests and doc wording only; the simulation above is the behavioral check.
Related Issues
Part of #37656
Checklist