Skip to content

test: derive notice, session-tool, crew-log and provider counts instead of pinning totals - #37685

Merged
KiroCrewCaptain merged 2 commits into
mainfrom
crew/linuxflackyguard/merge-safe-parity-counts
Nov 15, 2026
Merged

KiroCrewCaptain merged 2 commits into
mainfrom
crew/linuxflackyguard/merge-safe-parity-counts

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The notice-parity, session-tool, crew-log type and provider-catalogue tests stay green on main when two pull requests that were each green on the same base both merge.

Each pinned one shared total, in a test or in the doc a test reads. Two PRs that each add one entry rewrite that number to the same value, git merges the identical edit cleanly, and main holds one entry more than the number says. On main in 30 days: the transient-notice token totals were re-pinned 11, 11, 5, 11 and 10 times; "41" tools in session-control.md 7 times; EXPECTED_LIVE_TYPES = 53 6 times; the "38" providers in connections.md 4 times.

Why it matters

A red main fails every open PR's CI until someone re-pins the number, and the re-pin races the next one.

Not a goal

  • No product code changes; tests and the user docs the tests read.
  • No loosening: every replaced total becomes an exact set or per-item record that is stricter than the count (see negative controls).
  • Counts that sit on the same line as the names they count ("Seven entries: …") stay: two edits there conflict instead of merging wrong.
  • src/kiro_crew/docs/ui-index.generated.json's coverage line (84 re-pins in 30 days) is a generated file and gets its own change.

What changed (motivation → approach → change)

Area Before After
Transient notices: test/test_transient_notice_parity.py, website/src/test/gatewayNotice.test.tsx totals 91 / 91 / 81, toHaveLength(91), len(copy) == 107, len(texts) == 13 / 16; len(sites) == 36 The constant names the strict scan reads must equal every column-0 assignment in the source; the LEGACY entry count must equal the re: occurrences in the table; _TOKENIZED_SITES lists one site per line
Session tools: test/test_session_control_doc.py, src/kiro_crew/docs/session-control.md, src/kiro_crew/docs/index.md "41" in two docs; len(SESSION_CONTROL_TOOLS) == 26 Docs state no number, and a check refuses one coming back; the tool names in the doc's two bullets must equal the two tool groups in code
Crew-log types: test/test_docs_crew_log_reference.py, test/test_crew_log_types.py EXPECTED_LIVE_TYPES = 53, len(SESSION_ENTRY_TYPES) == 53 Deleted; the exact set comparisons beside them stay
Providers: test/test_connections_doc.py, src/kiro_crew/docs/connections.md "38", "15", 38-provider in index.md, "eleven" aliases Docs state no number; the Connect plus Set-up rows must equal get_visible_providers(). The old count-word test only knew words up to eleven, so the twelfth aliased provider would have failed its own PR

The rule is the one in docs/system-specs/common/testing-conventions.md § Merge-safe ratchets and counts (PR #37660). Two of these totals (len(sites) == 36, len(SESSION_ENTRY_TYPES) == 53) are among the sites that PR's K21 guard counts.

Backwards compatibility

Compatible: tests and doc wording only. The docs still name every tool and provider; they no longer state a count that drifts.

Tests

Two-branch merge simulation (scratch clone; A and B cut from one base, each adds one entry and updates the record exactly as the test asks, then git merge):

Scenario origin/main this branch
notice tokens clean merge, FAILED assert 93 == 92, assert 109 == 108, vitest to have a length of 92 but got 93 19 passed, vitest 149 passed
session tools clean merge, FAILED 'all 43 of its tools' in …, assert 28 == 27 10 passed
crew-log types clean merge, FAILED summary rows parsed: 55 417 passed, 52 skipped
providers clean merge, FAILED does not state the catalogue size (40) 16 passed

Negative controls on this branch (entry added, record not updated): all 7 FAILED. Two of them pass on main once the count alone is bumped: a constant written in a form the strict regex does not read, and a strict tool written into the "Sidebar shape" bullet.

On this branch rebased onto current main: the changed Python test files plus test/test_flake_pattern_ratchet.py, 827 passed, 50 skipped; gatewayNotice.test.tsx 147 passed; ./scripts/docs-lint.sh, black and flake8 clean.

Proof: not provable -- test-only change

Manual verification

N/A — tests and doc wording only; the simulation above is the behavioral check.

Related Issues

Part of #37656

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…ad of pinning totals

Two PRs that each add one item edit a stored total to the same new number, git merges them cleanly, and main is red. Replace each total with the exact per-item record or set the test already compares, a looser parse of the same source, or prose that names no number.
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 20:08
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 42734bfff329a0567c84c0a723a6d54b13032e93 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns. Each pinned total is replaced by an exact set or per-site record, which is stricter than the count and safe when two PRs merge.

Notes

  • test_the_page_states_the_tool_count_it_documents had pinned the doc's count on purpose ("a count is the one claim a reader cannot check"). This PR removes the count rather than calling the old pin a gap, and the new name-set checks keep the doc tied to the server's tools. If a maintainer wants the count kept for readers, they need to say so on the PR.

[DESIGN-REVIEWED] 42734bf

Description read

[DESCRIPTION-READ] 77e95326f495b9f750d7bb6eee3230d97b75f16c8d843eb0041fd7ae233b27b4

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 42734bfff329a0567c84c0a723a6d54b13032e93 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff changes tests and doc wording only, and adds or changes no dashboard control, so no screenshot is needed.

[UX-REVIEWED] 42734bf

Description read

[DESCRIPTION-READ] 098b2598e2f9e740c7c9181cbb29429a2ab1ae106907427a818469de2a5f67fc

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 42734bfff329a0567c84c0a723a6d54b13032e93 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every replaced total either becomes an exact set check or is dropped beside one that already exists. No new product surface comes with it.

Not checked: authentic.patch also carries #37662's hooks.return_answer webhook hunks, which are already on base ace367bf.

Notes

  • Item 7 — inherited: five new test comments cite "testing-conventions.md § Merge-safe ratchets and counts". Grepping Merge-safe ratchets under docs/ finds 0 matches on base, because PR test: keep ratchet and count tests green when two PRs merge #37660 has not merged. Subtraction: delete those citation parentheticals, or land this after test: keep ratchet and count tests green when two PRs merge #37660.
  • Item 8 — rides along: Not a goal says "every replaced total becomes an exact set or per-item record that is stricter than the count". But assert len(copy) == 107 becomes assert copy, and assert len(texts) == 13 is deleted outright. Neither gets a set check in this file.

What this change ships

Inventory (8 items) — 6 justified

Intent: keep main green when two PRs that each add an entry merge cleanly with the same re-pinned count. This is a FIX.

  1. Notice token totals are replaced by a looser scan of every column-0 assignment, in Python and vitest — justified
  2. The count of 36 tokenized sites becomes a per-line _TOKENIZED_SITES list — justified
  3. Site paths are compared slash-spelled on every OS — justified
  4. session-control.md and index.md stop stating "41". A test refuses a count, and the two bullets must equal the code's tool groups — justified
  5. The crew-log 53 pins are deleted, and the summary rows must equal the documented types — justified
  6. connections.md and index.md stop stating provider counts. The card rows must equal get_visible_providers() — justified
  7. Test comments cite a spec section that is absent on base — inherited
  8. The gatewayNotice copy count and the legacy-text count are dropped with no set replacement — rides along

[FIRST-PRINCIPLES-REVIEWED] 42734bf

Description read

[DESCRIPTION-READ] 0ca677e038bf16f69cfeda66c56ea088780e874225787e6916f1eca25fd5c0ea

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 42734bfff329a0567c84c0a723a6d54b13032e93 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 42734bf

Verdict parsed from the review's SHA-scoped output markers for commit 42734bfff329a0567c84c0a723a6d54b13032e93.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 42734bfff329a0567c84c0a723a6d54b13032e93 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 42734bfff329a0567c84c0a723a6d54b13032e93 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 42734bf

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 42734bfff329a0567c84c0a723a6d54b13032e93 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 7% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only change (plus doc wording). Fixed totals in four test areas become exact set or per-line checks, so two PRs that each add an entry no longer merge into a red main. Checked: _backend_tokens() returns a name→token dict, so comparing its keys to the new loose scan is valid. The session-control bullets list 26 + 15 = 41 tools, matching the code's two groups. Connections' Connect and Set-up rows must now equal get_visible_providers(). The crew-log summary rows must equal the documented types. as_posix() keeps the site paths portable on Windows. No other reader of the removed doc counts exists in the repo. All CI and AI review lanes passed on this head.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 42734bfff3 · 241 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 42734bf. The changed backend test files pass on this head and again after merging current main. I also confirmed that the frontend token scan and the new loose assignment scan agree on the same 91 tokens. Every total that was removed now has an exact set or per-site check against a non-empty source in its place, so none of the new assertions is vacuous. Coverage holds where a count was simply dropped, because the remaining checks still pin those sets. This is a test and doc wording change with all CI green, and I found no bug that blocks it.

@KiroCrewCaptain
KiroCrewCaptain merged commit 686d00f into main Nov 15, 2026
97 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/linuxflackyguard/merge-safe-parity-counts branch November 15, 2026 21:49
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants