Skip to content

test: record call-site ratchets per file, code and name so two green PRs merge green - #37682

Merged
KiroCrewCaptain merged 2 commits into
mainfrom
crew/linuxflackyguard/merge-safe-call-site-ratchets
Nov 16, 2026
Merged

KiroCrewCaptain merged 2 commits into
mainfrom
crew/linuxflackyguard/merge-safe-call-site-ratchets

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The call-site ratchets stay green on main when two pull requests that were each green on the same base both merge.

Each of these tests kept one shared total of call sites across the tree. Two PRs that each add (or each remove) a site in a different file rewrite that total to the same value, git merges the identical edit cleanly, and main holds a different count than the record says. In 30 days on main the totals in test_sandbox_protected_name_holds.py were re-pinned 5 times, test_chat_turn_timeout_consistency.py 4 times and test_ask_question_roundtrip.py 3 times; BYPASS_BASELINE in test_kill_chokepoint_ratchet.py 11 times in 90 days.

Why it matters

A red main fails every open PR's CI until someone re-pins the number.

Not a goal

  • No product code changes; tests and one spec reference only.
  • No loosening: every record stays exact per key. test_sandbox_protected_name_holds.py gets stricter (it compared totals with <=; it now matches each name, so a swap is caught).
  • Already merge-safe and unchanged: test_slack_post_message_call_sites.py (per module), test_core_path_redact_before_bound.py (a >= floor per package), test_x_feature_captures.py (a <= ceiling that only falls).

What changed (motivation → approach → change)

Test Before After
test/test_chat_turn_timeout_consistency.py assert total == 4 _DISPATCH_SITES, one line per file
test/test_ask_question_roundtrip.py len(coded) == 37 CODED_SITES, one line per code
test/test_sandbox_protected_name_holds.py three EXPECTED totals, <= LEAF_ONLY, one line per name (paths normalized: <crew>, ~, <isolated>); value is (spellings, tiers)
test/test_kill_chokepoint_ratchet.py BYPASS_BASELINE = 63 BYPASS_SITES, one line per module
test/test_fd_census_ratchet.py _BASELINE_SITES = 1 one line per file
test/test_http_server_bind_ratchet.py _BASELINE_SITES = 1 one line per file

Each failure message prints the line to write, e.g. "<key>": N, # was M. docs/system-specs/modules/runtime-ownership.md names BYPASS_SITES instead of BYPASS_BASELINE (2 places).

The rule is the one in docs/system-specs/common/testing-conventions.md § Merge-safe ratchets and counts (PR #37660).

Backwards compatibility

Compatible: tests and a spec name reference only.

Removes nothing: each total is replaced by an exact per-key record whose sum is the old total.

Tests

Two-branch merge simulation (scratch clone; A and B cut from one base, each changes one site in a different file and updates the record exactly as the test asks, then git merge):

Scenario origin/main this branch
chat-turn dispatch sites clean merge, FAILED Expected 5 helper-visible _run_chat dispatch sites, found 6. 7 passed
ask_question coded sites clean merge, FAILED scanner reached 39 coded sites, expected 38 passed
sandbox protected names clean merge, FAILED strict now holds 377 ... up from 374 (all three tiers) 25 passed
kill chokepoint (A and B each remove a site) clean merge, FAILED the real count is 61 but BYPASS_BASELINE says 62 16 passed
fd census clean merge, FAILED 3 test site(s) ... (recorded: 2) 2 passed
http server bind clean merge, FAILED 3 bare HTTPServer(...) construction(s) (recorded: 2) 2 passed

Negative control on this branch (site added, record not updated): all 6 scenarios FAILED, each message naming the line to add.

On this branch rebased onto current main: the 6 changed test files plus test/test_flake_pattern_ratchet.py and test/test_ratchet_scope.py, 241 passed; black, isort and flake8 clean. Not run on macOS: the sandbox record has no platform branch in the product code, and its path normalization covers /var-style symlinks.

Proof: not provable -- test-only change

Manual verification

N/A — tests only; the simulation above is the behavioral check.

Related Issues

Part of #37656

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 19:49
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision c2bf589545f898b18425e3b8618e3ef6d0940e2f ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision c2bf589545f898b18425e3b8618e3ef6d0940e2f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of c2bf589545f898b18425e3b8618e3ef6d0940e2f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only change, aimed at a real merge-skew harm, and it uses the merge-safe ratchet rule from testing-conventions.

Notes

  • LEAF_ONLY now matches each name exactly, where the old check allowed a lower total (<=). So a host whose namespace plan yields fewer names, such as the ~/… credential homes or macOS /var-style roots, now fails, while the old totals tolerated it. The description says "Not run on macOS"; one macOS run of test_leaf_only_names_match_the_record would settle it.
  • Every leaf removal now has to edit LEAF_ONLY, including the app leaves (<crew>/apps/...) that leave core under Move every built-in app out to KiroCrewApps and forbid built-in apps #26967. This is intended, but those PRs will need the line deleted.

[DESIGN-REVIEWED] c2bf589

Description read

[DESCRIPTION-READ] 922183b953f6e4a1edaf0d89a14b9239bcf90081189a5529a45f8250433a812b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed c2bf589545f898b18425e3b8618e3ef6d0940e2f — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • Six new comments cite "testing-conventions.md § Merge-safe ratchets and counts", but no doc has that section. The pointer leads nowhere; it doesn't change what the tests do.

[OPUS-REVIEWED] c2bf589

Verdict parsed from the review's SHA-scoped output markers for commit c2bf589545f898b18425e3b8618e3ef6d0940e2f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c2bf589545f898b18425e3b8618e3ef6d0940e2f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of c2bf589545f898b18425e3b8618e3ef6d0940e2f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c2bf589

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c2bf589545f898b18425e3b8618e3ef6d0940e2f false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/merge-safe-call-site-ratchets branch from 07e6072 to 2549baf Compare November 15, 2026 21:29
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 8 commits (d470c46 -> 4dec84c) touching docs/system-specs/modules/ beside docs/system-specs/modules/runtime-ownership.md that this PR edits; rebased cleanly. Re-ran the 6 changed test files plus test_flake_pattern_ratchet.py on the new base: 193 passed. New head: 2549baf.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 15, 2026
@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/merge-safe-call-site-ratchets branch from 2549baf to 367a7f1 Compare November 15, 2026 22:26
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 10 commits (4dec84c -> 5c88a0b) touching docs/system-specs/modules/ beside docs/system-specs/modules/runtime-ownership.md that this PR edits; rebased cleanly. Re-ran the 6 changed test files plus test_flake_pattern_ratchet.py on the new base: 193 passed. New head: 367a7f1.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 11 commits (d4f0a82 -> 4372f48) touching docs/system-specs/modules/ beside docs/system-specs/modules/runtime-ownership.md that this PR edits; rebased cleanly. Re-ran the 6 changed test files plus test_flake_pattern_ratchet.py on the new base: 193 passed. New head: c2bf589.

@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/merge-safe-call-site-ratchets branch from 367a7f1 to c2bf589 Compare November 15, 2026 23:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 15, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 594 lines, more than the 400 it may approve on its own.

Test-only change that replaces six single-total call-site ratchets with exact per-key records, so two PRs that were each green on the same base also merge green. Verified: BYPASS_SITES sums to 63 (the old BYPASS_BASELINE); CODED_SITES sums to 37; LEAF_ONLY gives 363/370/371 per tier, the old EXPECTED; the chat-turn, fd-census and http-bind records match the scanned files. Every check is at least as strict as before. The kill ratchet's grow and stale checks still force exact equality, now per module. The sandbox test moves from a <= total to an exact match per name. Record keys use as_posix() so they match on Windows. No remaining references to BYPASS_BASELINE or EXPECTED; the two runtime-ownership.md mentions are updated. CI is green on Linux and Windows.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit c2bf589545 · 594 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head c2bf589 and ran the six changed test files twice: once on the head, and once with current main merged in, serially and with an isolated data home. Every test passed. As a negative control I edited the sandbox leaf record and the kill bypass record, and the matching tests failed with the line to write, so the new per-file, per-code and per-name records are not vacuous. Each record sums to the total it replaces, and the exact match is at least as strict as the old check. The cited testing-conventions section exists on main. CI is green on Linux and Windows, and nothing here blocks.

@KiroCrewCaptain
KiroCrewCaptain merged commit 592d6e2 into main Nov 16, 2026
97 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/linuxflackyguard/merge-safe-call-site-ratchets branch November 16, 2026 01:21
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants