Repository navigation
test: record call-site ratchets per file, code and name so two green PRs merge green - #37682
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a test-only change, aimed at a real merge-skew harm, and it uses the merge-safe ratchet rule from testing-conventions. Notes
[DESIGN-REVIEWED] c2bf589 Description read[DESCRIPTION-READ] 922183b953f6e4a1edaf0d89a14b9239bcf90081189a5529a45f8250433a812b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] c2bf589 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
07e6072 to
2549baf
Compare
2549baf to
367a7f1
Compare
367a7f1 to
c2bf589
Compare
Code change approver recommendation: A person should decideWhy: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 594 lines, more than the 400 it may approve on its own. Test-only change that replaces six single-total call-site ratchets with exact per-key records, so two PRs that were each green on the same base also merge green. Verified: BYPASS_SITES sums to 63 (the old BYPASS_BASELINE); CODED_SITES sums to 37; LEAF_ONLY gives 363/370/371 per tier, the old EXPECTED; the chat-turn, fd-census and http-bind records match the scanned files. Every check is at least as strict as before. The kill ratchet's grow and stale checks still force exact equality, now per module. The sandbox test moves from a An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
I reviewed head c2bf589 and ran the six changed test files twice: once on the head, and once with current main merged in, serially and with an isolated data home. Every test passed. As a negative control I edited the sandbox leaf record and the kill bypass record, and the matching tests failed with the line to write, so the new per-file, per-code and per-name records are not vacuous. Each record sums to the total it replaces, and the exact match is at least as strict as the old check. The cited testing-conventions section exists on main. CI is green on Linux and Windows, and nothing here blocks.
Problem / Motivation
Goal: The call-site ratchets stay green on
mainwhen two pull requests that were each green on the same base both merge.Each of these tests kept one shared total of call sites across the tree. Two PRs that each add (or each remove) a site in a different file rewrite that total to the same value, git merges the identical edit cleanly, and
mainholds a different count than the record says. In 30 days onmainthe totals intest_sandbox_protected_name_holds.pywere re-pinned 5 times,test_chat_turn_timeout_consistency.py4 times andtest_ask_question_roundtrip.py3 times;BYPASS_BASELINEintest_kill_chokepoint_ratchet.py11 times in 90 days.Why it matters
A red
mainfails every open PR's CI until someone re-pins the number.Not a goal
test_sandbox_protected_name_holds.pygets stricter (it compared totals with<=; it now matches each name, so a swap is caught).test_slack_post_message_call_sites.py(per module),test_core_path_redact_before_bound.py(a>=floor per package),test_x_feature_captures.py(a<=ceiling that only falls).What changed (motivation → approach → change)
test/test_chat_turn_timeout_consistency.pyassert total == 4_DISPATCH_SITES, one line per filetest/test_ask_question_roundtrip.pylen(coded) == 37CODED_SITES, one line per codetest/test_sandbox_protected_name_holds.pyEXPECTEDtotals,<=LEAF_ONLY, one line per name (paths normalized:<crew>,~,<isolated>); value is(spellings, tiers)test/test_kill_chokepoint_ratchet.pyBYPASS_BASELINE = 63BYPASS_SITES, one line per moduletest/test_fd_census_ratchet.py_BASELINE_SITES = 1test/test_http_server_bind_ratchet.py_BASELINE_SITES = 1Each failure message prints the line to write, e.g.
"<key>": N, # was M.docs/system-specs/modules/runtime-ownership.mdnamesBYPASS_SITESinstead ofBYPASS_BASELINE(2 places).The rule is the one in
docs/system-specs/common/testing-conventions.md§ Merge-safe ratchets and counts (PR #37660).Backwards compatibility
Compatible: tests and a spec name reference only.
Removes nothing: each total is replaced by an exact per-key record whose sum is the old total.
Tests
Two-branch merge simulation (scratch clone; A and B cut from one base, each changes one site in a different file and updates the record exactly as the test asks, then
git merge):origin/mainExpected 5 helper-visible _run_chat dispatch sites, found 6.scanner reached 39 coded sites, expected 38strict now holds 377 ... up from 374(all three tiers)the real count is 61 but BYPASS_BASELINE says 623 test site(s) ... (recorded: 2)3 bare HTTPServer(...) construction(s) (recorded: 2)Negative control on this branch (site added, record not updated): all 6 scenarios FAILED, each message naming the line to add.
On this branch rebased onto current
main: the 6 changed test files plustest/test_flake_pattern_ratchet.pyandtest/test_ratchet_scope.py, 241 passed; black, isort and flake8 clean. Not run on macOS: the sandbox record has no platform branch in the product code, and its path normalization covers/var-style symlinks.Proof: not provable -- test-only change
Manual verification
N/A — tests only; the simulation above is the behavioral check.
Related Issues
Part of #37656
Checklist