Repository navigation
test: store only per-file entries in the error-code and i18n baselines - #37727
Conversation
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. The diff only changes test and i18n-gate baselines and the build scripts that read them, so no user sees anything new. With no visible change, the missing blind read leaves no evidence gap. [UX-REVIEWED] 7909b1f Description read[DESCRIPTION-READ] a9239a574127b4fe606e1e3d74c9768a24b5a7cccaf8ec8a2bee3a1c15b78025 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Every removed stored total was written by its tool but never checked, and the testing-conventions "Merge-safe ratchets and counts" rule requires dropping it. What this change shipsInventory (6 items) — 6 justifiedIntent: keep the ratchet baselines correct on
[FIRST-PRINCIPLES-REVIEWED] 7909b1f Description read[DESCRIPTION-READ] ebb13b8d67e140bfb9fd6e9bd6529a38858b812cf4a5c81911311054e21dca08 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns. Nothing outside the patch reads the removed [DESIGN-REVIEWED] 7909b1f Description read[DESCRIPTION-READ] 8e09bea7c8ce5224b63d7c8b87f0a52a273532dd97e1590fd5a5dfc360967ae6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. This removes the stored whole-tree totals ( An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved. On this head the review confirmed that nothing still reads the removed _totals, _compliant or _total keys in the gate scripts, tests, CI or docs. The printed totals now come from the per-file map and give the same numbers as before (332 for strings; the error-code totals still match the sum of files). The per-file ceilings, the stale check and the ratchet are unchanged.
Well done: the PR removes the shared total line instead of adding a guard. The new exact key-set check in test_baseline_file_is_parseable_and_shaped makes a merge that brings a total back fail loudly, and its message names the regenerate command.
Optional:
test/test_error_code_contract.py:291: the addedand baseline["files"]fails with no message once every bucket reaches zero, and--updatecannot fix that. Keeping only theisinstancecheck is enough, because the key-set assert above it already pins the shape.website/src/i18n/keyReference.test.ts:155: the fixture still writes_total: 0. Nothing reads it, but{ files: ... }would match the shape the writer now produces.- The three i18n
.mjsgates have no key-set check like the Python one, so a future writer change could store_totalagain without any test going red.
Problem / Motivation
Goal: The error-code and i18n baselines stay correct on
mainwhen two pull requests that were each green on the same base both merge.error-code-baseline.jsonstored whole-tree totals (_totals,_compliant) beside its per-file map. Two PRs that each fix a site in a different file rewrite those totals to the same value, git merges the identical edit cleanly, and the merged record is wrong.test/test_error_code_contract.py::test_baseline_totals_match_the_per_file_mapthen turnedmainred. Both keys were re-pinned 19 times onmainin 30 days. The i18n baselines carry the same kind of_total(re-pinned 6 times inuntranslated-baseline.json); no check reads it, so merges left it silently wrong.Why it matters
A red
mainfails every open PR's CI until someone re-pins the total, and a silently wrong stored total misleads the next reader.Not a goal
docs/request-for-change/README.md(one shared table every RFC PR inserts into, which has lost rows in conflict resolution, e.g. PR #36231) is left for its own change: making it merge-safe rewrites the whole table and conflicts with every open RFC PR.untranslated-strict-baseline.jsonkeeps its_total: that number is the whole record and only reports.What changed (motivation → approach → change)
test/test_error_code_contract.py+error-code-baseline.json_totalsand_compliantare shared lines two PRs both rewrite; one test compared_totalswith the per-file map{"_comment", "files"}. Totals are computed fromfilesand printed by--update. The shape test refuses any other key and an emptyfiles. The totals-vs-map test is deleted, since nothing stores a total to disagreewebsite/scripts/check-i18n-strings.mjs+website/src/i18n/untranslated-baseline.json_totalwas written and shown in the OK message, never checkedbase.fileswebsite/scripts/check-i18n-keys.mjs+dynamic-keys-baseline.json,website/scripts/check-i18n-render.mjs+render-baseline.json_totalnot written; the render gate computes it from its countsThe rule is the one in
docs/system-specs/common/testing-conventions.md§ Merge-safe ratchets and counts (PR #37660).Backwards compatibility
Compatible: the baselines lose keys that only the deleted test read. Regenerate with
python test/test_error_code_contract.py --updateif an open PR conflicts on the deleted lines.Tests
Two-branch merge simulation (scratch clone; A and B cut from one base, each fixes one site in a different file and updates the baseline as the test asks, then
git merge):origin/main`_totals` disagrees with `files` - the baseline was hand-editedcodeand no baseline update FAILEDmissing_code 13 -> 14; re-adding_totalsFAILEDonly per-file entries are storedstored _total: 331 sum(files): 330(silently wrong)stored _total: (none) sum(files): 330On this branch rebased onto current
main:test/test_error_code_contract.pyandtest/test_flake_pattern_ratchet.py, 74 passed;check-i18n-strings.mjsandcheck-i18n-keys.mjsexit 0.check-i18n-render.mjsneeds a dev build and runs in CI.Full backend suite, the way CI runs it (8 file shards,
-n 4,--no-cov,-m "not ipv6_required and not real_venv"), on one DevSpace, first onmaind470c46 and then on this branch (2df3f95, before the final rebase):maind470c46Every test that failed on this branch also failed on
main; none is new. The onemain-only failure,test/test_session_token_identity.py::test_native_settings_sources_repeated_permissive_reads_do_not_hide_a_restriction, passed here. The shared 56 are host-environment failures of that machine, clustered intest_dashboard_templates_forgotten_footer.py(15),test_cli_stt_memory_doctor.py(10),test_local_gate.py(7),test_dashboard_templates_builtin.py(7) and 11 other files; none of those files reads a file this PR changes. After rebasing ontomainbb611ba:test/test_error_code_contract.pyandtest/test_flake_pattern_ratchet.py, 77 passed;check-i18n-strings.mjsandcheck-i18n-keys.mjsexit 0.Proof: not provable -- test-only change
Manual verification
N/A — test and gate scripts only; the simulation above is the behavioral check.
Related Issues
Part of #37656
Checklist