Skip to content

test: store only per-file entries in the error-code and i18n baselines - #37727

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/merge-safe-gates
Nov 16, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/merge-safe-gates

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The error-code and i18n baselines stay correct on main when two pull requests that were each green on the same base both merge.

error-code-baseline.json stored whole-tree totals (_totals, _compliant) beside its per-file map. Two PRs that each fix a site in a different file rewrite those totals to the same value, git merges the identical edit cleanly, and the merged record is wrong. test/test_error_code_contract.py::test_baseline_totals_match_the_per_file_map then turned main red. Both keys were re-pinned 19 times on main in 30 days. The i18n baselines carry the same kind of _total (re-pinned 6 times in untranslated-baseline.json); no check reads it, so merges left it silently wrong.

Why it matters

A red main fails every open PR's CI until someone re-pins the total, and a silently wrong stored total misleads the next reader.

Not a goal

  • No product code changes; a test, three i18n gate scripts and their baselines.
  • No loosening: the per-file ceilings that decide every gate are unchanged.
  • The RFC index table in docs/request-for-change/README.md (one shared table every RFC PR inserts into, which has lost rows in conflict resolution, e.g. PR #36231) is left for its own change: making it merge-safe rewrites the whole table and conflicts with every open RFC PR.
  • untranslated-strict-baseline.json keeps its _total: that number is the whole record and only reports.

What changed (motivation → approach → change)

File Why it went wrong after a merge Change
test/test_error_code_contract.py + error-code-baseline.json _totals and _compliant are shared lines two PRs both rewrite; one test compared _totals with the per-file map Store only {"_comment", "files"}. Totals are computed from files and printed by --update. The shape test refuses any other key and an empty files. The totals-vs-map test is deleted, since nothing stores a total to disagree
website/scripts/check-i18n-strings.mjs + website/src/i18n/untranslated-baseline.json _total was written and shown in the OK message, never checked Not written; the OK message sums base.files
website/scripts/check-i18n-keys.mjs + dynamic-keys-baseline.json, website/scripts/check-i18n-render.mjs + render-baseline.json same _total not written; the render gate computes it from its counts

The rule is the one in docs/system-specs/common/testing-conventions.md § Merge-safe ratchets and counts (PR #37660).

Backwards compatibility

Compatible: the baselines lose keys that only the deleted test read. Regenerate with python test/test_error_code_contract.py --update if an open PR conflicts on the deleted lines.

Tests

Two-branch merge simulation (scratch clone; A and B cut from one base, each fixes one site in a different file and updates the baseline as the test asks, then git merge):

Scenario origin/main this branch negative control (this branch)
error-code baseline clean merge, FAILED `_totals` disagrees with `files` - the baseline was hand-edited clean merge, 6 passed a new site without a code and no baseline update FAILED missing_code 13 -> 14; re-adding _totals FAILED only per-file entries are stored
untranslated baseline clean merge, stored _total: 331 sum(files): 330 (silently wrong) clean merge, stored _total: (none) sum(files): 330 —

On this branch rebased onto current main: test/test_error_code_contract.py and test/test_flake_pattern_ratchet.py, 74 passed; check-i18n-strings.mjs and check-i18n-keys.mjs exit 0. check-i18n-render.mjs needs a dev build and runs in CI.

Full backend suite, the way CI runs it (8 file shards, -n 4, --no-cov, -m "not ipv6_required and not real_venv"), on one DevSpace, first on main d470c46 and then on this branch (2df3f95, before the final rebase):

Tree Failed Passed (sum of shards)
main d470c46 57 281,351
this branch 56 281,351

Every test that failed on this branch also failed on main; none is new. The one main-only failure, test/test_session_token_identity.py::test_native_settings_sources_repeated_permissive_reads_do_not_hide_a_restriction, passed here. The shared 56 are host-environment failures of that machine, clustered in test_dashboard_templates_forgotten_footer.py (15), test_cli_stt_memory_doctor.py (10), test_local_gate.py (7), test_dashboard_templates_builtin.py (7) and 11 other files; none of those files reads a file this PR changes. After rebasing onto main bb611ba: test/test_error_code_contract.py and test/test_flake_pattern_ratchet.py, 77 passed; check-i18n-strings.mjs and check-i18n-keys.mjs exit 0.

Proof: not provable -- test-only change

Manual verification

N/A — test and gate scripts only; the simulation above is the behavioral check.

Related Issues

Part of #37656

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 16, 2026 00:08
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 7909b1f91e6135d7817245f2992b4c619ef8caf9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff only changes test and i18n-gate baselines and the build scripts that read them, so no user sees anything new. With no visible change, the missing blind read leaves no evidence gap.

[UX-REVIEWED] 7909b1f

Description read

[DESCRIPTION-READ] a9239a574127b4fe606e1e3d74c9768a24b5a7cccaf8ec8a2bee3a1c15b78025

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 7909b1f91e6135d7817245f2992b4c619ef8caf9 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every removed stored total was written by its tool but never checked, and the testing-conventions "Merge-safe ratchets and counts" rule requires dropping it.

What this change ships

Inventory (6 items) — 6 justified

Intent: keep the ratchet baselines correct on main after two separately green PRs both merge. This is a FIX.

  1. error-code-baseline.json no longer stores _totals or _compliant; --update prints them instead — justified
  2. The error-code shape test now rejects any key other than _comment/files, and rejects an empty files — justified
  3. The totals-vs-files test is deleted, and the description says so and why — justified
  4. The i18n strings gate stops writing _total; its OK message sums files — justified
  5. The dynamic-keys baseline stops writing _total — justified
  6. The render ledger stops writing _total; the update message computes it — justified

[FIRST-PRINCIPLES-REVIEWED] 7909b1f

Description read

[DESCRIPTION-READ] ebb13b8d67e140bfb9fd6e9bd6529a38858b812cf4a5c81911311054e21dca08

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7909b1f91e6135d7817245f2992b4c619ef8caf9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns. Nothing outside the patch reads the removed _totals, _compliant or _total keys. The deleted test only checked that a stored total matched the per-file map, and no total is stored any more.

[DESIGN-REVIEWED] 7909b1f

Description read

[DESCRIPTION-READ] 8e09bea7c8ce5224b63d7c8b87f0a52a273532dd97e1590fd5a5dfc360967ae6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7909b1f91e6135d7817245f2992b4c619ef8caf9 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7909b1f

Verdict parsed from the review's SHA-scoped output markers for commit 7909b1f91e6135d7817245f2992b4c619ef8caf9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7909b1f91e6135d7817245f2992b4c619ef8caf9 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7909b1f91e6135d7817245f2992b4c619ef8caf9 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7909b1f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7909b1f91e6135d7817245f2992b4c619ef8caf9 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

This removes the stored whole-tree totals (_totals, _compliant, _total) from the error-code and i18n baselines and computes them from the per-file entries wherever they are shown. A repo-wide search found no remaining reader of the removed keys. Every other baseline reader (the composition tests, i18n-translate.mjs, render-verdict.mjs, the check-i18n-keys/strings gate loops) already uses only files/surfaces. The strict baseline keeps its _total, as stated. The new shape test fails if a stored total comes back, and the per-file ratchet and staleness tests are unchanged. CI is green.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 7909b1f91e · 79 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 16, 2026 02:07

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On this head the review confirmed that nothing still reads the removed _totals, _compliant or _total keys in the gate scripts, tests, CI or docs. The printed totals now come from the per-file map and give the same numbers as before (332 for strings; the error-code totals still match the sum of files). The per-file ceilings, the stale check and the ratchet are unchanged.

Well done: the PR removes the shared total line instead of adding a guard. The new exact key-set check in test_baseline_file_is_parseable_and_shaped makes a merge that brings a total back fail loudly, and its message names the regenerate command.

Optional:

  • test/test_error_code_contract.py:291: the added and baseline["files"] fails with no message once every bucket reaches zero, and --update cannot fix that. Keeping only the isinstance check is enough, because the key-set assert above it already pins the shape.
  • website/src/i18n/keyReference.test.ts:155: the fixture still writes _total: 0. Nothing reads it, but { files: ... } would match the shape the writer now produces.
  • The three i18n .mjs gates have no key-set check like the Python one, so a future writer change could store _total again without any test going red.

@KiroCrewCaptain
KiroCrewCaptain merged commit 2307d4f into main Nov 16, 2026
153 of 155 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/linuxflackyguard/merge-safe-gates branch November 16, 2026 02:07
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants