Repository navigation
spec-guards: published figures go red on every spec merge -- lag is not a defect of the change #7174
Description
spec-guards on master is red again: python3 tools/published_figures.py --check fails whenever any merge adds or removes spec text. At 670480b four figures drifted (x.len() 1371 -> 1364, x.len 2149 -> 2160, a::b::c 631 -> 635, test blocks 15725 -> 15835). The pins were re-set by hand three times in two days (#6899, #6894, #6969), and each re-pin was stale before its own CI finished: the Actions queue is hours long and many lanes merge specs in parallel. A pure re-pin PR cannot win that race.
Why the gate exists
The file keeps figures quoted in published issues (#3479, #3489, #3495, #3497, #3501, #3473) reproducible: every figure carries its matcher, and --check re-derives it. It caught two real things: #3482 deleting 188 duplicate tests, and published sentences whose matcher asked a different question from the sentence (len( 142 was a diagnostic count). Neither of those is "another lane merged a spec".
The rule already exists
specs/ci/derived_data.t27 (#6601) is the decision rule for checked-in derived data: a PR is red only for a regression it causes or for a derived value it wrote that the tool would not write; lag is PENDING, a PR that moves a record gets a REPORT, and on master the record is REFRESHed. A pinned figure is derived data with two inputs, the spec corpus and the matcher.
Plan
specs/ci/derived_data.t27: addKIND_PUBLISHED_FIGUREwith inputsIN_SPEC + IN_MATCHER, andfigure_regression(): a matcher that changed meaning with its pin left alone is a regression (red). Tests and negative controls for each arm.tools/published_figures.py --check: I/O only. It compiles the spec'st27c gen-coutput and asksverdict()per figure. It compares against the commit before the change (PR: the merge ref's first parent; master: the previous commit). Red: a matcher whose count on the same corpus differs from the base version of the tool, unless the pin was restated; or a hand pin that equals neither the count at the merge ref nor at the PR head. Not red: lag from other merges (PENDING), drift from this change set (REPORT, annotated), drift on master (REFRESH, a warning and a job summary).tools/published_figures.py --bless --ref '#N': rewrites drifted pins and appends a per-merge trail computed fromgit diffof each first-parent commit since the table last changed, in the existing note style..github/workflows/spec-guards.yml: checkoutfetch-depth: 2so the base commit is present.- Re-pin the drifted figures to current master with the trail.
- Negative control: a branch-local matcher edit still fails
--check.
Foreign paths touched (owner standing approval, label owner-approved-foreign): tools/published_figures.py, .github/workflows/spec-guards.yml, listed in tools/policy/foreign-exceptions.txt citing this issue.
Activity
tools(published_figures): re-pin after merging master; bless names th…
tools(published_figures): a pin this branch wrote is judged where it …
tools(published_figures): re-pin test blocks to master fd28018; bless…
Merge master into fix/published-figures-drift (#7174)
ci(spec-guards): bless published figures at master f4317a9 (#7174)
ci(spec-guards): bless published figures at master 3c2c23f (#7174)
ci(spec-guards): bless published figures at master 5a3485e (#7174)
ci(spec-guards): bless published figures at master a91d7a4 (#7174)
ci(spec-guards): published figures are red only for a defect of the c…