Skip to content

spec-guards: published figures go red on every spec merge -- lag is not a defect of the change #7174

Closed

Description

@gHashTag

spec-guards on master is red again: python3 tools/published_figures.py --check fails whenever any merge adds or removes spec text. At 670480b four figures drifted (x.len() 1371 -> 1364, x.len 2149 -> 2160, a::b::c 631 -> 635, test blocks 15725 -> 15835). The pins were re-set by hand three times in two days (#6899, #6894, #6969), and each re-pin was stale before its own CI finished: the Actions queue is hours long and many lanes merge specs in parallel. A pure re-pin PR cannot win that race.

Why the gate exists

The file keeps figures quoted in published issues (#3479, #3489, #3495, #3497, #3501, #3473) reproducible: every figure carries its matcher, and --check re-derives it. It caught two real things: #3482 deleting 188 duplicate tests, and published sentences whose matcher asked a different question from the sentence (len( 142 was a diagnostic count). Neither of those is "another lane merged a spec".

The rule already exists

specs/ci/derived_data.t27 (#6601) is the decision rule for checked-in derived data: a PR is red only for a regression it causes or for a derived value it wrote that the tool would not write; lag is PENDING, a PR that moves a record gets a REPORT, and on master the record is REFRESHed. A pinned figure is derived data with two inputs, the spec corpus and the matcher.

Plan

  • specs/ci/derived_data.t27: add KIND_PUBLISHED_FIGURE with inputs IN_SPEC + IN_MATCHER, and figure_regression(): a matcher that changed meaning with its pin left alone is a regression (red). Tests and negative controls for each arm.
  • tools/published_figures.py --check: I/O only. It compiles the spec's t27c gen-c output and asks verdict() per figure. It compares against the commit before the change (PR: the merge ref's first parent; master: the previous commit). Red: a matcher whose count on the same corpus differs from the base version of the tool, unless the pin was restated; or a hand pin that equals neither the count at the merge ref nor at the PR head. Not red: lag from other merges (PENDING), drift from this change set (REPORT, annotated), drift on master (REFRESH, a warning and a job summary).
  • tools/published_figures.py --bless --ref '#N': rewrites drifted pins and appends a per-merge trail computed from git diff of each first-parent commit since the table last changed, in the existing note style.
  • .github/workflows/spec-guards.yml: checkout fetch-depth: 2 so the base commit is present.
  • Re-pin the drifted figures to current master with the trail.
  • Negative control: a branch-local matcher edit still fails --check.

Foreign paths touched (owner standing approval, label owner-approved-foreign): tools/published_figures.py, .github/workflows/spec-guards.yml, listed in tools/policy/foreign-exceptions.txt citing this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    owner-approved-foreignOwner-approved exception to the only-t27 rule: hand-written foreign code allowed in this PR

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions