Skip to content

ci: gate what the corpus is worth, and retire a waiver's completed instruction - #49

Merged
fas89 merged 2 commits into
open-data-protocol:mainfrom
fas89:chore/coverage-gate-and-stale-waiver
Sep 13, 2026
Merged

fas89 merged 2 commits into
open-data-protocol:mainfrom
fas89:chore/coverage-gate-and-stale-waiver

Conversation

@fas89

@fas89 fas89 commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

Two things that had quietly stopped being true.

1. mutation_coverage.py ran nowhere

run.py answers "do the cases pass" — a corpus of zero cases answers that perfectly. mutation_coverage.py answers the harder question: delete a constraint from the schema and see whether any case notices. It has been in this repo, runnable, documented, and wired into nothing, so the figure could have fallen to zero between releases in silence.

The floor is the current number rounded down (40.7% → 40), not a target. That is where mutation-testing gates converge: a break threshold set where you actually are catches a regression on day one, whereas an aspirational one goes red immediately and gets deleted by day three — worse than no gate.

Deliberately not a ratchet, and the divergence is on purpose. Ratchets exist to absorb run-to-run noise; there is none here, because the run is deterministic — mutate the schema, replay the corpus, count. A fixed floor does the same work with less machinery, and raising it stays a deliberate edit made when a corpus wave earns it.

Proven to fail, three ways:

Condition --min-coverage 40
today's tree exit 0
floor raised to 41 exit 1
metadata group trimmed to a single case exit 1
group restored exit 0

The third is the one that matters — it is a real corpus regression, not a moved goalpost.

2. A waiver told the next reader to fix something already fixed

compat-waivers.txt said docs/releases/0.7.2.md "should be corrected to name this change". It was corrected, in #46 — the note now opens "One breaking change, corrected here after the fact". The instruction outlived the work.

The waiver line itself stays, permanently: 0.7.2 is published and its history cannot be rewritten, so the gate has to be told this break is known rather than new. The comment now says that, and says what a waiver must never become — a way to quiet the gate about a break nobody dealt with. This one was dealt with by correcting the release note, not by adding the line.

Borrow-before-build

Searched:
- mutation testing CI gate ratchet vs fixed threshold → "set break thresholds, not
  aspirational targets"; "a gate that reports a baseline on day one and is disabled
  by day three is worse than no gate"
- JSON-Schema-Test-Suite conformance coverage / mutation prior art → the official suite
  organises by keyword category (this corpus already mirrors that, 11 groups); Bowtie is
  the cross-implementation reporter (check_reference.py is the local equivalent).
  No project publishes a mutation-coverage GATE for a schema corpus.

Reuse strategy: adapt-the-pattern (break-threshold-at-current), diverge on the ratchet.

Corpus 480/480 and meta-test 19/19 unchanged.

🤖 Generated with Claude Code

fas89 and others added 2 commits September 13, 2026 20:21
…struction

Two things that had quietly stopped being true.

1. mutation_coverage.py ran nowhere.

`run.py` answers "do the cases pass" — a corpus of zero cases answers that
perfectly. `mutation_coverage.py` answers the harder question: delete a
constraint from the schema and see whether any case notices. It has been in
this repo, runnable, documented and wired into nothing, so the figure could
have fallen to zero between releases in silence.

The floor is the current number rounded down (40.7% -> 40), not a target.
That is where mutation-testing gates converge (Stryker, mutmut): a break
threshold set where you actually are catches a regression on day one, whereas
an aspirational one goes red immediately and gets deleted by day three, which
is worse than no gate.

Deliberately NOT a ratchet, and the divergence is on purpose. Ratchets exist
to absorb run-to-run noise; there is none here, because the run is
deterministic — mutate the schema, replay the corpus, count. A fixed floor
does the same work with less machinery, and raising it stays a deliberate edit
made when a corpus wave earns it.

Proven to fail, three ways: --min-coverage 41 exits 1 against today's tree;
trimming the metadata group to a single case drops it below 40 and turns the
gate red; restoring it goes green again.

2. A waiver told the next reader to fix something already fixed.

compat-waivers.txt said docs/releases/0.7.2.md "should be corrected to name
this change". It was corrected, in open-data-protocol#46 — the note now opens "One breaking
change, corrected here after the fact". The instruction outlived the work.

The waiver line itself stays, permanently: 0.7.2 is published and its history
cannot be rewritten, so the gate has to be told this break is known rather than
new. The comment now says that, and says what a waiver must never become — a
way to quiet the gate about a break nobody dealt with. This one was dealt with
by correcting the release note, not by adding the line.

Corpus 480/480 and meta-test 19/19 unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…g paths

A path-filtered check cannot be a REQUIRED check. On a pull request that does
not match the filter the context never reports at all, so branch protection
waits for a report that will never arrive and the pull request hangs
permanently, with nothing to click.

That is why main still requires no status checks: turning the existing three on
as they were would have bricked the next docs PR rather than protected
anything.

Not hypothetical here. PR open-data-protocol#47 was a docs change and received these three checks
only because it happened to also touch conformance/run.py. PR open-data-protocol#48 — LICENSE
and CONTRIBUTING.md — reported no checks at all and merged with nothing having
run.

The push trigger keeps its filter: nothing waits on a push, so skipping the
corpus for a README typo on main costs nothing and saves a runner. On pull
requests the corpus is about forty seconds, which is cheaper than one person
wondering why their pull request will not merge.

Two properties this file already had, now load-bearing and worth stating:

It declares no `schedule:`. The two workflows GitHub auto-disabled for
inactivity — link-check and schema-sync — are exactly the two that carry one,
and this repository went 73 days between commits, past the 60-day threshold. A
required context living in a scheduled workflow would be switched off by a
quiet spell and hang every pull request thereafter.

`corpus` and `backward compatibility` both declare `needs: meta`. A job skipped
because its `needs` failed still reports a check run, and branch protection
counts skipped as satisfied. So the three must be required TOGETHER: requiring
only the latter two would let a pull request merge while `gates can fail` is
red, and that job's entire purpose is proving the other two can go red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fas89

fas89 commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Once this lands I can set main to require all three contexts — gates can fail, corpus, backward compatibility — which is the change this PR exists to make safe.

Order matters and is not reversible cheaply: the contexts have to be produced by main before they are named as required, or every open PR hangs waiting for a report that will never come. So: merge this, confirm the three report on the next PR, then flip protection.

Two things that need your call at that point:

  1. lock_branch: true is set on main. Worth confirming that is deliberate.
  2. enforce_admins: false means three of the four collaborators bypass the gate entirely. Required checks are real for the fourth and advisory for everyone else until that flips.

@fas89
fas89 merged commit 15a066a into open-data-protocol:main Sep 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant