ci: gate what the corpus is worth, and retire a waiver's completed instruction - #49
Merged
fas89 merged 2 commits intoSep 13, 2026
Conversation
…struction Two things that had quietly stopped being true. 1. mutation_coverage.py ran nowhere. `run.py` answers "do the cases pass" — a corpus of zero cases answers that perfectly. `mutation_coverage.py` answers the harder question: delete a constraint from the schema and see whether any case notices. It has been in this repo, runnable, documented and wired into nothing, so the figure could have fallen to zero between releases in silence. The floor is the current number rounded down (40.7% -> 40), not a target. That is where mutation-testing gates converge (Stryker, mutmut): a break threshold set where you actually are catches a regression on day one, whereas an aspirational one goes red immediately and gets deleted by day three, which is worse than no gate. Deliberately NOT a ratchet, and the divergence is on purpose. Ratchets exist to absorb run-to-run noise; there is none here, because the run is deterministic — mutate the schema, replay the corpus, count. A fixed floor does the same work with less machinery, and raising it stays a deliberate edit made when a corpus wave earns it. Proven to fail, three ways: --min-coverage 41 exits 1 against today's tree; trimming the metadata group to a single case drops it below 40 and turns the gate red; restoring it goes green again. 2. A waiver told the next reader to fix something already fixed. compat-waivers.txt said docs/releases/0.7.2.md "should be corrected to name this change". It was corrected, in open-data-protocol#46 — the note now opens "One breaking change, corrected here after the fact". The instruction outlived the work. The waiver line itself stays, permanently: 0.7.2 is published and its history cannot be rewritten, so the gate has to be told this break is known rather than new. The comment now says that, and says what a waiver must never become — a way to quiet the gate about a break nobody dealt with. This one was dealt with by correcting the release note, not by adding the line. Corpus 480/480 and meta-test 19/19 unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…g paths A path-filtered check cannot be a REQUIRED check. On a pull request that does not match the filter the context never reports at all, so branch protection waits for a report that will never arrive and the pull request hangs permanently, with nothing to click. That is why main still requires no status checks: turning the existing three on as they were would have bricked the next docs PR rather than protected anything. Not hypothetical here. PR open-data-protocol#47 was a docs change and received these three checks only because it happened to also touch conformance/run.py. PR open-data-protocol#48 — LICENSE and CONTRIBUTING.md — reported no checks at all and merged with nothing having run. The push trigger keeps its filter: nothing waits on a push, so skipping the corpus for a README typo on main costs nothing and saves a runner. On pull requests the corpus is about forty seconds, which is cheaper than one person wondering why their pull request will not merge. Two properties this file already had, now load-bearing and worth stating: It declares no `schedule:`. The two workflows GitHub auto-disabled for inactivity — link-check and schema-sync — are exactly the two that carry one, and this repository went 73 days between commits, past the 60-day threshold. A required context living in a scheduled workflow would be switched off by a quiet spell and hang every pull request thereafter. `corpus` and `backward compatibility` both declare `needs: meta`. A job skipped because its `needs` failed still reports a check run, and branch protection counts skipped as satisfied. So the three must be required TOGETHER: requiring only the latter two would let a pull request merge while `gates can fail` is red, and that job's entire purpose is proving the other two can go red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
Author
|
Once this lands I can set Order matters and is not reversible cheaply: the contexts have to be produced by Two things that need your call at that point:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things that had quietly stopped being true.
1.
mutation_coverage.pyran nowhererun.pyanswers "do the cases pass" — a corpus of zero cases answers that perfectly.mutation_coverage.pyanswers the harder question: delete a constraint from the schema and see whether any case notices. It has been in this repo, runnable, documented, and wired into nothing, so the figure could have fallen to zero between releases in silence.The floor is the current number rounded down (40.7% → 40), not a target. That is where mutation-testing gates converge: a break threshold set where you actually are catches a regression on day one, whereas an aspirational one goes red immediately and gets deleted by day three — worse than no gate.
Deliberately not a ratchet, and the divergence is on purpose. Ratchets exist to absorb run-to-run noise; there is none here, because the run is deterministic — mutate the schema, replay the corpus, count. A fixed floor does the same work with less machinery, and raising it stays a deliberate edit made when a corpus wave earns it.
Proven to fail, three ways:
--min-coverage 40The third is the one that matters — it is a real corpus regression, not a moved goalpost.
2. A waiver told the next reader to fix something already fixed
compat-waivers.txtsaiddocs/releases/0.7.2.md"should be corrected to name this change". It was corrected, in #46 — the note now opens "One breaking change, corrected here after the fact". The instruction outlived the work.The waiver line itself stays, permanently: 0.7.2 is published and its history cannot be rewritten, so the gate has to be told this break is known rather than new. The comment now says that, and says what a waiver must never become — a way to quiet the gate about a break nobody dealt with. This one was dealt with by correcting the release note, not by adding the line.
Borrow-before-build
Corpus 480/480 and meta-test 19/19 unchanged.
🤖 Generated with Claude Code