From f3689661f82359a800af65c9a50235210ed24732 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Sun, 13 Sep 2026 20:21:16 +0200 Subject: [PATCH 1/2] ci: gate what the corpus is worth, and retire a waiver's completed instruction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things that had quietly stopped being true. 1. mutation_coverage.py ran nowhere. `run.py` answers "do the cases pass" — a corpus of zero cases answers that perfectly. `mutation_coverage.py` answers the harder question: delete a constraint from the schema and see whether any case notices. It has been in this repo, runnable, documented and wired into nothing, so the figure could have fallen to zero between releases in silence. The floor is the current number rounded down (40.7% -> 40), not a target. That is where mutation-testing gates converge (Stryker, mutmut): a break threshold set where you actually are catches a regression on day one, whereas an aspirational one goes red immediately and gets deleted by day three, which is worse than no gate. Deliberately NOT a ratchet, and the divergence is on purpose. Ratchets exist to absorb run-to-run noise; there is none here, because the run is deterministic — mutate the schema, replay the corpus, count. A fixed floor does the same work with less machinery, and raising it stays a deliberate edit made when a corpus wave earns it. Proven to fail, three ways: --min-coverage 41 exits 1 against today's tree; trimming the metadata group to a single case drops it below 40 and turns the gate red; restoring it goes green again. 2. A waiver told the next reader to fix something already fixed. compat-waivers.txt said docs/releases/0.7.2.md "should be corrected to name this change". It was corrected, in #46 — the note now opens "One breaking change, corrected here after the fact". The instruction outlived the work. The waiver line itself stays, permanently: 0.7.2 is published and its history cannot be rewritten, so the gate has to be told this break is known rather than new. The comment now says that, and says what a waiver must never become — a way to quiet the gate about a break nobody dealt with. This one was dealt with by correcting the release note, not by adding the line. Corpus 480/480 and meta-test 19/19 unchanged. Co-Authored-By: Claude Opus 5 --- .github/workflows/conformance.yml | 21 +++++++++++++++++++++ scripts/compat-waivers.txt | 15 +++++++++++---- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index bd4b95e..b470c97 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -72,6 +72,27 @@ jobs: - name: run the corpus against the published schemas run: python3 conformance/run.py --junit conformance-report.xml + # What the corpus is WORTH, not just whether it is green. + # + # `run.py` answers "do the cases pass" — a corpus of zero cases answers + # that perfectly. `mutation_coverage.py` answers the harder question: + # delete a constraint from the schema and see whether any case notices. + # It has been in this repo, runnable, and wired into nothing, so the + # figure it produces could fall to zero between releases in silence. + # + # The floor is the CURRENT number rounded down (40.7% -> 40), not a + # target. That is the convention mutation-testing gates converge on + # (Stryker, mutmut): set the break threshold where you actually are, so + # it catches a regression on day one — an aspirational threshold goes red + # immediately and gets deleted by day three, which is worse than no gate. + # + # Deliberately NOT a ratchet. Ratchets exist to absorb run-to-run noise + # and there is none here: the run is deterministic — mutate the schema, + # replay the corpus, count. Raising the floor is a deliberate edit to + # this line, made when a corpus wave earns it. + - name: the corpus still kills the mutations it used to + run: python3 conformance/mutation_coverage.py --min-coverage 40 + # The corpus defines conformance; forge-cli is implementation #1 under # test, never the referee. Installing latest-stable here is deliberate: # it is how the Command Center installs it, so a divergence between the diff --git a/scripts/compat-waivers.txt b/scripts/compat-waivers.txt index a24446b..b00303d 100644 --- a/scripts/compat-waivers.txt +++ b/scripts/compat-waivers.txt @@ -25,10 +25,17 @@ # build.execution.notifications[0].retryOnFailure # 0.7.1: VALID 0.7.2: REJECTED (additionalProperties) # -# docs/releases/0.7.2.md says "100% backward compatible with 0.7.1" and -# "No breaking changes". That claim is false and should be corrected to name -# this change. This waiver exists so the gate can run green on history while -# the release note is fixed -- not to make the problem go away. +# docs/releases/0.7.2.md USED TO say "100% backward compatible with 0.7.1" and +# "No breaking changes". Both were false. The note was corrected in #46 and now +# opens by naming the break: "One breaking change, corrected here after the +# fact". The instruction this paragraph used to carry is therefore done, and +# leaving it standing would send the next reader to fix something already fixed. +# +# The waiver itself stays, permanently. 0.7.2 is published and its history +# cannot be rewritten, so the gate has to be told this break is known rather +# than new. What a waiver must never become is a way to stop the gate +# complaining about a break nobody dealt with -- this one was dealt with by +# correcting the release note, not by adding this line. 0.7.1->0.7.2 /$defs/notification # --------------------------------------------------------------------------- From a89d6783775e55a486e76f661dce3aae03168e7e Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Sun, 13 Sep 2026 20:25:21 +0200 Subject: [PATCH 2/2] ci: run the conformance gates on every pull request, not only matching paths MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A path-filtered check cannot be a REQUIRED check. On a pull request that does not match the filter the context never reports at all, so branch protection waits for a report that will never arrive and the pull request hangs permanently, with nothing to click. That is why main still requires no status checks: turning the existing three on as they were would have bricked the next docs PR rather than protected anything. Not hypothetical here. PR #47 was a docs change and received these three checks only because it happened to also touch conformance/run.py. PR #48 — LICENSE and CONTRIBUTING.md — reported no checks at all and merged with nothing having run. The push trigger keeps its filter: nothing waits on a push, so skipping the corpus for a README typo on main costs nothing and saves a runner. On pull requests the corpus is about forty seconds, which is cheaper than one person wondering why their pull request will not merge. Two properties this file already had, now load-bearing and worth stating: It declares no `schedule:`. The two workflows GitHub auto-disabled for inactivity — link-check and schema-sync — are exactly the two that carry one, and this repository went 73 days between commits, past the 60-day threshold. A required context living in a scheduled workflow would be switched off by a quiet spell and hang every pull request thereafter. `corpus` and `backward compatibility` both declare `needs: meta`. A job skipped because its `needs` failed still reports a check run, and branch protection counts skipped as satisfied. So the three must be required TOGETHER: requiring only the latter two would let a pull request merge while `gates can fail` is red, and that job's entire purpose is proving the other two can go red. Co-Authored-By: Claude Opus 5 --- .github/workflows/conformance.yml | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index b470c97..9eab28e 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -22,14 +22,21 @@ on: - "scripts/check-compat.py" - "scripts/compat-waivers.txt" - ".github/workflows/conformance.yml" + # NO `paths:` filter here, and that is the point. A path-filtered check + # cannot be a REQUIRED check: on a pull request that does not match, the + # context never reports at all, and branch protection waits for a report + # that will never come. The pull request hangs, permanently, with nothing + # to click. + # + # That is not hypothetical for this repo. PR #47 (a docs PR) only received + # these three checks because it happened to also touch conformance/run.py, + # and PR #48 — LICENSE and CONTRIBUTING.md — reported no checks at all. + # + # The push trigger above keeps its filter: nothing waits on a push, so + # skipping the corpus for a README typo on main costs nothing and saves a + # runner. On pull requests the corpus is ~40s, which is cheaper than one + # person wondering why their PR will not merge. pull_request: - paths: - - "schema/**" - - "tests/**" - - "conformance/**" - - "scripts/check-compat.py" - - "scripts/compat-waivers.txt" - - ".github/workflows/conformance.yml" workflow_dispatch: permissions: