Green must mean measured: assert the reproduction population - #4
Merged
Conversation
…t its failures Run 31289030963 was five green jobs out of seven, and two of those five were skips. verify-reproduction.sh exits 0 when a package's declared version is not published, which is correct and invisible: at job level a skip and a reproduction are the same tick. Coverage had fallen from six packages to five, caused by a CORRECT fix. x402-op-authorize was bumped off the 0.4.0 it had drifted from, which moved the one package this check was written for out of the check's reach. Nothing went wrong and coverage dropped anyway. - verify-reproduction.sh records measured|skipped|inconclusive at every exit path. A dist/ mismatch counts as MEASURED: the comparison ran and the answer was no. A job that dies without writing counts as no result, never as a pass. - assert-reproduction-population.sh asserts three separable things: every package under packages/ reported, the measured count is at or above the declared floor, and the floor itself is readable. Expected population is read from packages/ rather than the matrix, so a package added to the repo and forgotten in the workflow fails here. - scripts/reproduction-floor.txt declares the floor (5) as a committed decision, so accepting less coverage is an edit a reviewer sees rather than a side effect. - Three self-tests: the assertion must fail when everything is skipped (naming the floor it read, not merely exiting non-zero), must fail when a package reports nothing, and must PASS when the floor is met, so the first two are not satisfied by an assertion that always fails. Verified locally: all-skipped exit 1, one-missing exit 1, absent floor exit 1, unparseable floor exit 1, floor met exit 0.
…request Found by this PR going red on run 31296563830 with "scripts/reproduction-floor.txt: No such file or directory" — a file this branch adds and main does not have. The job's setup step ran `git checkout -B guard-selftest origin/main`, which it did to give HEAD an upstream so the guard reaches the checks under test. It also replaced the working tree. Workflow STEPS come from the pull request; the scripts/ they exercised came from main. This is invisible while a PR leaves the guard alone, and it inverts the moment one does not: a pull request that BROKE refuse-dirty-publish.sh would have its self-tests pass, because they would be exercising the unbroken copy on main. The control was green for a reason unrelated to the code under review. It only surfaced here because this branch adds a file rather than changing one, turning a silent pass into a missing-file error. - check out the head of the branch under test, which gives HEAD an upstream AND keeps the tree being reviewed. On a push to main the two are the same commit. - assert that the checked-out SHA is the one under review, so the next edit to that step cannot reintroduce this quietly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
Run 31289030963 reported five green jobs out of seven. Two of those five green jobs were skips.
verify-reproduction.shexits 0 when a package's declared version is not published. That is correct, and it is invisible: at job level a skip and a reproduction are the same tick. So the run described a state where five packages were compared against the registry and two were compared against nothing.The coverage drop was caused by a correct fix.
x402-op-authorizewas bumped to0.4.1-rc.0, off the0.4.0its tree had drifted from. That bump moved the one package this check was written for out of the check's reach. Nothing went wrong and coverage fell anyway, which is why the fix here is a population assertion rather than a change to the bump.Population as measured 2026-08-09
fireblocks-op-authorize@0.1.0,l402-op-authorize@0.4.0,mppx-op-account@0.3.1,ows-op-verify@0.3.0,wdk-op-policy@0.4.1ap2-op-authorize@0.1.0(never published),x402-op-authorize@0.4.1-rc.0(not yet published)l402-op-authorizecounts as measured although its job is red. The comparison ran and returned an answer. A red measurement is coverage; a skip is not.What changed
verify-reproduction.shrecordsmeasured/skipped/inconclusiveat every exit path. A job that dies without writing counts as no result, never as a pass.assert-reproduction-population.shasserts three separable things: every package underpackages/reported, the measured count is at or above the declared floor, and the floor file is readable (fail closed). The expected population is read frompackages/rather than from the matrix, so a package added to the repo and forgotten in the workflow fails here instead of being silently uncovered.scripts/reproduction-floor.txtdeclares the floor as a committed decision. Lowering it is an edit a reviewer sees, not a side effect of a version bump.The check can fail
Three self-tests in
guard-self-test, because a coverage check that cannot go red is worth less than none:Verified locally before pushing: all-skipped
1, one-missing1, absent floor1, unparseable floor1, floor met0.Expected CI state on this PR
reproduce (l402-op-authorize)stays red, and that red is true: the published0.4.0tarball carries adist/core/no clean build produces. That is held on Boyd for a clean republish and is untouched here.