No gate reports success over a corpus that is not there - #241
Merged
Conversation
A gate reports on the population it read, and no exit code separates "this
catalog holds no violation" from "this run held no catalog". Naming the gates
that confuse the two is a list of the ones somebody found, so this asks all of
them.
`scripts/tests/empty-corpus.py` copies the tracked tree, removes every file under
the directories the gates read, and runs each gate. The population is every
executable under `scripts/`, derived from the tree, so a gate added later is
probed without anyone remembering to add it. A gate must exit non-zero, and must
not do it by crashing.
That second requirement is its own defect. An unguarded `read_text()` on an
absent manifest raises FileNotFoundError, which exits 1 — the status this repo
uses for "the gate rejected the tree". By status alone a crash and a finding are
the same answer, and the traceback then names a pathlib internal rather than the
input that is missing.
Four gates read a manifest that way and crashed instead of refusing:
check-falco-rule-floor, check-log-volume-budget, check-platform-crs and
check-policy-admission. `gatelib.read_yaml` is the guarded single-document read
they now use, alongside the `read_yaml_all` and `read_json` already there.
check-hardcoded-org exited 0 outright, printing `Scanned 0 applied
ApplicationSet(s)` and its success line under `--blocking`.
Of the 25 gates, 23 are probed and refuse. Two are exempted and asserted: they
answer about `.github/workflows`, which this probe does not empty, so running
them here would assert nothing about them.
The harness rejects its own failure modes:
FAIL check-vacuous-probe.py exited 0 over an empty corpus — ✓ examined 0
ApplicationSet(s), nothing wrong
FAIL check-log-volume-budget.py CRASHED rather than refusing (exit 1) —
FileNotFoundError: [Errno 2] No such file or directory: …
FAIL check-gone.py is exempted as answering about nothing rather than the
catalog, but no such gate exists under scripts/ — the exemption
outlived its file.
FAIL only 23 gate(s) were probed, under the floor of 200 — the population
this harness reads shrank, and a clean run over it says nothing about
the rest.
Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
… masking
Three gates reported success over a corpus that was not there, inside the change
that exists to close that class.
`kubeconform-scan.sh` takes its corpus as an argument. The probe ran every gate
with no argv, so this one read nothing and exited 0 — and rather than being
caught it was exempted, on the claim that it answers about `.github/workflows`.
It does not: its callers pass `applicationsets/` and `rendered`. Both halves of
that exemption's assertion held while the claim was false, which is what an
exemption asserted on the wrong operand looks like. The probe now carries
GATE_ARGS, the same mechanism controls.py already had for the same reason, and
the gate itself floors on what kubeconform reported reading:
FAIL kubeconform validated 0 resources under: applicationsets/
Nothing was schema-checked, which is not the same as everything being
valid.
`check-platform-crs.py` printed `walked` and gated on nothing;
`check-policy-admission.py` printed its rendered count and compared it only
against a render failure. Both now refuse:
FAIL 4 custom resource(s) carry an operator API group, below the floor of 5.
FAIL 7 manifest(s) rendered, below the floor of 40.
`validate-dashboards.py` declared MIN_DASHBOARD_REFS and never compared it; the
enforced floor was one. It is compared now.
Both floors are constants, and the note at each says why rather than dressing one
up. A derivation was written for both and both were circular: check-platform-crs
counting candidates by API group against candidates by schema kind reads the same
walk on both sides, and check-policy-admission bounding the render by
`discover()` reads the same ApplicationSets the render does — so a corpus that
shrinks shrinks the bound with it and the comparison holds. Nothing else in this
tree enumerates either population. `test_corpus_floors.py` holds both bounds
against the tree instead, above zero and below the real corpus.
The probe records what it cannot separate. Several gates derive their coordinates
from an ApplicationSet, so emptying every corpus takes that input away too and
they exit 2 naming the appset rather than saying anything about the corpus they
check. Those runs are marked. A two-pass probe keeping `applicationsets/` was
written and does not separate the cases — for a gate whose corpus IS
applicationsets, the same file is both — and what would separate them is the
per-gate map this harness exists to avoid. The limit is in the header.
24 gates probed, all refusing, none by crashing.
Co-authored-by: stxkxs <275011021+stxkxsbot@users.noreply.github.com>
CI Results
One or more checks failed — see the job logs. |
The probe's population was the executables under `scripts/`. This repository runs
other things as gates in the same workflow, blocking the same merges, and a
harness named for flooring every corpus whose population quietly excludes them
has the defect it exists to find.
Three are now probed as their callers invoke them: `kyverno test`, `trivy config`
over the rendered tree, and the kustomize build loop. One passed over nothing:
FAIL kyverno test: exited 0 with policies/ emptied — No test yamls available
`scripts/kyverno-test.sh` floors it on tests executed. The CLI exits 0 over a
directory holding no tests, which is the status a passing suite gets, and this is
the job standing behind every Kyverno policy in the catalog — a renamed directory
or a narrowed path reported success.
The probe's own floor was unfloored. `MIN_PROBED` decides whether a run proved
anything and nothing read it, so setting it to zero left the tree green over a
probe that examined nothing — the harness's shape one level up. It is now bounded
both ways, and the upper bound is the gate population itself rather than a second
constant, so it cannot be raised past what any tree can satisfy.
MIN_CRS drops from 5 to 2. `candidates` counts documents whose API group the
operator owns, but the walk keeps only kinds the pinned chart ships a schema for,
so a healthy run's number depends on what that chart resolves to where the gate
runs — a reviewer measured four where this tree walks eight, and the job was red
on the tip as pushed. A floor above the smallest resolution is red somewhere it
should be green.
28 gates probed, all refusing, none by crashing.
Co-authored-by: stxkxs <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
stxkxs
marked this pull request as ready for review
September 4, 2026 22:43
stxkxs
added a commit
that referenced
this pull request
Sep 4, 2026
#241 landed the vacuity harness, GATE_ARGS, gatelib.read_yaml across four gates, and corpus floors on check-platform-crs, check-policy-admission, validate-dashboards and kyverno-test.sh. This branch plants violations against those same offline verdicts and asserts the control-exempt gates, so the two work the same ground from opposite directions. Four textual conflicts, all of them both-sides-added rather than either-side-changed. Each is resolved by keeping both, in one order that the Taskfile and the workflow now share: .github/workflows/ci.yml — both branches inserted a step after the positive controls. `empty-corpus.py` asserts a GATE rejects an absent corpus and `reverify-tests.sh` asserts the TESTS reject a reverted behaviour, so the gate-level probe runs beside the other gate-level ones and the suite-level one after them. Taskfile.yaml, twice — the target definitions and the `validate` list, ordered to match the workflow so reading either gives the same sequence. scripts/tests/controls.py — both added a NOT_GATES entry. Both harnesses are exempt for the same reason and both keep their recorded reason. scripts/tests/run.py — both added to EXPECTED. `test_corpus_floors` joins the harness layer with a line saying what it holds. The comment above `test_controls` counted the exempt harnesses; it now states the rule instead, because the count was about to be wrong and would have stayed readable while being false. ──────────────── What the merge changed beyond the conflicts ──────────────── `empty-corpus.py` is a harness registered in NOT_GATES, so it enters the population of this branch's newest rule: a harness exempted for self-assertion must be invoked. It is — by `validate:empty-corpus` and by its own CI step — and the rule reaches it. Removing both commands while leaving the task named after it makes `invoked_anywhere` answer False and `check_vacuity` name the file with its recorded reason, so the coverage is real rather than incidental. `check-platform-crs.py` gained a `Cannot run` refusal when the operator ApplicationSet is absent, and its per-gate coverage floor read 49% against a floor of 50 — a floor doing exactly what a ratchet is for. Resolved by covering the verdict rather than lowering the number: `TheVersionThePinResolves` holds the four things `pinned_chart_version` decides, since the version it reads is upstream of every schema assertion in that module. Four probes in `reverify-tests.sh`, floor 27 → 31, each killed and named. Coverage 52%, floors raised to 51 and 31. `reverify-tests.sh` re-runs `test_corpus_floors` too. The floors #241 added sit in three gates this harness already mutates against, so a mutant reverting the floor half is held by that module and by nothing else in the list — without it, a caught mutant read as a miss. Its "all N modules green" line now says these are the modules the harness mutates against, which is what N was always counting. ──────────────── Verification on the merged tree ──────────────── ruff clean; mypy clean over 42 files; 297 tests across 15 modules; coverage ratchet 31.5% combined (floor 30 → 31), 12 per-gate floors held, 10 of 23 gates with no unit coverage against a ceiling of 11; controls.py 17 controls each rejecting at exit 1 naming the mutated file, so no planted violation trips a new floor first; empty-corpus.py 28 probed gates; reverify-tests.sh 31/31; task validate exit 0; check-named-things.py 187 references resolve; check-workflows.sh clean at MEDIUM and above; renovate-config-validator accepts the config. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
stxkxs
added a commit
that referenced
this pull request
Sep 5, 2026
#241 landed the vacuity harness, GATE_ARGS, gatelib.read_yaml, and corpus floors on check-platform-crs, check-policy-admission, validate-dashboards and kyverno-test.sh. This branch had independently added floors to two of those same gates and a module by the same name, so most of the conflict is two answers to one question rather than two changes to one file. ──────────────────────────── Conflicts ──────────────────────────── scripts/check-platform-crs.py — both sides added a floor at the same point. This branch floored `walked`; main asserts completeness instead: candidates by API GROUP against reached by SCHEMA KIND, with the number floor on candidates alone. main's is strictly stronger and subsumes the other — `walked` is incremented exactly where `reached` gains an entry, so candidates >= MIN_CRS with candidates - reached empty forces walked >= MIN_CRS. Keeping both would leave a rule whose failure is unreachable, so this branch's floor and its duplicate MIN_CRS go. scripts/check-policy-admission.py, twice — the same floor at the same value, arrived at independently. main's returns the named gatelib.CANNOT_RUN rather than a literal 2 and records why the bound cannot be derived and why the canary makes the floor necessary. Taken whole. scripts/tests/test_corpus_floors.py (add/add) — two modules of the same name written against overlapping gates. Resolved as their union, not as a choice: main's structure and its record of the two circular derivations that were rejected, plus this branch's degenerate-render bound, its IMAGELESS_CHARTS rot check, and its per-chart image-floor assertion. The `TheFloorsGuardTheRightQuantity` table is main's, because it names the expressions the merged gates actually contain. scripts/tests/run.py — both sides list test_corpus_floors; kept once. CLAUDE.md — two different task targets added to one list; both exist. ──────────────── A digest with no repository is not a reference ──────────────── The image-vulnerability gate refuses on any reference the classifier cannot place, and five of the ten it named in CI are bare `sha256:<hex>` values with no repository before them. Those name no image: there is no registry and no repository in one, only the value. Excluded by SHAPE rather than by a declaration, because the two rules this repository already has cannot both hold over it. A NOT_A_CONTAINER entry excuses whatever carries its bare name, so an entry for `sha256` would excuse every reference whose only matched token is a digest — and declaration_rot deletes an entry the render does not support, which is every tree that happens not to carry a bare digest. One rule demands the entry and the other removes it; a shape that can never be a reference is not an exemption to keep re-reading. Nothing is lost. A digest belonging to a repository is matched with that repository, whole, by the alternative above it, and stays classified as a digest. Two mutants, both killed and named: the exclusion removed, and the exclusion written unanchored so it eats repository digests too. ──────────────── Verification on the merged tree ──────────────── ruff clean; mypy clean over 38 files; 171 tests across 10 modules; coverage ratchet 23.1% combined (floor 22 → 23); controls.py 17 controls each rejecting at exit 1 naming the mutated file; empty-corpus.py 29 probed gates, up from 28 because check-image-vulnerabilities.py joins the sweep; reverify-gates.sh 30/30; task validate exit 0; check-image-pins.py 80 images across 28 charts; check-named-things.py 191 references resolve; check-workflows.sh clean at MEDIUM and above. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
stxkxs
added a commit
that referenced
this pull request
Sep 5, 2026
No conflicts. #241's changes and this branch's touch the same four files — .github/workflows/ci.yml, CLAUDE.md, Taskfile.yaml, scripts/tests/run.py — in disjoint regions: #241 registers the vacuity harness, this branch registers the Node pin and the Renovate manager-default record. Nothing else changed. The Renovate work is unaltered by the merge. Verification on the merged tree: ruff clean; mypy clean over 36 files; 190 tests across 9 modules; coverage ratchet 24.9% combined; controls.py 17 controls all proving; empty-corpus.py 28 probed gates; reverify-gates.sh 35/35; task validate exit 0; check-named-things.py 191 references resolve; check-workflows.sh clean at MEDIUM and above; renovate-config-validator accepts the config; check-renovate-defaults.mjs exit 0 with the record and the node pin both OK against renovate 44.65.0; check-renovate-coverage.py exit 0 at 32 chart pins, 18 other-family pins and 27 toolchain pins. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The vacuity class, enumerated rather than instanced.
Partly covered already — what is and is not new
You asked me to say so if an open branch covers this. #239 covers four gates
(
check-hardcoded-org,check-platform-crs,check-policy-admission,check-image-vulnerabilities) with picked-number floors and a two-sided test.What it does not do is enumerate — it fixes the three I happened to find.
This branch is the enumeration, and it found two things #239 does not reach.
Overlap to reconcile at merge:
check-hardcoded-org.py'sMIN_APPSETSblockappears on both branches, written identically so it resolves as one hunk rather
than two designs.
check-platform-crs.pyandcheck-policy-admission.pyaretouched by both, in different places.
The enumeration
scripts/tests/empty-corpus.pycopies the tracked tree, removes every file underthe directories the gates read, and runs each gate. The population is every
executable under
scripts/, derived from the tree — a gate added later isprobed without anyone remembering to add it. Corpus emptying is wholesale rather
than per gate, for the same reason: a per-gate map of what each one walks is the
hand-maintained list this replaces, and it would go stale in the direction that
matters.
Of 25 gates, 23 are probed and refuse. Two are exempted and asserted — they
answer about
.github/workflows, which this probe does not empty, so runningthem here would assert nothing about them.
What it found
One gate exited 0.
check-hardcoded-org.py, printingScanned 0 applied ApplicationSet(s)and its success line under--blocking. Already fixed on #239;fixed identically here so the harness can pass.
Four gates crashed rather than refusing — and this is the part #239 does not
reach. An unguarded
read_text()on an absent manifest raises FileNotFoundError,which exits 1, the status this repo uses for "the gate rejected the tree". By
status alone a crash and a finding are the same answer, and the traceback names a
pathlib internal rather than the input that is missing:
check-falco-rule-floor.pycheck-log-volume-budget.pycheck-platform-crs.py— Scan the images the pins reference, and make a CRITICAL a decision #239 gives itMIN_CRS, which sits after thecrash point, so that floor never runs on this path
check-policy-admission.py— same, withMIN_RENDEREDgatelib.read_yamlis the guarded single-document read they now use, alongsidethe
read_yaml_allandread_jsonalready there.Every verdict has a planted violation it names
The first plants a gate whose body is the vacuous shape itself; the second
reverts one of the four fixes; the third and fourth attack the harness's own
population and exemption list.
On "derived rather than picked"
I could not make the floors themselves derived, and the reason is worth stating
rather than hiding. A derived floor needs a second enumerator that does not
go empty with the corpus.
git ls-filesdoes not qualify — empty the directoryand the index agrees, so an equality between them holds at zero and closes
nothing.
git ls-tree HEADdoes qualify but is circular in CI, where HEAD is theworking tree.
So the floors stay constants, and what is derived is the enumeration of who needs
one and the proof that each refuses. #239's
test_corpus_floors.pyholds theother half against the tree: above zero, and below the real corpus.
Verification
All exit 0 on the branch head:
task validate— 25 gates, now includingvalidate:empty-corpusruff check .;mypyover 34 files./scripts/tests/empty-corpus.py— 23 probed, all refuse, none by crashing./scripts/tests/controls.py— 17 controlsyamllint;./scripts/check-workflows.sh;./scripts/check-named-things.pyWired into
task validateand thegate-testsCI job; exempted inNOT_GATESas a harness, with the exemption asserted.