Prove the gates a positive control cannot reach - #238
Merged
Conversation
`scripts/tests/controls.py` proves a gate rejects by planting the violation it
names and requiring a non-zero exit. It cannot do that for a gate whose input
arrives over a network, so `NEEDS_NETWORK_PY` exempts seven. Six of those seven
also carried no unit tests, which left 1,107 of 3,072 statements in the gate
corpus with no proof of any kind — and not a random sixth of it. The set is
`check-policy-admission.py`, which decides whether any addon would be denied at
admission on an Enforce cluster; `render-addons.py`, whose `discover()` is the
corpus two other gates import rather than re-derive; `check-platform-crs.py`,
`validate-dashboards.py`, `check-image-pins.py` and `check-log-volume-budget.py`.
An exemption from one kind of proof is not an exemption from proof.
─────────────────────── The assertion, not the number ───────────────────────
`scripts/tests/run.py` gains a check that does not depend on a percentage: every
gate `controls.py` exempts must carry unit coverage. The exemption list is read
out of `controls.py` at run time rather than copied, because a second copy of an
exemption list drifts permissively — the gate added to one and not the other is
excused by both.
Three ways that check can quietly stop meaning anything, each of which now fails:
a control-exempt gate arriving with no tests
FAIL 1 gate(s) have neither a positive control nor unit coverage:
check-hardcoded-org.py. controls.py exempts them because their input
arrives over the network, so a unit test on the half that decides the
verdict is the only proof available that they say anything at all
the exemption list renamed out from under the reader
FAIL controls.py could not be read for its control exemptions
(AttributeError: module 'controls' has no attribute
'NEEDS_NETWORK_PY'), so no gate could be checked for having neither
kind of proof
the exemption list emptied
FAIL controls.py exempts no gate from a positive control, so the assertion
that every exempt gate carries unit tests holds over an empty set and
states nothing
The coverage floors move too — combined 12 to 29, the no-unit-coverage ceiling 17
to 11, and seven new per-gate floors — but they are the weaker half and the
comment in `run.py` says so. A count loose enough to survive an honest refactor
is loose enough to miss a decision going untested.
─────────────────────────── What the tests assert ───────────────────────────
157 tests across six modules, each over the decision that produces a verdict
rather than over parsing or setup. Every one of them was verified by reverting
the behaviour it names and watching the suite refuse:
scripts/tests/test_policy_admission.py (29)
Manifest normalisation, rule-name derivation and the two-sided verdict.
try the shorter autogen prefix first
FAIL: test_the_cronjob_prefix_is_stripped_whole
keep kind-less documents in the resource file
ERROR: test_a_kindless_document_is_dropped
admit the runtime pod on absence of denials alone
FAIL: test_a_pod_no_rule_evaluated_is_not_admitted
FAIL: test_a_skip_is_neither_a_pass_nor_a_denial
FAIL: test_a_clean_canary_cannot_carry_a_missing_runtime_pod
count the runtime pod as a foreign addon
FAIL: test_the_runtime_pod_is_not_counted_among_flagged_addons
scripts/tests/test_platform_crs.py (29)
The four rules the API server enforces and no other gate here checks:
required-with-defaulting, pruning, declared types, list identity.
let a bool satisfy integer, as Python does
FAIL: test_a_bool_does_not_satisfy_integer
read `required` without the defaulting exemption
FAIL: test_a_missing_required_property_that_declares_a_default_is_admitted
check list identity only at the top level
FAIL: test_a_set_list_identifies_a_scalar_by_itself
FAIL: test_an_absent_key_participates_in_the_identity
FAIL: test_three_entries_sharing_an_identity_report_each_repeat
scripts/tests/test_dashboards.py (28)
The offline extractors: variable references in all four Grafana spellings,
the `json: |` block, nested panels, and which datasources count as wired.
match only the ${name} variable form
FAIL: test_the_bare_form_every_dashboard_here_uses
FAIL: test_the_bracket_form
FAIL: test_a_builtin_beside_a_real_variable_leaves_the_real_one
scan panels flat, without descending into a collapsed row
FAIL: test_an_alert_inside_a_collapsed_row_is_found
FAIL: test_a_top_level_alert_panel_is_found
FAIL: test_an_untitled_alert_panel_is_still_reported
glob the datasource directory instead of reading the kustomization
FAIL: test_a_datasource_file_absent_from_resources_is_not_wired
scripts/tests/test_render_addons.py (29)
Unit discovery and the exit-code classification that separates a pin which
does not resolve from a registry that could not be reached.
call any missing-chart message a finding about this repo
FAIL: test_a_message_naming_both_is_read_as_unreachable
append the chart name to every OCI repoURL
FAIL: test_a_repo_url_ending_in_the_chart_name_is_used_as_is
take the addon path from any values file
FAIL: test_a_per_environment_file_alone_does_not_supply_the_path
scripts/tests/test_image_pins.py (23)
Reference classification and both directions of the exemption check.
look for the tag colon anywhere in the reference
FAIL: test_a_registry_port_is_not_read_as_a_tag
stop re-checking exemptions against the render
FAIL: test_an_exemption_the_fleet_no_longer_renders_mutably_fails
FAIL: test_an_exemption_for_an_image_now_pinned_by_tag_fails
ERROR: test_the_recorded_reason_is_quoted_back
scripts/tests/test_log_volume_budget.py (19)
The relation the gate exists for — the fill alert leads the ingestion cutoff
— plus every way the alert can stop supplying its operand.
let the alert fire at the cutoff rather than before it
FAIL: test_a_warning_at_the_cutoff_leaves_no_window
accept a rule keyed on the edge counter
FAIL: test_a_rule_querying_the_edge_counter_fails_twice
FAIL: test_a_rule_querying_both_still_fails_on_the_edge_counter
Bytecode caching is disabled while a mutant is under test. It keys on mtime and
size, so a restore and a re-mutation inside the same second can hand the next run
the previous mutant's module — which reads as the suite catching a mutant it
never saw.
──────────────────────── Three verdicts made reachable ────────────────────────
Three gates computed their verdict inline with the network call that fed it, so
the decision could only be exercised through a round trip. Each is extracted
whole, with no change to what it decides or to what it prints:
scripts/check-log-volume-budget.py `environment_verdict(cfg, warn, rel)`
returns the problems found and whether the alert leads a declared cutoff.
The second value is what the closing line counts, so an environment that
failed the comparison is not also reported as one the comparison covered.
scripts/check-image-pins.py `verdict(images, allowed)`, `bare_name(ref)`
`bare_name` splits on the last colon only when it sits in the final path
segment, so a registry port is not read as a tag.
scripts/render-addons.py `registry_answered(err)`
true when helm's complaint says the registry replied and refused the pin.
A message naming both a missing chart and a broken connection is not one:
an unreachable registry cannot testify about what it holds.
The summary lines these gates print are unchanged — 55 rendered images across 26
charts, four Loki environments checked against an alert at 0.75.
────────────────────────────── Why not a number ──────────────────────────────
Raising the coverage floor alone would have left the same hole. The floor is a
ratchet over the whole corpus, so a gate arriving with no tests dilutes it by a
percentage point and passes; and the six gates in question were the largest files
in the tree, so the number they were absent from was the number they dominated.
`.coveragerc` said as much and the figure stayed low anyway. What closes it is a
per-gate property with the exemption list as its operand.
Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
The previous commit's mutation results were produced out of tree, so the note
about bytecode caching named a procedure a reader cannot run and the seventeen
probes existed nowhere. A proof nobody can re-run is the shape this repo rejects
everywhere else.
`scripts/tests/reverify-tests.sh` is that procedure, as a sibling to
`scripts/tests/reverify-gates.sh`. The existing harness proves a GATE rejects a
planted defect; this proves the TESTS reject a reverted behaviour, which a
passing suite cannot establish on its own — a test that asserts what it just
constructed passes forever, and so does one whose subject was rewritten
underneath it.
─────────────────────────── Naming, not just failing ───────────────────────────
Each probe declares the test ids that must fail, and a run that goes red without
them is reported as a MISS rather than a pass:
MISS classify: find the tag colon anywhere in the reference rejected, but
not by test_a_version_tag_is_accepted
RESULT pass=16 fail=1
A non-zero exit alone proves the suite noticed something, not that the assertion
which noticed is the one describing the behaviour. A mutant caught through an
unrelated fixture has demonstrated detection without demonstrating coverage, and
the two are indistinguishable from an exit code.
The other two ways it can stop meaning anything fail as well:
MISS alert: accept a rule keyed on the edge counter the suite
passed the mutant
RESULT pass=16 fail=1
RESULT pass=4 fail=0
FAIL ran 4 probe(s), under the floor of 17 — this harness
planted almost nothing, which is not the same as every test holding.
──────────────────────────────── What it plants ────────────────────────────────
Seventeen probes across the six modules, asserting the clean tree is green before
anything is planted and green again after. On the tree as it stands:
── Clean tree: the suite must PASS before anything is planted ──
ok all six modules green on the unmodified tree
── One behaviour reverted at a time; the suite must name it ──
ok autogen: strip the shorter prefix first
FAIL: test_the_cronjob_prefix_is_stripped_whole
ok prepare: keep kind-less documents
ERROR: test_a_kindless_document_is_dropped
ok runtime pod: admit on absence of denials alone
FAIL: test_a_pod_no_rule_evaluated_is_not_admitted
FAIL: test_a_skip_is_neither_a_pass_nor_a_denial
FAIL: test_a_clean_canary_cannot_carry_a_missing_runtime_pod
ok judge: count the runtime pod as a foreign addon
FAIL: test_the_runtime_pod_is_not_counted_among_flagged_addons
ok types: let a bool satisfy integer, as Python does
FAIL: test_a_bool_does_not_satisfy_integer
ok required: drop the defaulting exemption
FAIL: test_a_missing_required_property_that_declares_a_default_is_admitted
ok list identity: check only the top level
FAIL: test_a_set_list_identifies_a_scalar_by_itself
FAIL: test_an_absent_key_participates_in_the_identity
FAIL: test_three_entries_sharing_an_identity_report_each_repeat
ok variables: match only the braced form
FAIL: test_the_bare_form_every_dashboard_here_uses
FAIL: test_the_bracket_form
FAIL: test_a_builtin_beside_a_real_variable_leaves_the_real_one
ok alerts: scan panels flat, skipping collapsed rows
FAIL: test_an_alert_inside_a_collapsed_row_is_found
FAIL: test_a_top_level_alert_panel_is_found
FAIL: test_an_untitled_alert_panel_is_still_reported
ok datasources: glob the directory, not the kustomization
FAIL: test_a_datasource_file_absent_from_resources_is_not_wired
ok helm failure: call every missing-chart message a finding
FAIL: test_a_message_naming_both_is_read_as_unreachable
ok oci: append the chart name to every repoURL
FAIL: test_a_repo_url_ending_in_the_chart_name_is_used_as_is
ok addon path: take it from any values file
FAIL: test_a_per_environment_file_alone_does_not_supply_the_path
ok classify: find the tag colon anywhere in the reference
FAIL: test_a_registry_port_is_not_read_as_a_tag
ok exemptions: stop re-checking them against the render
FAIL: test_an_exemption_the_fleet_no_longer_renders_mutably_fails
FAIL: test_an_exemption_for_an_image_now_pinned_by_tag_fails
ok cutoff: let the alert fire AT it rather than before
FAIL: test_a_warning_at_the_cutoff_leaves_no_window
ok alert: accept a rule keyed on the edge counter
FAIL: test_a_rule_querying_the_edge_counter_fails_twice
FAIL: test_a_rule_querying_both_still_fails_on_the_edge_counter
── Restored tree must be green again ──
ok all six modules green again
RESULT pass=17 fail=0
──────────────────────────── Two properties it needs ────────────────────────────
Bytecode is cleared and disabled around every probe. The cache keys on mtime and
size, so a restore and the next mutation inside the same second can hand the run
the previous mutant's module — and the suite then reports the previous mutant's
failing test against this mutant's label. That is a green line naming the wrong
assertion, which is worse than a red one, and the naming requirement above is
what surfaces it.
Backups live outside the scratch directory the cleanup removes, and restoration
runs on INT and TERM as well as EXIT. The probes plant defects in tracked gate
scripts; between planting and restoring the backup is the only copy of the
original, and `scripts/tests/reverify-gates.sh` records what happens when that
copy sits inside the directory the teardown deletes.
Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
`scripts/tests/controls.py` enumerates every executable under `scripts/` and
requires each to carry either a positive control or an entry saying why it cannot.
Adding `scripts/tests/reverify-tests.sh` without one failed that check before it
ran anything:
FAIL tests/reverify-tests.sh ships no positive control and is on no
exemption list. A gate nobody has shown to fail is an untested
assertion about the tree.
The exemption is the right answer here rather than a control, for the reason
`NOT_GATES` already records for its two neighbours: a harness that drives the
gates is not itself checking the tree, and a control over it would be the thing
under test and the thing testing it at once. What stands in for one is its own
self-assertion on every invocation — the clean-tree and restored-tree checks, the
requirement that each mutant be named by the assertion describing it, and the
probe floor.
The entry is asserted in both directions, like every other exemption here.
Renaming the file fails on both sides at once:
FAIL tests/reverify-tests.sh is exempted as a harness rather than a gate,
but no such executable exists under scripts/ — the exemption outlived
its file.
FAIL tests/reverify-tests-renamed.sh ships no positive control and is on no
exemption list.
Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
Two verdicts decided the exit code of a control-exempt gate and were asserted
only to be empty on a healthy tree. That claim is satisfied by every
implementation that reports nothing, `return []` included, so it is a control
against false positives rather than evidence of detection.
`check_local_dashboards` is the whole offline half of validate-dashboards: an
unwired panel datasource, an undeclared template variable, a locally-authored
board whose JSON cannot be read, and a floor that fires when the corpus declares
variables and references none. Each is now planted against a fixture tree and the
message it produces is named. Replacing the function body with `return []` is
killed by five tests.
`check_exclusion_parity` is the offline half of check-policy-admission, and a
divergence between the four exclusion lists is invisible to every other gate —
each policy is individually valid, renders, and passes kyverno's unit tests. It
surfaces on a vended Enforce cluster as one workload denied where its three
siblings were admitted. A divergent list, an extra namespace, a reordering that
is not a divergence, and a second `exclude.any` entry are now planted; returning
True unconditionally is killed by seven tests, and reading only `any[0]` by one.
The harness gains three probes so its naming requirement reaches the decisions
rather than the extractors beneath them:
ok offline verdict: report nothing, whatever the tree holds
FAIL: test_a_panel_naming_an_unwired_datasource_is_reported
FAIL: test_an_undeclared_template_variable_is_reported
FAIL: test_a_locally_authored_board_whose_json_cannot_be_read_is_reported
FAIL: test_a_declared_variable_nothing_references_reports_a_broken_parser
FAIL: test_every_detection_is_reported_from_one_pass
ok parity verdict: agree, whatever the four policies hold
FAIL: test_four_identical_lists_agree
FAIL: test_one_list_missing_a_namespace_is_a_mismatch
FAIL: test_one_list_carrying_an_extra_namespace_is_a_mismatch
FAIL: test_the_namespaces_returned_are_the_shared_baseline
FAIL: test_the_shipped_policies_agree
ok parity: read only the first exclude.any entry
FAIL: test_the_union_of_both_entries_is_the_list
RESULT pass=20 fail=0
Coverage is not the instrument for any of this and moves the wrong way when a
verdict is emptied: deleting an uncovered branch raises the ratio. The floors
ratchet anyway — combined 29 to 30, check-policy-admission 50 to 63,
validate-dashboards 45 to 49 — and run.py now records that
`security-critical-100` asks for a per-file 100% override on the approval gate
and that its floor sits at 63, so the rule is named with its requirement rather
than as though citing it satisfied it.
Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
`NOT_GATES` excuses a file from carrying a positive control on the ground that it
asserts its own outcome on every ordinary invocation. That is a claim about a
thing something runs. Neither `Taskfile.yaml` nor any workflow ran either
re-verification harness, so the exemption covered two executables that never
executed and its recorded reason was free text nothing read.
`check_vacuity` now fails an entry the Taskfile and the workflows do not name:
FAIL tests/reverify-gates.sh is exempted as a harness that asserts its own
outcome, but neither Taskfile.yaml nor a workflow under
.github/workflows/ runs it. A harness nothing invokes asserts nothing.
`scripts/tests/reverify-tests.sh` runs in `task validate` and in the `gate-tests`
job — offline, about eleven seconds. `scripts/tests/reverify-gates.sh` gets
`task reverify:gates` and stays out of the merge path: it runs `task validate`
twice and plants defects against gitleaks and trivy.
The corpus assertion on `discover()` was non-emptiness, which one unit satisfies
while the omission that walk produces is partial. It is now per file: every
ApplicationSet pinning a Helm chart must contribute a unit, and every matrix
element pinning one must appear among that appset's units. Three gates read this
corpus, and all three derive their printed counts from it.
Two prose corrections. `.coveragerc` and `run.py` claimed the control-exempt
gates are the largest files in the tree; `check-renovate-coverage.py` at 253
statements is larger than every one of them and is not exempt, so the claim is
gone rather than restated. `reverify-tests.sh` attributed the sed-portability
rationale to `reverify-gates.sh`, which does not record it — `controls.py` does.
The harness also prints its module count from `ALL_MODULES` instead of the
literal six.
Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
`check_vacuity` decides controls.py's exit code and nothing exercised it. Its
list rules are the ones the rest of this file's claim rests on — a gate on no
list, a control naming a gate that is gone, an exemption for a harness nothing
runs — and each was correct and asserted by nobody. That is the shape one level
down from the one the harness-invocation rule was added to reject.
The rules move to `vacuity_problems`, which takes the lists as arguments, so a
violation is planted by supplying them rather than by editing the repository the
harness is running inside. `self_test` plants all four and the accepting case:
── Vacuity-rule self-test ──
ok a gate on no list at all: check-x.py ships no positive control and is…
ok a control naming a gate that is gone: check-x.py is named by a control…
ok an exemption naming a harness that is gone: tests/gone.py is exempted…
ok a harness nothing invokes: tests/gone.py is exempted as a harness that…
ok a gate covered by a control: accepted
Gutting the invocation rule now fails there rather than passing:
FAIL a harness nothing invokes: accepted
Mutation contract FAILED its own self-test: 1 case(s).
Three repairs alongside it.
`run.py` stated the approval gate's floor as 50 where the table sets 63 — the
prose whose job is to keep `security-critical-100` from reading as satisfied
misreported the number it cites, and a reader auditing the gap would edit the
table down to match and unratchet the gate. The sentence now points at the table
and says why a figure written twice drifts.
The CI step gave a wall-clock figure as the reason for putting the harness in the
merge path. A measurement stated as documentation goes stale on the next probe
and on any machine but the one it was taken on; the step states its cost as a
property instead — one test module per probe, no render, no registry, no cluster.
`validate-dashboards.py` carried `return 1 if local else 0` directly after an
unconditional `return 2` inside the same block, reading as a second, softer exit
path the gate does not have.
Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
A harness exempted from carrying a positive control excuses itself by asserting its own outcome whenever it runs, so the exemption is worth what "whenever it runs" is worth. The reader holding that half searched the raw bytes of Taskfile.yaml and the workflows for the harness's path, which a comment naming the harness satisfies — and .github/workflows/ci.yml carries two such comments about scripts/tests/controls.py. ─── The reader ─── `invoked_anywhere` now asks the callers what they execute. Each caller is parsed, the commands are taken from the positions a runner takes them from — a task's `cmds:` entries, a workflow step's `run:` script — and each command is split into the words it would run, shell comments blanked and quoting resolved. A YAML comment is not in the parse; a `desc:`, a step `name:` and a quoted mention inside a command are not commands. `status:` and `preconditions:` are left out and the reason is recorded: they decide whether a task's work runs, so a harness reached only through one asserts nothing. Two limits are stated in the function, both toward reporting a harness as un-invoked rather than as invoked: a path assembled at run time from a variable is not resolved, and a word found in a command is not proved to be that command's executable. What the reader separates is prose from command text. A caller that will not parse is now its own finding. It and a caller that runs nothing are the same value to `any()` and different facts to whoever has to fix one, and the first used to print as four stale exemptions. ─── The proof ─── scripts/tests/test_controls.py plants a violation against the reader rather than against a stub standing in for it: seven callers that name the harness and do not run it, five that run it, and both directions of the unparseable case. Two more assert the rule reaches the reader by default — one harness this repo runs, one it does not — because a reader returning a constant passes either alone. scripts/tests/reverify-tests.sh reverts seven of those behaviours one at a time and requires the test that describes each to be the test that names it. Its floor moves to 27 and test_controls joins the modules it asserts green before and after. The quoted-mention fixture puts the path mid-string. At the end of one the closing quote stays attached under any splitter that ignores quoting, and the case is then answered correctly by a reader that cannot read quoting at all. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
Member
Author
|
The invocation reader is now behavioural, and it is asserted by planted violations rather than by a stub. What the reader does. Proof on the real tree, both directions. The two comments in Proof of the reader itself. Seven probes in One fixture was fixed by its own probe. The quoting probe first came back Stated limits, in the function. Both point toward reporting a harness as un-invoked rather than as invoked: a path assembled at run time from a variable is not resolved, and a word found in a command is not proved to be that command's executable — what the reader separates is prose from command text, not one word of a command from another. Gates: |
CI Results
All checks passed. |
#241 landed the vacuity harness, GATE_ARGS, gatelib.read_yaml across four gates, and corpus floors on check-platform-crs, check-policy-admission, validate-dashboards and kyverno-test.sh. This branch plants violations against those same offline verdicts and asserts the control-exempt gates, so the two work the same ground from opposite directions. Four textual conflicts, all of them both-sides-added rather than either-side-changed. Each is resolved by keeping both, in one order that the Taskfile and the workflow now share: .github/workflows/ci.yml — both branches inserted a step after the positive controls. `empty-corpus.py` asserts a GATE rejects an absent corpus and `reverify-tests.sh` asserts the TESTS reject a reverted behaviour, so the gate-level probe runs beside the other gate-level ones and the suite-level one after them. Taskfile.yaml, twice — the target definitions and the `validate` list, ordered to match the workflow so reading either gives the same sequence. scripts/tests/controls.py — both added a NOT_GATES entry. Both harnesses are exempt for the same reason and both keep their recorded reason. scripts/tests/run.py — both added to EXPECTED. `test_corpus_floors` joins the harness layer with a line saying what it holds. The comment above `test_controls` counted the exempt harnesses; it now states the rule instead, because the count was about to be wrong and would have stayed readable while being false. ──────────────── What the merge changed beyond the conflicts ──────────────── `empty-corpus.py` is a harness registered in NOT_GATES, so it enters the population of this branch's newest rule: a harness exempted for self-assertion must be invoked. It is — by `validate:empty-corpus` and by its own CI step — and the rule reaches it. Removing both commands while leaving the task named after it makes `invoked_anywhere` answer False and `check_vacuity` name the file with its recorded reason, so the coverage is real rather than incidental. `check-platform-crs.py` gained a `Cannot run` refusal when the operator ApplicationSet is absent, and its per-gate coverage floor read 49% against a floor of 50 — a floor doing exactly what a ratchet is for. Resolved by covering the verdict rather than lowering the number: `TheVersionThePinResolves` holds the four things `pinned_chart_version` decides, since the version it reads is upstream of every schema assertion in that module. Four probes in `reverify-tests.sh`, floor 27 → 31, each killed and named. Coverage 52%, floors raised to 51 and 31. `reverify-tests.sh` re-runs `test_corpus_floors` too. The floors #241 added sit in three gates this harness already mutates against, so a mutant reverting the floor half is held by that module and by nothing else in the list — without it, a caught mutant read as a miss. Its "all N modules green" line now says these are the modules the harness mutates against, which is what N was always counting. ──────────────── Verification on the merged tree ──────────────── ruff clean; mypy clean over 42 files; 297 tests across 15 modules; coverage ratchet 31.5% combined (floor 30 → 31), 12 per-gate floors held, 10 of 23 gates with no unit coverage against a ceiling of 11; controls.py 17 controls each rejecting at exit 1 naming the mutated file, so no planted violation trips a new floor first; empty-corpus.py 28 probed gates; reverify-tests.sh 31/31; task validate exit 0; check-named-things.py 187 references resolve; check-workflows.sh clean at MEDIUM and above; renovate-config-validator accepts the config. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
One conflict, in scripts/tests/run.py. Both sides changed the paragraph above COMBINED_FLOOR: main's states that the control-exempt gates have neither unit nor behavioural coverage, which is what this branch's tests made false. This branch's paragraph and its `control_exempt_gates` check — which reads the exemption list out of controls.py and fails on any gate in both it and the uncovered set — are what hold on the merged tree, so they stand. The floor itself is measured on the merge rather than taken from either side: 37, against a reading of 37.5%. #240's changes come in cleanly. `.node-version`, the nodenv manager, the reach assertion at every certification site, and the Renovate defaults record touch none of the files this branch changed. Verification on the merged tree: ruff clean; mypy clean over 43 files; 386 tests across 16 modules; coverage ratchet 37.5% (floor 31 → 37), 12 per-gate floors held, 10 of 23 gates with no unit coverage against a ceiling of 11, and all 7 control-exempt gates carrying unit coverage; controls.py 17 controls each rejecting at exit 1 naming the mutated file; empty-corpus.py 28 probed gates; reverify-tests.sh 31/31; reverify-gates.sh 35/35; task validate exit 0; check-named-things.py 191 references resolve; check-workflows.sh clean at MEDIUM and above; check-renovate-coverage.py and check-renovate-defaults.mjs both exit 0; renovate-config-validator accepts the config. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
stxkxs
marked this pull request as ready for review
September 5, 2026 00:59
stxkxs
added a commit
that referenced
this pull request
Sep 5, 2026
#238 landed controls.py's vacuity rules taking their lists as arguments, run.py's control-exempt coverage check, the per-file corpus assertion on discover(), and reverify-tests.sh in task validate. This branch reads the image population out of that same discover(), so the two meet there. ──────────────── scripts/check-image-pins.py ──────────────── Two hunks, both from #238 extracting the mutable-tag verdict out of main() into a testable `verdict(images, allowed)` and bringing a `bare_name` with it. The extraction is kept — it is what the tests #238 added call. The `bare_name` that came with it is dropped as a duplicate: this branch's strips the digest before looking for a tag, and #238's does not, so it would key `ghcr.io/x/y@sha256:<hex>` as `ghcr.io/x/y@sha256` and no exemption would match it. One reader, and it is the one that reads both forms. `verdict()` uses this branch's MUTABLE_REMEDIATION rather than the sentence it was extracted with, because a test reads the reference forms out of that constant and checks the pattern reads each one. A message retyped inside the function would drift from the pattern with nothing to notice. main() keeps this branch's shape — the per-chart floor, and the split between a chart that did not render and a reference that could not be placed — and calls verdict() for the mutable half. ──────────────── scripts/tests/run.py ──────────────── Four hunks. EXPECTED and PER_GATE_FLOORS are unions: both sides added entries, and both name check-image-pins.py, whose floor is measured on the merged tree that carries both sides' tests for it — 88 against a reading of 89%. COMBINED_FLOOR was raised from one base by both sides. A ratchet is measured, not chosen: 40, against 40.8%. MAX_UNCOVERED_GATES ratchets DOWNWARD, so the lower of the two holds: 11. This branch's paragraph stays with it, because what it records is still true — two of the files counted as covered are covered by IMPORT rather than by tests, since check-image-vulnerabilities.py loads check-image-pins.py by path and that loads render-addons.py in turn. ──────────────── One corpus, confirmed rather than assumed ──────────────── #238's per-file assertion requires every ApplicationSet pinning a chart to contribute a unit and every matrix element pinning one to appear among that appset's units. `inventory` walks exactly those units, and `chart_coverage` then requires each to have contributed an image. Passing over a reference helm reported pulling sits between the two. It cannot take a real contribution: a chart artifact is not an image the chart deploys and never entered the population. What it must not do is leave a chart looking covered while contributing nothing, and that is asserted rather than argued — a chart whose render carries only its own pulled coordinate contributes no image and the per-chart floor names it. Measured on the tree under the stream shape that engages the reader — helm's pull report on stdout, cold OCI cache: 32 units discovered, 28 contributing, no chart named by the inventory that discover() does not have, and chart_coverage empty. The reverify probe holds that shape durably. ──────────────── Verification on the merged tree ──────────────── ruff clean; mypy clean over 46 files; 476 tests across 18 modules; coverage ratchet 40.8% (floor 37 → 40), 13 per-gate floors held, 10 of 24 gates with no unit coverage against a ceiling of 11, and all 8 control-exempt gates carrying unit coverage; controls.py 17 controls; empty-corpus.py 29 probed gates; reverify-tests.sh 31/31; reverify-gates.sh 38/38; task validate exit 0; check-named-things.py 195 references resolve; check-workflows.sh clean at MEDIUM and above. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the standing REJECT that capped
testingat D: six gate scripts carryingneither unit coverage nor a positive control.
Which verdicts are proved, and which are only covered
A verdict is proved when a violation is planted and the assertion that names
it fails. It is covered when tests exercise it but nothing plants the
violation the verdict exists to catch — asserting that a gate reports nothing on
a healthy tree is a control against false positives, satisfied by
return [].check-policy-admission.judge/judge_runtime_podcheck-policy-admission.check_exclusion_parityvalidate-dashboards.check_local_dashboardscheck-image-pins.classify/verdictrender-addons.registry_answered/_path_from_valuefiles/oci_refcheck-log-volume-budget.environment_verdict/alert_thresholdcheck-platform-crs.walk(types, required, list identity)check-policy-admission.check_namespace_coveragecheck-policy-admission._prepare,_rule_keyvalidate-dashboardsextractors (template_vars,alert_panels,wired_datasource_refs)render-addons.discovercorpuscheck-image-vulnerabilities(on #239, not this branch)Twenty probes in
scripts/tests/reverify-tests.sh,RESULT pass=20 fail=0. Eachdeclares the test ids that must fail, so a mutant caught by an unrelated
assertion is a MISS rather than a pass.
What was uncovered
scripts/tests/controls.pyproves a gate rejects by planting the violation itnames. It cannot do that for a gate whose input arrives over a network, so
NEEDS_NETWORK_PYexempts seven. Six of those also carried no unit tests, whichleft 1,107 of 3,072 statements in the gate corpus with no proof of any kind:
check-policy-admission.pycheck-platform-crs.pyvalidate-dashboards.pyrender-addons.pydiscover()is the corpus two other gates import rather than re-derivecheck-image-pins.pycheck-log-volume-budget.pyThe assertion, not the number
scripts/tests/run.pygains a check with no percentage in it: every gatecontrols.pyexempts must carry unit coverage, with the exemption list read outof
controls.pyat run time rather than copied. Three ways it could stop meaninganything, each of which now fails:
NOT_GATESexcuses a harness on the ground that it asserts its own outcome onevery invocation — a claim about a thing something runs. Nothing ran either
harness, so that is now asserted too:
reverify-tests.shruns intask validateand thegate-testsjob (offline,~11s);
reverify-gates.shgetstask reverify:gatesand stays off the mergepath — it runs
task validatetwice.Coverage is the wrong instrument, and moves the wrong way
Gutting
check_exclusion_parityraised the figure from 29.5% to 29.6%: deletingan uncovered branch improves the ratio. The ratchet is a floor under the tests
that exist, not a check that the decisions are still made. It moves anyway —
combined 12 → 30,
check-policy-admission50 → 63,validate-dashboards45 →49, uncovered-gate ceiling 17 → 15 — but the planted violations above are what
the claim rests on.
Verification
All exit 0 on the branch head:
task validate— 25 gates (kyverno CLI at the CI pin v1.18.2)ruff check .;mypyover 39 filesCOVERAGE_REQUIRED=1 coverage run --rcfile=.coveragerc scripts/tests/run.py—271 tests across 13 modules, 30.7% combined, all 7 control-exempt gates covered
./scripts/tests/controls.py— 17 controls completed a proof./scripts/tests/reverify-tests.sh— pass=20 fail=0yamllint;./scripts/check-workflows.sh;./scripts/check-named-things.pyReview findings, disposition
.coveragercandrun.py;check-renovate-coverage.py(253) is larger and not exemptNOT_GATESreason unchecked, nothing invokes the harnessesdiscover()corpus floor is non-emptinesscontrols.pyrecords itNOT_GATESheader says "Both" over four entriesALL_MODULEStest_dashboardsdocstring asserts history with no operandsecurity-critical-100named without its requirementrun.pyrecords the rule as UNMET and names the gap: it asks 100%, the floor is 636e3a293and1ab7929One message defect of my own:
d5c0ea9's body credits itself with thereverify-tests.shprose fixes (finding 7 and 10), which actually landed ina8c1010because that file was staged there. The combined squash message will beaccurate; the per-commit attribution is not.
Final round — the four minors
run.pystated the approval gate floor as 50 where the table sets 63check_vacuitywas a verdict with no planted violationreturn 1 if local else 0under the dashboard corpus floorThe vacuity rules now plant against themselves
You planted the violation by hand and it fired; the branch was right and held by
nobody. The rules move to
vacuity_problems, which takes the lists as arguments,so a violation is supplied rather than edited into the repository the harness is
running inside.
self_testplants all four and the accepting case:Gutting the invocation rule fails there rather than passing:
That also closes what you named alongside it — the
NOT_GATESreason forcontrols.py, "self_test() runs on every invocation", covered everything exceptthe branch it was written beside.
Verification:
task validateexit 0;ruff;mypyover 39 files;controls.py17 controls with the new self-test;reverify-tests.shpass=20fail=0.