Repository navigation
fix(gates): make the red gates able to fail, and prove it - #52
Merged
Merged
Conversation
An estate-wide scan (>=30 lifetime runs, zero lifetime failures, taken from the API's own failure count) found this repo's 🔴 Gate: checks have never failed in 100-120 runs each. Reading them explains why: - 🔴 Gate: Guix-only package policy had NO exit path at all. It printed a warning for foreign lock files and always succeeded. A gate named 'Guix-only' enforced nothing. - 🔴 Gate: Security policy present detected three things and failed on one. Weak crypto (MD5/SHA1) and plaintext HTTP printed a warning and passed. - 🔴 Gate: Code quality had continue-on-error on the EditorConfig step. These are the repos that adopted the tier taxonomy most carefully, which makes a fake gate here worse than elsewhere: the red label suppresses the scrutiny that would otherwise find it. Changes: weak crypto and plaintext HTTP now fail the security gate; a foreign lock file now fails the Guix-only gate; EditorConfig no longer continues on error. And per standards R10 (canonical wrongness), a gate that has never failed is indistinguishable from one that CANNOT fail, so the claim is now demonstrated rather than asserted: scripts/test-gate-falsifiability.sh plants deliberately-wrong input in a scratch tree for each detection and asserts it is REJECTED, plus a fifth canary asserting clean input is NOT rejected (a suite that always fires proves nothing). Wired into the quality gate. Worth recording: the first version of the plaintext-HTTP canary used example.net and did NOT fire, because the gate excludes anything containing 'example'. The fixture was not actually wrong — exactly the silent failure R10 exists to catch, found by the canary on its first run. Noted in-file as a caution. EXPECT RED. These gates have been passing because they could not fail; some of what they now catch will be real and long-standing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
Note Automatic reviews are paused because your trial's included automatic processing has been used for this period. Upgrade now, or comment "Gitar review" to run a review anytime. Code Review ✅ ApprovedFixes red CI gates that previously lacked exit paths or suppressed failures, and adds falsifiability canary tests to ensure they catch bad inputs. No issues found.
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Important Your trial ends in 5 days — upgrade now to keep code review, CI analysis, auto-apply, custom automations, and more. Was this helpful? React with 👍 / 👎 | Gitar |
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
…cope `trap ... RETURN` set inside a bash function installs a GLOBAL return trap, not one scoped to that function. Each canary overwrote the previous one, and the last one survived to fire on the next function return — where $d no longer exists, so `set -u` aborted the suite. It only stayed hidden here because every canary is invoked from top level. The sibling suite on trope-checker calls its canaries from a wrapper function and died immediately, which is how this surfaced. Replaced with explicit cleanup after the value under test is captured. The suite now exits 0 with every canary firing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
This PR turned Hypatia's critical-findings gate red, and it was right to. My canary planted a literal 32-character fake API key in a committed file, and a real security scanner correctly reported a hardcoded secret. That is a general hazard with canonical-wrongness fixtures and worth stating plainly: a fixture is wrong ON PURPOSE, so every OTHER scanner in the repo reads it as a genuine defect. Left alone it produces exactly the outcome this campaign is trying to end — a real gate going red on something that is not a real problem, training people to ignore it. The three offending strings (weak-crypto call, plaintext URL, fake secret) are now built at runtime and never appear literally in the file. The fixtures are still genuinely wrong when the gate sees them. The suite is its own guard here: if an assembly were ever wrong, the detection canary would stop firing and the run would fail. That the canaries still pass is the evidence the strings are still wrong — which is the whole point of R10, applied to the R10 harness itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
…tired it The last continue-on-error in this quality gate was on a trufflehog step, which I flagged for a ruling rather than deciding. Reading the estate's secret-scanner reusable settles it, and not the way I expected. Its header records the decision already taken: "Rationale for gitleaks over trufflehog: ... Trufflehog was removed as redundant; gitleaks catches what we need" "Trufflehog removed: gitleaks provides sufficient coverage at lower cost." So this is not a coverage trade-off. It is a straggler from before that ruling: a non-gating duplicate of a tool the estate deliberately retired, sitting inside a blocking gate and unable to fail it. The concern I raised was also unfounded. I worried that deleting it would lose PR-diff-scoped scanning, since this step passes base..head. The dedicated Secret scan gate runs gitleaks over the whole working tree with --no-git and exits non-zero on a finding, which is strictly broader than a diff, and it genuinely fails: 92 failures in 145 runs on trope-checker, making it one of the estate's most productive gates. Net effect: the quality gate now has no step that cannot fail it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
… above it Removing trufflehog from actions.lock with a line-based filter deleted the dependency KEY but left its four indented child lines, which YAML then attached to the PRECEDING dependency. Every Secret scan gate went startup_failure immediately. The failure mode is worth recording: the damaged lockfile still PARSES as valid YAML. Nothing local catches it. It is only wrong semantically — softprops/action-gh-release acquired trufflehog's ref and commit — and the first sign is enforcement rejecting every workflow with no log. Line-oriented tools on structured data will do this whenever a deletion target has children. Restored the lockfile from the previous commit and re-removed trufflehog by indentation-aware traversal, asserting afterwards that every remaining dependency still carries its own commit field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
|
hyperpolymath
added a commit
to hyperpolymath/trope-particularity-workbench
that referenced
this pull request
Aug 5, 2026
…ld not be red (#45) Five checks here carry the `🔴 Gate:` prefix and have **never failed in 71–102 runs each** (from the API's own lifetime failure count, not a sample). Reading them splits the five two ways — which is the useful result, because "never failed" on its own does not distinguish a broken gate from clean code. ## Three were fake Same template as `haec` — the files differ only in action-pin style (`haec` uses `actions.lock`, this repo inline SHAs), so the gate logic is byte-identical and the same fixes apply. Companion PR: hyperpolymath/haec#52. | Gate | What it actually did | |---|---| | `🔴 Gate: Guix-only package policy` | **No `exit 1` anywhere.** A foreign lock file printed⚠️ and passed. | | `🔴 Gate: Security policy present` | Failed on secrets **only**. Weak crypto (MD5/SHA1) and plaintext HTTP printed⚠️ and passed. | | `🔴 Gate: Code quality (fmt/lint/test)` | `continue-on-error: true` on the EditorConfig step. | ## One was mislabelled rather than broken `🔴 Gate: Static analysis` **cannot fail three separate ways** — it skips when its tools are absent, pipes findings through `|| true`, and carries `continue-on-error`. Two honest resolutions existed: make it able to fail, or stop calling it a gate. **The second is correct here.** `panic-attack` has a known false-positive problem (panic-attack#171), much of it firing on tests and benches rather than shipping code. Making that blocking would teach people to route around the gate, which is worse than the fake gate it replaces. Retiered to **`🟡 CHECK: Static analysis`**. Its header claimed *"Required by branch protection rules."* **It is in no ruleset** — verified against the API before renaming, because renaming a genuinely-required context strands it as a phantom that silently blocks every PR. (This repo's actual required contexts are governance, Hypatia, SonarCloud and `analyze (actions, none)`.) The false comment is corrected. ### The skips are no longer silent `::notice::` does not surface on the checks board, so **an absent scanner looked identical to a clean scan**. They are now `::warning::`, and the step summary reads **DID NOT RUN** rather than "Skipped" — absence of evidence, not evidence of absence. The skip paths themselves stay, which is defensible now the check is advisory. ## One was already honest `🔴 Gate: Trope conformance` accumulates `fail=1` across every consistency assertion and exits with it. Never failing here was **clean code, not a broken gate**. It needed proof, not repair — see below. ## The proof, not the claim — standards R10 A gate that has never failed is **indistinguishable from a gate that cannot fail**. `scripts/test-gate-falsifiability.sh` settles it: each canary plants deliberately-wrong input and asserts the gate **rejects** it — the pass condition inverted. ``` PASS weak crypto (md5) is detected PASS plaintext HTTP is detected PASS hardcoded secret is detected PASS foreign lock file is detected PASS trope conformance rejects a missing vocabulary entry PASS trope conformance accepts the unmodified tree (fixture copy is complete) gate falsifiability canaries: 6/6 ``` Two details worth the reading time: **Canary 5 runs the real gate.** `tests/check-vocabulary.sh` derives its root from its own location, so the canary copies the tree to a temp directory, deletes `vocabulary/p-fusing.adoc`, and runs the actual script. A canary that *re-encodes* a gate's logic proves only that it agrees with itself. **Canary 6 is load-bearing, not decoration.** Running the same real gate on a *pristine* copy guards the opposite error — a suite that always fires proves nothing — and it also proves the fixture copy is complete. If `tree_copy` omitted a file the gate reads, canary 5 would pass **for the wrong reason** and nothing else would notice. The suite runs inside the quality gate, so falsifiability is re-checked on every PR rather than proved once and left to drift. ## Expect red These gates have been passing **because they could not fail**. Some of what they now catch will be real and long-standing — plaintext HTTP URLs and MD5 usages that predate this change. That is success, not regression, but worth expecting: the instinct when a 100-run-green gate turns red is to assume the change broke it. ## One left for your decision `quality.yml` still has `continue-on-error: true` on **trufflehog**, which is duplicated by this repo's dedicated `secret-scanner.yml`. Making a secret scanner blocking has real false-positive consequences, and deleting a security scan is equally consequential, so I changed neither. Same open question as haec#52. Found during the 2026-08-05 estate CI/CD census. Companion standards PRs: #576 (CI signal discipline) and #577 (R10 canonical wrongness). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Correction to the evidence above, and a second fix Opening this PR revealed the repo was **repo-wide dead**: all 20 workflows returned `startup_failure` in 0s, so the only check that ran was SonarCloud. Workflow-lockfile enforcement is active on this account and this repo had no `.github/workflows/actions.lock`; `haec` and `trope-checker` have one, and their workflows run. Added via `gh actions-lock` (which also normalised the inline SHA pins to tag refs recorded in the lockfile — my gate edits survive that rewrite unchanged). **This has a consequence for the finding that opened this PR, and it cuts against me.** `startup_failure` is *not* counted by the API's `?status=failure` query. So a repo whose CI never starts looks **identical** to a repo whose gates always pass — and the scan that flagged these gates used exactly that query. Checking each gate's actual run history: | Workflow | success | startup_failure | verdict | |---|---:|---:|---| | `quality.yml` | 14 | **86** | "102 runs, 0 failures" **substantially overstated** — it mostly wasn't running, going back to June | | `security-policy.yml` | 95 | 2 | ran, and never failed | | `guix-nix-policy.yml` | 95 | 2 | ran, and never failed | | `static-analysis-gate.yml` | 98 | 2 | ran, and never failed | So three of the four gates are exactly as described: they genuinely executed ~95–98 times and never once failed, because they could not. The Code quality claim is the one to discount — it was *both* fake (the `continue-on-error`) *and* largely not running. Both are fixed here, but the run-count evidence for it was weaker than I stated. The `quality.yml` startup failures predate today's repo-wide death (they run back to June while the others show only 2), so that workflow has a separate, older problem which `actions.lock` may not resolve. Flagged rather than assumed fixed — worth re-checking once this PR's CI reports. **Estate-wide implication:** the fake-gate census needs re-running with `startup_failure` counted, not just `failure`. Recorded in `dev-notes/cicd-fake-gate-classification-2026-08-05.md`. --------- Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
hyperpolymath
added a commit
to hyperpolymath/trope-checker
that referenced
this pull request
Aug 5, 2026
Four `🔴 Gate:` checks here have **never failed**: quality 135/0, security policy 135/0, static analysis 132/0, well-known 50/0. The estate scan reported only *one* of these — queried directly, **it under-counted**, which is worth knowing before anyone treats that scan's totals as complete. The correction is recorded in `dev-notes/cicd-fake-gate-classification-2026-08-05.md`. The reassuring half: this repo's **other** red gates fail regularly — proofs 18/33, trope conformance 21/118, secret scan 92/145. That is what a real gate looks like, and it is why "never failed" is a candidate signal rather than a verdict. Reading the four splits them three ways. ## Two were fake Same template as `haec` and `trope-particularity-workbench` (the files differ only in action-pin style). Companion PRs: hyperpolymath/haec#52, hyperpolymath/trope-particularity-workbench#45. | Gate | What it actually did | |---|---| | `🔴 Gate: Security policy present` | Failed on secrets **only**. Weak crypto (MD5/SHA1) and plaintext HTTP printed⚠️ and passed. | | `🔴 Gate: Code quality (fmt/lint/test)` | `continue-on-error: true` on the EditorConfig step. | ## One was mislabelled rather than broken `🔴 Gate: Static analysis` **cannot fail three separate ways** — skip-on-missing-tool, `|| true`, and `continue-on-error`. Retiered to **`🟡 CHECK: Static analysis`** rather than forced blocking. It wraps `panic-attack`, which has a known false-positive problem (panic-attack#171) concentrated in tests and benches rather than shipping code; making that blocking would teach people to route around gates, which is worse than the fake gate it replaces. Its header claimed *"Required by branch protection rules."* **It is in no ruleset** — checked against the API before renaming, because renaming a genuinely-required context strands it as a phantom that silently blocks every PR. While verifying that, I checked all five of this repo's required contexts against what a real PR actually emits. **All five report** — no phantoms here. (I had expected the bare `CodeQL` requirement to be one, since normally only GitHub's default setup emits that string and default setup is `not-configured` here. It reports because a *job* is literally named `CodeQL`. Worth recording: that heuristic is a generalisation, not a rule.) ### The skips are no longer silent `::notice::` never reaches the checks board, so **an absent scanner looked identical to a clean scan**. Now `::warning::`, with the step summary reading **DID NOT RUN** — absence of evidence, not evidence of absence. ## One was honest but unproven — and hiding a real hole `🔴 Gate: .well-known / RFC 9116` has genuine `exit 1` paths. Never failing was clean input, not a broken gate. Its logic moved to `scripts/check-security-txt.sh`, so the gate and its canaries run **the same code**. That matters: a canary that re-implements a gate's logic proves only that it agrees with itself. Extracting it exposed a defect that had been live the whole time: ```bash EXPIRES=$(grep "^Expires:" "$SECTXT" | ...) if date -d "$EXPIRES" > /dev/null 2>&1; then ... expiry checks ... fi # <-- no else. Unparseable date => every expiry check skipped => PASS ``` If `date -d` could not parse the `Expires:` value, the expiry check was **skipped entirely and the gate passed**. RFC 9116 requires an ISO 8601 timestamp, so an unreadable one is a violation, not an exemption. Now rejected, with a canary pinning it so it cannot silently return. ## The proof, not the claim — standards R10 ``` PASS no security.txt is rejected PASS security.txt without Contact is rejected PASS security.txt without Expires is rejected PASS an EXPIRED security.txt is rejected PASS an UNPARSEABLE Expires is rejected (was silently accepted) PASS a valid security.txt is accepted PASS this repo's own security.txt passes the gate PASS weak crypto (md5) is detected PASS plaintext HTTP is detected PASS hardcoded secret is detected gate falsifiability canaries: 10/10 ``` Two of the ten are deliberately *not* rejection tests, and both earn their place: a **valid** security.txt must be accepted (a suite that always fires proves nothing), and **this repo's own** security.txt must pass its own gate (a gate is only meaningful if the repo it guards satisfies it). The suite runs inside the quality gate, so falsifiability is re-checked every PR rather than proved once and left to drift. ### The harness had a bug of its own Worth naming, because it is a general bash trap: `trap … RETURN` inside a function installs a **global** return trap, not a function-scoped one. The leftover trap fired on the *next* function's return, where its `$d` no longer existed, and `set -u` killed the suite. It stayed invisible in the sibling repos because their canaries are all called from top level; this suite calls them from a wrapper and died immediately. Replaced with explicit cleanup, and the fix backported to haec#52 and tpw#45. ## Expect red These gates passed **because they could not fail**. Some of what they now catch will be real and long-standing. Found during the 2026-08-05 estate CI/CD census. Companion standards PRs: #576 (CI signal discipline) and #577 (R10 canonical wrongness). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An estate-wide scan — every workflow with ≥30 lifetime runs and zero lifetime failures, taken from the API's own failure count rather than a sample or a proxy — found this repo's
🔴 Gate:checks have never failed in 100–120 runs each.Reading them explains why.
🔴 Gate: Guix-only package policyexit 1anywhere. Printed a🔴 Gate: Security policy present🔴 Gate: Code quality (fmt/lint/test)continue-on-error: trueon the EditorConfig step.This repo and
trope-particularity-workbenchadopted the tier taxonomy most carefully — which makes a fake gate here worse than elsewhere. The red label suppresses the scrutiny that would otherwise find it.Changes
FAILED=truelike the secrets check already did).The proof, not the claim — standards R10
A gate that has never failed is indistinguishable from a gate that cannot fail. So
scripts/test-gate-falsifiability.shsettles it: for each detection it plants deliberately-wrong input in a scratch tree and asserts the gate rejects it — the pass condition inverted, green meaning "the bad thing was correctly caught".A fifth canary asserts that clean input is not rejected, guarding the opposite error: a canary suite that always fires proves nothing.
Wired into the quality gate, so the gates stay falsifiable rather than being proved once and drifting.
The canary earned its place on its first run
The initial plaintext-HTTP fixture used
http://data.example.net/and did not fire — because the gate excludes anything containingexample. The fixture was not actually wrong.That is exactly the silent failure R10 exists to catch: a negative test whose wrongness has evaporated passes green forever, and nothing in the surface text signals it. Caught here by the canary, on its first execution, before it could become load-bearing. Recorded in-file as a caution.
Expect red
These gates have been passing because they could not fail. Some of what they now catch will be real and long-standing — plaintext HTTP URLs and MD5 usages that predate this change. That is success, not regression, but it is worth expecting: the instinct when a long-green gate turns red is to assume the change broke it.
One left for your decision
quality.ymlstill hascontinue-on-error: trueon trufflehog — which is duplicated by this repo's dedicatedsecret-scanner.yml. Making a secret scanner blocking is a real decision with false-positive consequences, and removing a security scan is equally consequential, so I have changed neither. Options: drop it here and rely on the dedicated workflow, or make it blocking.Found during the 2026-08-05 estate CI/CD census. Companion standards PRs: #576 (CI signal discipline) and #577 (R10 canonical wrongness).
🤖 Generated with Claude Code