Repository navigation
fix(gates): make the red gates able to fail, and prove it - #64
Conversation
Four 🔴 Gate: checks here have never failed — quality 135/0, security policy 135/0, static analysis 132/0, well-known 50/0. (The estate scan reported only one of these; queried directly, it under-counted. The other three red gates DO fail regularly — proofs 18/33, trope conformance 21/118, secret scan 92/145 — which is what a real gate looks like.) TWO WERE FAKE — same template as haec and trope-particularity-workbench, differing only in action-pin style: - 🔴 Gate: Security policy present failed on secrets only; weak crypto (MD5/SHA1) and plaintext HTTP printed a warning and passed. - 🔴 Gate: Code quality had continue-on-error on the EditorConfig step. ONE WAS MISLABELLED — 🔴 Gate: Static analysis cannot fail three ways (skip-on-missing-tool, || true, continue-on-error). Retiered to 🟡 CHECK: rather than forced blocking, because it wraps panic-attack, which has a known false-positive problem (panic-attack#171) concentrated in tests and benches. Blocking on it would teach people to bypass gates. Its header claimed 'Required by branch protection rules'; it is in no ruleset — verified against the API first, since renaming a genuinely required context strands it as a phantom that blocks every PR. The skip paths now emit ::warning:: and say DID NOT RUN, because ::notice:: never reaches the checks board and an absent scanner looked identical to a clean one. ONE WAS HONEST BUT UNPROVEN — 🔴 Gate: .well-known / RFC 9116 has real exit paths. Never failing was clean input, not a broken gate. Its logic moved to scripts/check-security-txt.sh so the gate and its canaries run the SAME code; a canary that re-implements a gate proves only that it agrees with itself. Extracting it exposed a real hole: if `date -d` could not parse the Expires value, the expiry check was skipped entirely and an unparseable date passed. RFC 9116 requires ISO 8601, so unreadable is a violation, not an exemption. Now rejected, with a canary pinning it. scripts/test-gate-falsifiability.sh: 10 canaries, each planting deliberately-wrong input and asserting REJECTION. Two are controls — a valid security.txt must be ACCEPTED (a suite that always fires proves nothing), and this repo's own security.txt must pass its own gate. EXPECT RED. These gates passed because they could not fail. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
| c_weak_crypto() { | ||
| local d; d=$(scratch) | ||
| printf 'fn h() { md5(x) }\n' > "$d/lib.rs" | ||
| local hit | ||
| hit=$(cd "$d" && grep -rE 'md5\(|sha1\(' --include="*.rs" . 2>/dev/null \ | ||
| | grep -v 'checksum\|cache\|test\|spec' | head -5 || true) | ||
| rm -rf "$d" | ||
| if [ -n "$hit" ]; then ok "weak crypto (md5) is detected" | ||
| else bad "weak crypto NOT detected — the security gate is blind to it"; fi | ||
| } | ||
|
|
||
| # NOTE: the fixture host must avoid every exclusion term the gate applies | ||
| # (localhost, 127.0.0.1, example, test, spec). The first version of this canary | ||
| # used example.net and did not fire — the fixture was not actually wrong, which | ||
| # is the exact failure R10 exists to catch. Kept as a caution. |
There was a problem hiding this comment.
⚠️ Quality: Security-policy canaries re-implement gate logic, can drift
Unlike the well-known canaries, which invoke the real extracted script (check-security-txt.sh), the c_weak_crypto/c_plaintext_http/c_secret canaries copy the grep regexes inline instead of running the actual security-policy.yml gate. If someone edits a pattern in security-policy.yml (e.g. adds an extension or tightens a regex), these canaries keep testing the old copied pattern and still report green — proving only that the canary agrees with itself, the exact failure mode standards R10 (cited in this PR) exists to catch. Extract the three security-policy detections into a shared script (as was done for security.txt) and have both the gate and the canaries call it.
Was this helpful? React with 👍 / 👎
| echo "installed=true" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "::notice::panic-attack binary not available — skipping assail" | ||
| echo "::warning::panic-attack binary NOT AVAILABLE — assail DID NOT RUN. Empty findings below mean "not scanned", not "clean"." |
There was a problem hiding this comment.
💡 Quality: Nested double quotes collapse in warning strings
The echo strings embed unescaped double quotes inside a double-quoted argument, e.g. mean "not scanned", not "clean". Bash concatenates the fragments and strips the inner quotes, so the emphasis quotes vanish from the emitted ::warning:: text (three occurrences: assail, Hypatia, Patch Bridge steps). It does not break the workflow, only the intended rendering. Escape them (mean "not scanned") or use single quotes for the inner emphasis.
Was this helpful? React with 👍 / 👎
There was a problem hiding this comment.
Configure merge blocking · Maintainers can dismiss this review.
|
Note Automatic reviews are paused because your trial's included automatic processing has been used for this period. Upgrade now, or comment "Gitar review" to run a review anytime. Code Review
|
| Compact |
|
Important
Your trial ends in 5 days — upgrade now to keep code review, CI analysis, auto-apply, custom automations, and more.
Was this helpful? React with 👍 / 👎 | Gitar
This PR turned Hypatia's critical-findings gate red, and it was right to. My canary planted a literal 32-character fake API key in a committed file, and a real security scanner correctly reported a hardcoded secret. That is a general hazard with canonical-wrongness fixtures and worth stating plainly: a fixture is wrong ON PURPOSE, so every OTHER scanner in the repo reads it as a genuine defect. Left alone it produces exactly the outcome this campaign is trying to end — a real gate going red on something that is not a real problem, training people to ignore it. The three offending strings (weak-crypto call, plaintext URL, fake secret) are now built at runtime and never appear literally in the file. The fixtures are still genuinely wrong when the gate sees them. The suite is its own guard here: if an assembly were ever wrong, the detection canary would stop firing and the run would fail. That the canaries still pass is the evidence the strings are still wrong — which is the whole point of R10, applied to the R10 harness itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
CI status — all four fixed gates green, four reds are pre-existing
Four checks remain red, all failing on
None are touched here. One thing went wrong, and it is now part of the standardMy canary planted a literal 32-character fake API key in a committed file, and a real secret-scanning gate correctly reported it. A canonical-wrongness fixture is wrong on purpose, so every other scanner reads it as a genuine defect — producing precisely the outcome this campaign exists to end. The strings are now constructed at runtime. Written up as R10.7 in metadatastician/standards#577, together with R10.6 (a suite must also assert valid input is accepted — a fixture whose wrongness evaporates passes silently forever) and R10.8 (exercise the real check, not a reimplementation — which is how the |
…tired it The last continue-on-error in this quality gate was on a trufflehog step, which I flagged for a ruling rather than deciding. Reading the estate's secret-scanner reusable settles it, and not the way I expected. Its header records the decision already taken: "Rationale for gitleaks over trufflehog: ... Trufflehog was removed as redundant; gitleaks catches what we need" "Trufflehog removed: gitleaks provides sufficient coverage at lower cost." So this is not a coverage trade-off. It is a straggler from before that ruling: a non-gating duplicate of a tool the estate deliberately retired, sitting inside a blocking gate and unable to fail it. The concern I raised was also unfounded. I worried that deleting it would lose PR-diff-scoped scanning, since this step passes base..head. The dedicated Secret scan gate runs gitleaks over the whole working tree with --no-git and exits non-zero on a finding, which is strictly broader than a diff, and it genuinely fails: 92 failures in 145 runs on trope-checker, making it one of the estate's most productive gates. Net effect: the quality gate now has no step that cannot fail it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
… above it Removing trufflehog from actions.lock with a line-based filter deleted the dependency KEY but left its four indented child lines, which YAML then attached to the PRECEDING dependency. Every Secret scan gate went startup_failure immediately. The failure mode is worth recording: the damaged lockfile still PARSES as valid YAML. Nothing local catches it. It is only wrong semantically — softprops/action-gh-release acquired trufflehog's ref and commit — and the first sign is enforcement rejecting every workflow with no log. Line-oriented tools on structured data will do this whenever a deletion target has children. Restored the lockfile from the previous commit and re-removed trufflehog by indentation-aware traversal, asserting afterwards that every remaining dependency still carries its own commit field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
⚙ Auto: BoJ server build trigger has failed every run, with zero jobs each
time. The reason is a job-level guard:
if: ${{ vars.BOJ_SERVER_URL != '' || secrets.BOJ_SERVER_URL != '' }}
GitHub rejects this outright — "Unrecognized named-value: 'secrets'". The
secrets context is not available in a job-level if:, only in env:, with:,
run: and reusable-workflow inputs. So the file has never been a valid
workflow, and the run fails during expression evaluation before any job
exists. That is why it produces no log and no check run: there is nothing
to log.
The guard is also redundant. The step already sets
env: BOJ_URL: ${{ secrets.BOJ_SERVER_URL || vars.BOJ_SERVER_URL }}
which IS a legal use of the secrets context, and then exits 0 early when
BOJ_URL is empty. Deleting the broken guard restores exactly the intended
behaviour: a clean skip when the server is not configured.
This is not a local defect. 51 repositories in the estate carry the same
line, which means 51 mains have been permanently red on a workflow that
has never once executed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
🟡 Check: E2E + aspect + bench has failed 20 times out of 20, with zero jobs each time, because it was never instantiated. Its own header says so: INSTRUCTIONS: Uncomment and customise the section matching your stack. Every job block is still commented out, so the file declares `jobs:` with nothing under it. yaml.safe_load returns None for that key. A workflow with no jobs is invalid, which is why every run fails before producing a job or a log. It claimed coverage of four merge-requirement categories — E2E, aspect, benchmarks, and Component Readiness Grade — and delivered none of them. That is worse than absence: a repo reading its own workflow list would conclude those categories were handled. Deleted rather than left red. What real coverage exists lives in rust-ci.yml and proofs.yml, and both genuinely fail when they should — proofs 18 failures in 33 runs, trope conformance 21 in 118. This is not local: 29 repositories carry the same un-instantiated file, including rsr-template-repo, which is where minted repos inherit it from. Intent recorded in an issue so it is not lost with the noise. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
|
Four
🔴 Gate:checks here have never failed: quality 135/0, security policy 135/0, static analysis 132/0, well-known 50/0.The estate scan reported only one of these — queried directly, it under-counted, which is worth knowing before anyone treats that scan's totals as complete. The correction is recorded in
dev-notes/cicd-fake-gate-classification-2026-08-05.md.The reassuring half: this repo's other red gates fail regularly — proofs 18/33, trope conformance 21/118, secret scan 92/145. That is what a real gate looks like, and it is why "never failed" is a candidate signal rather than a verdict.
Reading the four splits them three ways.
Two were fake
Same template as
haecandtrope-particularity-workbench(the files differ only in action-pin style). Companion PRs: hyperpolymath/haec#52, hyperpolymath/trope-particularity-workbench#45.🔴 Gate: Security policy present🔴 Gate: Code quality (fmt/lint/test)continue-on-error: trueon the EditorConfig step.One was mislabelled rather than broken
🔴 Gate: Static analysiscannot fail three separate ways — skip-on-missing-tool,|| true, andcontinue-on-error.Retiered to
🟡 CHECK: Static analysisrather than forced blocking. It wrapspanic-attack, which has a known false-positive problem (panic-attack#171) concentrated in tests and benches rather than shipping code; making that blocking would teach people to route around gates, which is worse than the fake gate it replaces.Its header claimed "Required by branch protection rules." It is in no ruleset — checked against the API before renaming, because renaming a genuinely-required context strands it as a phantom that silently blocks every PR.
While verifying that, I checked all five of this repo's required contexts against what a real PR actually emits. All five report — no phantoms here. (I had expected the bare
CodeQLrequirement to be one, since normally only GitHub's default setup emits that string and default setup isnot-configuredhere. It reports because a job is literally namedCodeQL. Worth recording: that heuristic is a generalisation, not a rule.)The skips are no longer silent
::notice::never reaches the checks board, so an absent scanner looked identical to a clean scan. Now::warning::, with the step summary reading DID NOT RUN — absence of evidence, not evidence of absence.One was honest but unproven — and hiding a real hole
🔴 Gate: .well-known / RFC 9116has genuineexit 1paths. Never failing was clean input, not a broken gate.Its logic moved to
scripts/check-security-txt.sh, so the gate and its canaries run the same code. That matters: a canary that re-implements a gate's logic proves only that it agrees with itself.Extracting it exposed a defect that had been live the whole time:
If
date -dcould not parse theExpires:value, the expiry check was skipped entirely and the gate passed. RFC 9116 requires an ISO 8601 timestamp, so an unreadable one is a violation, not an exemption. Now rejected, with a canary pinning it so it cannot silently return.The proof, not the claim — standards R10
Two of the ten are deliberately not rejection tests, and both earn their place: a valid security.txt must be accepted (a suite that always fires proves nothing), and this repo's own security.txt must pass its own gate (a gate is only meaningful if the repo it guards satisfies it).
The suite runs inside the quality gate, so falsifiability is re-checked every PR rather than proved once and left to drift.
The harness had a bug of its own
Worth naming, because it is a general bash trap:
trap … RETURNinside a function installs a global return trap, not a function-scoped one. The leftover trap fired on the next function's return, where its$dno longer existed, andset -ukilled the suite. It stayed invisible in the sibling repos because their canaries are all called from top level; this suite calls them from a wrapper and died immediately. Replaced with explicit cleanup, and the fix backported to haec#52 and tpw#45.Expect red
These gates passed because they could not fail. Some of what they now catch will be real and long-standing.
Found during the 2026-08-05 estate CI/CD census. Companion standards PRs: #576 (CI signal discipline) and #577 (R10 canonical wrongness).
🤖 Generated with Claude Code