Skip to content

fix(deploy): replace the pasted stopgap runbook with a fail-closed script - #466

Merged
hyperpolymath merged 10 commits into
mainfrom
claude/zen-brahmagupta-opqqle
Nov 10, 2026
Merged

hyperpolymath merged 10 commits into
mainfrom
claude/zen-brahmagupta-opqqle

Conversation

@hyperpolymath

Copy link
Copy Markdown
Owner

Summary

Follow-up to #461 (merged). Before it merged, CodeRabbit's second review left five findings, and its autofix commits fixed two of them. The autofix also introduced a defect: it added || exit 1 to console blocks that the operator pastes into an SSH shell, so any failed step would log the operator out of the host partway through containment.

Three findings were still open on main:

  • step 5 relied on $RUNNING, a variable set in an earlier shell;
  • the §4 end-to-end example still POSTs to /api/verify as a success check;
  • §5 tells the operator to re-enable the auto-update timer, which does nothing once AutoUpdate=registry is gone.

Pasted multi-step console blocks keep producing failure modes like these, so this PR replaces them with one tested script.

Changes

  • New deploy/hetzner/stopgap-apply.sh (apply DIR | verify, set -euo pipefail). It stops before restarting anything when a step cannot be confirmed:

    • checks that the source files are the stopgap versions (Pull=never, no AutoUpdate, the @public matcher);
    • requires exactly one install layout (quadlet or appendix), and changes nothing otherwise;
    • disables podman-auto-update.timer;
    • records the running container's image ID in ~/.local/state/echidna/stopgap-running-image, and re-points :latest at it if the tag has moved;
    • installs the files and restarts the right units: echidna caddy on a quadlet host, container-caddy on an appendix host;
    • verifies the same image, the timer disabled, /api/health 200 and POST /api/verify 403.

    verify reads the recorded ID, so it works from any later shell.

  • New tests/deploy/stopgap-apply.test.sh runs the script against stub podman, systemctl and curl. It has 29 cases, wired into the MVP Smoke workflow and just test-deploy.

  • README.adoc: the stopgap section documents the script; §4 marks the expected 403; §5 explains that restoring rollout means putting AutoUpdate=registry back, which ends the stopgap and belongs to the auth change.

RSR Quality Checklist

Required

  • Tests pass: 29/29.
  • Linter is clean: shellcheck clean on both scripts; asciidoctor -v reports 0 warnings.
  • No banned language patterns
  • SPDX headers present (MPL-2.0 on both new scripts)
  • No secrets

As Applicable

  • Documentation updated

Testing

Check Result
bash tests/deploy/stopgap-apply.test.sh 29 passed, 0 failed
Mutation: remove the one-layout check / the same-image check / the 403 check 6 / 1 / 1 cases fail. Each mutation is caught.
shellcheck on both files clean
ruby -ryaml parse of mvp-smoke.yml ok (no new uses:, so actions.lock is unaffected)
tests/shell/validate_scripts.sh 10 failures, identical on main, none in files this PR touches

Not run: the script against the live host, and the real caddy-rl:2 image.


Generated by Claude Code

cargo audit fails on rmcp 3.5.0 (crates/echidna-mcp) with
RUSTSEC-2026-0457 (fixed >=3.7.0), RUSTSEC-2026-0461 (>=3.6.0) and
RUSTSEC-2026-0462 (>=3.9.0), published after main last passed the
Dependency audit check. `cargo update -p rmcp` moves rmcp and
rmcp-macros to 3.9.0, inside the existing "3" requirement; no other
entry changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019a95kk88TZ1iegsBK3kox3
`cargo fmt --check` fails on main (2ab7574) in five places in
src/rust/proof_search.rs, so rust-ci 'Cargo check + clippy + fmt' is
red on every PR. Formatting only; no code change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019a95kk88TZ1iegsBK3kox3
…ript

Follow-up to #461 after CodeRabbit's second review and its autofix
commits. The autofix put `|| exit 1` into console blocks that an operator
pastes into an SSH shell, so a failed step would log them out of the host
halfway through containment. Three findings were also still open:

- step 5 relied on `$RUNNING` from an earlier shell;
- the §4 end-to-end example still POSTed to /api/verify as a success
  check;
- §5 told the operator to re-enable the auto-update timer, which does
  nothing without AutoUpdate=registry.

deploy/hetzner/stopgap-apply.sh (`apply DIR` | `verify`) replaces the
pasted steps. It runs under set -euo pipefail and stops before any
restart when a step cannot be confirmed:
- it checks that the source files are the stopgap versions;
- it requires exactly one install layout (quadlet or appendix);
- it disables the timer;
- it records the running image ID in ~/.local/state and re-points
  :latest at that image if the tag has moved;
- it installs the files and restarts the right units for the layout;
- it verifies the same image, the timer disabled, /api/health 200 and
  POST /api/verify 403.
`verify` reads the recorded image, so it works from any shell.

tests/deploy/stopgap-apply.test.sh drives the script against stub
podman, systemctl and curl commands: 29 cases, covering both layouts,
both-or-neither refusal, inspect failure, tag drift, non-stopgap
sources, and five verify failures. Three planted mutations of the
script are each caught. The test is wired into the MVP Smoke workflow
and `just test-deploy`. shellcheck is clean.

README: the stopgap section now documents the script; §4 marks the
expected 403; §5 explains what restoring rollout actually requires.

Not run against the live host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WS4mfQeC3N1DWjcCQJaQqw
@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Hypatia's code_safety/zig_ptr_cast and zig_align_cast now flag the nine
`@ptrCast(@aligncast(raw))` lines in src/zig_ffi/chapel_bridge.zig and
src/zig_ffi/runtime_owner.zig as high, and the governance Hypatia
baseline gate is red on main (2ab7574) as a result.

Each of those lines recovers a callback's own `*@this()` from the
type-erased `*anyopaque` it was registered with (runtime_owner.Work
.context, or a std.mem.Allocator vtable's ptr), so the cast restores the
original type and alignment. Fixed at source with a reasoned inline
`hypatia: allow` per rule on each line, as the scanner documents,
rather than a baseline entry. No code change: comments only.

Checked by simulating ScannerSuppression.inline_allowed?/4 and
directive_re at hypatia 82396d81 (the CI pin): 18 unsuppressed
findings on main, 0 with this change. The scanner itself could not be
built here (hex.pm is unreachable).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019a95kk88TZ1iegsBK3kox3
The "Dependency audit" and the rust-ci audit fail on main for three
advisories against rmcp 3.5.0, which echidna-mcp depends on as
`version = "3"`:
- RUSTSEC-2026-0457: the OAuth client's SSRF filter checks host
  strings, not resolved addresses;
- RUSTSEC-2026-0461: unbounded tool-schema cache in the Streamable
  HTTP server transport;
- RUSTSEC-2026-0462: unbounded response-body buffering in Streamable
  HTTP clients.

`cargo update -p rmcp -p rmcp-macros --precise 3.9.0` on the pinned
toolchain (1.99.0). rmcp 3.9.0 adds libc and sha2 0.11 to its own
dependency list.

The lockfile also re-points some Windows-only `windows-sys` and
`getrandom` edges (anstyle-query, colored, errno, rustix, socket2 and
others). The pinned cargo makes that same change on a no-op
`cargo update -p rmcp --precise 3.5.0`, so it is the resolver's output
on the current index, not part of this upgrade. No package versions
other than rmcp and rmcp-macros change.

cargo audit -f Cargo.lock: 0 vulnerabilities (was 3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WS4mfQeC3N1DWjcCQJaQqw

Copy link
Copy Markdown
Owner Author

Dependency audit was red, and the cause is not in this PR's change.


Generated by Claude Code

…ig casts) into #466

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WS4mfQeC3N1DWjcCQJaQqw

Copy link
Copy Markdown
Owner Author

Three checks here were also red on main (2ab7574), so they did not come from this PR:

  • Dependency audit / Cargo audit: rmcp RUSTSEC-2026-0457/0461/0462.
  • Cargo check + clippy + fmt: rustfmt drift in src/rust/proof_search.rs.
  • governance / Validate Hypatia Baseline: the zig_ptr_cast / zig_align_cast findings in src/zig_ffi/.

#465 fixes all three. I merged its branch into this one (bdc00b6), so this PR no longer waits on it. Once #465 lands, that merge becomes a no-op. The stopgap tests still pass locally: 29 passed, 0 failed.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Two more red checks on bdc00b6 that this PR did not cause, because both are red on main (2ab7574) too:

  • panic-attack assail. The pinned scanner (PANIC_ATTACK_REF 410a8f8) exits 2 with "scanner execution failed or timed out", so the completion receipt is missing. The same job fails on main (run 42351375995, job 128340059747).
  • Deposit findings for gitbot-fleet. It depends on the scan above and also fails on main.

This PR touches no scanner, workflow or Rust source involved in either check. No fix exists yet. The fault is in the panic-attack scanner or its pin, not here, so I've noted it as a separate item rather than widen this PR.


Generated by Claude Code

@hyperpolymath
hyperpolymath marked this pull request as ready for review November 9, 2026 15:49
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T02:37:18.650623Z c230138 New commits
🔒 Security Review ❌ Failed 2026-11-10T02:31:52.221217Z
We couldn’t complete this review. Comment @codex security review to try again.
c230138 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bdc00b6e2c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread deploy/hetzner/stopgap-apply.sh Resolved
Comment thread deploy/hetzner/stopgap-apply.sh Resolved
Comment on lines +114 to +117
if systemctl --user list-unit-files podman-auto-update.timer >/dev/null 2>&1; then
systemctl --user disable --now podman-auto-update.timer || die "cannot disable podman-auto-update.timer"
fi
record_and_pin

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Stop an in-flight auto-update before recording the image

If podman-auto-update.timer has already triggered its service when apply starts, disabling and stopping the timer does not stop the independently running podman-auto-update.service. That service can still pull, retag, and restart the container after record_and_pin has captured the supposedly frozen image, so the new image may run before the final mismatch check fails. Podman's documentation confirms that the timer triggers the service and that an update pulls the image and restarts the systemd unit; stop the service and confirm it is inactive before inspecting the running image. (Podman auto-update documentation)

Useful? React with 👍 / 👎.

Comment thread deploy/hetzner/stopgap-apply.sh Resolved
Codex P1 review findings on #466, each fixed in the script and covered by
new cases in tests/deploy/stopgap-apply.test.sh. Every new case fails on
the previous script and passes on this one (89 passed, 0 failed).

1. Running podman-auto-update.service. Disabling the timer did not end a
   run already in progress, which could pull, retag and restart after the
   image was recorded. apply now stops the service and refuses unless
   `systemctl --user is-active` reports inactive or failed, before the
   image is recorded or pinned; verify checks the same.
   Tests: section 10 (running run stopped first; still active/activating
   after stop refused with nothing changed; active run fails verify).

2. Unit directive matching. The AutoUpdate and Pull checks were exact
   prefix greps. Directives are now normalised as systemd reads them
   (indentation, spaces around `=`, `#`/`;` comments), the [Container]
   section must hold exactly one Pull=never, and AutoUpdate= in any
   section or an io.containers.autoupdate label refuses.
   Tests: section 11 (spaced/indented AutoUpdate, label via PodmanArgs,
   second Pull=, commented Pull=never refused; spaced Pull=never and
   commented AutoUpdate accepted).

3. Image= validation. [Container] must hold exactly one
   Image=ghcr.io/hyperpolymath/echidna:latest, checked with the same
   normalisation before any change.
   Tests: section 12 (other image, second Image= refused; spaced Image=
   with the expected reference accepted).

4. Containment validation. (a) The Caddyfile check accepted any file
   containing `@public`. It is now a structural check: one @public matcher
   holding exactly `method GET HEAD` and `path /api/health /api/provers`;
   one reverse_proxy, the only child of `handle @public`; one sibling
   catch-all handle holding only `respond ... 403`; no other handle or
   named matcher; no import/invoke/handle_path/rewrite/uri/php_fastcgi/
   forward_auth; method only inside @public. `caddy adapt` is not used:
   the proxy image carries the rate_limit plugin a stock caddy rejects,
   and the host has no caddy outside that image.
   (b) The live probe tested only POST /api/verify. It now requires 403
   from POST /api/prove, /api/verify, /api/verify_raw,
   /api/verify_parallel, /api/suggest, /api/session/create,
   /api/agent/plan, GET /api/search, POST /api/health and an unknown
   /api/ route, with GET /api/health still 200; any other answer fails.
   Tests: sections 13 and 14.

README.adoc describes the new checks and the probe's request count
against the 30/min rate limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WS4mfQeC3N1DWjcCQJaQqw
@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a15a06fdec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +87 to +90
[ "$(grep '^\[Container\] Pull=' <<<"$d" || true)" = "[Container] Pull=never" ] \
|| die "$f is not the stopgap version (needs exactly one Pull=never in [Container])"
[ "$(grep '^\[Container\] Image=' <<<"$d" || true)" = "[Container] Image=$IMAGE" ] \
|| die "$f must set exactly one Image=$IMAGE in [Container]"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject Quadlets that publish the API port

When the supplied unit contains a valid directive such as PublishPort=8081:8081 and that port is reachable through the host firewall, check_unit still accepts it because it validates only Image, Pull, and auto-update settings. Restarting then exposes the server—which binds 0.0.0.0:8081—directly, bypassing the Caddy denial policy, while every verification request to the HTTPS hostname can still pass; reject published ports, host networking, and equivalent PodmanArgs before installing the unit.

Useful? React with 👍 / 👎.

@coderabbitai

coderabbitai Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Add Carrot credits or activate Agent usage billing to use Autofix

Repository owner deleted a comment from chatgpt-codex-connector Bot Nov 9, 2026
@hyperpolymath
hyperpolymath enabled auto-merge (squash) November 9, 2026 22:03
Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

@sonarqubecloud

Copy link
Copy Markdown

@hyperpolymath
hyperpolymath enabled auto-merge (squash) November 10, 2026 02:34
@hyperpolymath
hyperpolymath merged commit f546ee9 into main Nov 10, 2026
81 of 84 checks passed
@hyperpolymath
hyperpolymath deleted the claude/zen-brahmagupta-opqqle branch November 10, 2026 02:34

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c230138604

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Justfile
Comment on lines +11 to +12
# Initialize from RSR template (placeholder replacement)
init:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the gate self-test recipe

Replacing the module imports with this flattened copy drops the existing test-gates recipe from just/build.just, even though it remains part of the checked-in just --list contract. Any developer or automation invoking just test-gates now gets an unknown-recipe error instead of running scripts/gates/lib.selftest.sh and scripts/gates/w0.selftest.sh; keep the modular imports and add test-deploy to the appropriate module, or include the omitted recipe in the flattened file.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants