From 3dd587e40097fded0543b19fce3f92965db36528 Mon Sep 17 00:00:00 2001 From: Ian Clarke Date: Sun, 16 Aug 2026 11:33:14 -0500 Subject: [PATCH 1/2] docs(api): write down the gkapi deploy procedure, including setcap Deploying this crate is undocumented tribal knowledge, and one step in it is invisible until it bites: the service runs unprivileged and binds port 80 for the ACME challenge server, which needs CAP_NET_BIND_SERVICE as a file capability. `cp` does not preserve file capabilities, so replacing the binary without re-running setcap makes it bind 443, log "Listening on 0.0.0.0:443", then panic with PermissionDenied and restart-loop. The same trap catches the rollback, which is what makes it genuinely nasty: copying a backup back into place also drops the capability, so the service keeps panicking and it looks like the new build is to blame rather than the copy. `mv` keeps the inode and therefore the capability; `cp` does not. This bit during the deploy of #94 and took the API down for about 100 seconds (16:27:05 to 16:28:45 UTC on 2026-08-16). Writing it down so the next person does not rediscover it in production. Also records what CI does and does not do here: merging a change to rust/api ships nothing, because deploy.yml only publishes the Hugo site. #94 sat merged and undeployed for 18 days for exactly that reason. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01CCLUFDUS75xXcRZ83yDzq1 --- rust/api/README.md | 93 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/rust/api/README.md b/rust/api/README.md index 3ce52c60..9c575b9f 100644 --- a/rust/api/README.md +++ b/rust/api/README.md @@ -2,6 +2,99 @@ WARNING: This file is publicly readable, do NOT put anything secret in here. +## Deploying gkapi + +There is **no CI deployment for this crate**. `deploy.yml` builds the Hugo site and +publishes it to GitHub Pages; it never touches the API. `rust-api-tests.yml` only runs +fmt, build and test. Merging a change to `rust/api` therefore ships nothing: the binary +on vega has to be replaced by hand, and has in the past sat months behind main. + +There is also no Rust toolchain on vega (`~gkapi/.cargo` exists but `bin/` is empty), so +the binary is built elsewhere and copied. vega and nova are both Ubuntu 24.04 on the same +glibc, so a release build from nova runs there as-is. Check with `ldd --version` on both +before relying on that. + +### The step that is easy to miss + +The service runs as the unprivileged `gkapi` user and binds **port 80** for the HTTP-01 +ACME challenge server, which needs `CAP_NET_BIND_SERVICE`. That capability lives on the +binary as a file capability, and **`cp` does not preserve it**. Replace the binary without +re-running `setcap` and the service binds 443, logs `Listening on 0.0.0.0:443`, then panics +at `main.rs` with `PermissionDenied` on the port 80 bind and systemd restart-loops until it +gives up. + +The same trap catches the rollback: copying a backup *back* into place also produces a file +with no capability, so a panicking service stays panicking and it looks like the new build +is at fault. `mv` preserves the capability because it keeps the inode; `cp` does not. + +### Procedure + +```bash +# 1. build on nova, from main, and prove the build contains what you expect +cd ~/code/freenet/web/main/rust && cargo build --release -p ghostkey-api +strings target/release/ghostkey-api | grep -c payment_claim # sanity: expect non-zero + +# 2. upload and re-check after transfer +scp target/release/ghostkey-api vega:/tmp/ghostkey-api.new +ssh vega 'strings /tmp/ghostkey-api.new | grep -q payment_claim && echo ok' + +# 3. back up with mv (keeps the capability on the backup, so rollback is clean) +ssh vega 'sudo mv /home/gkapi/bin/ghostkey-api \ + /home/gkapi/bin/ghostkey-api.rollback-$(date +%Y%m%d-%H%M%S)' + +# 4. install, own, and RESTORE THE CAPABILITY before restarting +ssh vega ' + sudo cp /tmp/ghostkey-api.new /home/gkapi/bin/ghostkey-api + sudo chown gkapi:gkapi /home/gkapi/bin/ghostkey-api + sudo chmod 775 /home/gkapi/bin/ghostkey-api + sudo setcap cap_net_bind_service+ep /home/gkapi/bin/ghostkey-api + getcap /home/gkapi/bin/ghostkey-api # must print cap_net_bind_service=ep +' + +# 5. only then restart +ssh vega 'sudo systemctl reset-failed gkapi && sudo systemctl restart gkapi' +``` + +Verify `getcap` prints the capability **before** restarting. If it is empty, fix it rather +than restarting to see what happens. + +### Verifying a deploy + +```bash +ssh vega 'systemctl is-active gkapi; strings /home/gkapi/bin/ghostkey-api | grep -c payment_claim' +curl -s https://gkapi.freenet.org/ # {"message":"Hello, world!"} +curl -s -o /dev/null -w '%{http_code}\n' http://gkapi.freenet.org/.well-known/acme-challenge/probe +``` + +That last one matters: a 404 means the port 80 challenge listener is up. Connection refused +means the capability is missing and certificate renewal will fail at the next attempt even +if HTTPS looks healthy. + +Then confirm every donation tier still resolves its notary keypair, which is a separate +failure mode from the binary (see the tier comment in +`hugo-site/themes/freenet/layouts/shortcodes/stripe-donation-form.html`): + +```bash +for a in 1 5 20 50 100 500 2500 10000; do + curl -s -X POST https://gkapi.freenet.org/create-donation \ + -H 'Content-Type: application/json' -d "{\"amount\":$((a*100)),\"currency\":\"usd\"}" \ + | grep -q notary_certificate_base64 && echo "\$$a ok" || echo "\$$a FAIL" +done +``` + +### Rollback + +```bash +ssh vega ' + sudo cp /home/gkapi/bin/ghostkey-api.rollback- /home/gkapi/bin/ghostkey-api + sudo chown gkapi:gkapi /home/gkapi/bin/ghostkey-api + sudo setcap cap_net_bind_service+ep /home/gkapi/bin/ghostkey-api + sudo systemctl reset-failed gkapi && sudo systemctl restart gkapi +' +``` + +The `setcap` line is required on the way back too, for the reason above. + ## letsencrypt Verify that certificate was automatically renewed by root cron job on vega by looking at write times From 145b0cae0537bfba665ac59909860c7eb4abfdab Mon Sep 17 00:00:00 2001 From: Ian Clarke Date: Sun, 16 Aug 2026 11:54:11 -0500 Subject: [PATCH 2/2] docs(api): close the no-binary window and gate on sudo getcap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adversarial review of the procedure found two ways the runbook could cause an outage worse than the one it documents. The first is a window where no binary exists. Backing up with `mv` in one ssh call and installing with `cp` in the next leaves /home/gkapi/bin/ghostkey-api absent in between, so anything that restarts the unit in that gap fails with status=203/EXEC and stays down — strictly worse than never starting, since the old process was still serving fine from its open inode. Restructured to stage the new binary at its final directory first, fully prepared and capability verified, then swap with two adjacent renames. That reordering also improves the failure mode of the check itself: the getcap gate now runs while the live binary is completely untouched, so failing it costs nothing. The second is that `getcap` was called without sudo while every neighbouring command had it. It lives in /usr/sbin, which is not on a normal user's PATH, so the doc's one verification gate could fail as "command not found" and be skipped rather than enforced. Also notes that /create-donation has no dry-run mode, so the tier verification loop creates real (uncharged, unattached) PaymentIntents in the live Stripe account, and says to substitute the rollback timestamp rather than pasting the placeholder. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01CCLUFDUS75xXcRZ83yDzq1 --- rust/api/README.md | 73 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 54 insertions(+), 19 deletions(-) diff --git a/rust/api/README.md b/rust/api/README.md index 9c575b9f..6ef95c05 100644 --- a/rust/api/README.md +++ b/rust/api/README.md @@ -38,25 +38,45 @@ strings target/release/ghostkey-api | grep -c payment_claim # sanity: expect n scp target/release/ghostkey-api vega:/tmp/ghostkey-api.new ssh vega 'strings /tmp/ghostkey-api.new | grep -q payment_claim && echo ok' -# 3. back up with mv (keeps the capability on the backup, so rollback is clean) -ssh vega 'sudo mv /home/gkapi/bin/ghostkey-api \ - /home/gkapi/bin/ghostkey-api.rollback-$(date +%Y%m%d-%H%M%S)' - -# 4. install, own, and RESTORE THE CAPABILITY before restarting +# 3. stage it at its final location, fully prepared, WITHOUT displacing the live binary ssh vega ' - sudo cp /tmp/ghostkey-api.new /home/gkapi/bin/ghostkey-api - sudo chown gkapi:gkapi /home/gkapi/bin/ghostkey-api - sudo chmod 775 /home/gkapi/bin/ghostkey-api - sudo setcap cap_net_bind_service+ep /home/gkapi/bin/ghostkey-api - getcap /home/gkapi/bin/ghostkey-api # must print cap_net_bind_service=ep + sudo cp /tmp/ghostkey-api.new /home/gkapi/bin/ghostkey-api.staged + sudo chown gkapi:gkapi /home/gkapi/bin/ghostkey-api.staged + sudo chmod 775 /home/gkapi/bin/ghostkey-api.staged + sudo setcap cap_net_bind_service+ep /home/gkapi/bin/ghostkey-api.staged + sudo getcap /home/gkapi/bin/ghostkey-api.staged ' +# Must print: /home/gkapi/bin/ghostkey-api.staged cap_net_bind_service=ep +# If it does not, STOP. Nothing has changed yet and the live binary is untouched. -# 5. only then restart -ssh vega 'sudo systemctl reset-failed gkapi && sudo systemctl restart gkapi' +# 4. swap with two adjacent renames, then restart +ssh vega ' + set -e + STAMP=$(date +%Y%m%d-%H%M%S) + sudo mv /home/gkapi/bin/ghostkey-api /home/gkapi/bin/ghostkey-api.rollback-$STAMP + sudo mv /home/gkapi/bin/ghostkey-api.staged /home/gkapi/bin/ghostkey-api + echo "rollback binary: /home/gkapi/bin/ghostkey-api.rollback-$STAMP" + sudo systemctl reset-failed gkapi + sudo systemctl restart gkapi +' ``` -Verify `getcap` prints the capability **before** restarting. If it is empty, fix it rather -than restarting to see what happens. +Two things about that shape are deliberate, and both exist to avoid a worse failure than +the one being fixed: + +- **Prepare the capability on the staged file, and verify it, before anything is + displaced.** The `getcap` gate in step 3 is the whole point of splitting the steps: if it + fails you stop with the running service completely untouched. Do not restart to see what + happens. +- **Do not `mv` the old binary away in one command and `cp` the new one in with another.** + Between those two the path does not exist, and anything that restarts the unit in that + window (a crash, an OOM, a reboot, someone running `systemctl restart` out of order) fails + with `status=203/EXEC` and stays down. Step 4 closes that to two adjacent renames. + `mv` is also what carries the capability across, since it keeps the inode. + +`sudo` on `getcap` is not decoration: it lives in `/usr/sbin`, which is not on a normal +user's `PATH`, so a bare `getcap` can fail as "command not found" and skip the check +entirely. ### Verifying a deploy @@ -72,7 +92,12 @@ if HTTPS looks healthy. Then confirm every donation tier still resolves its notary keypair, which is a separate failure mode from the binary (see the tier comment in -`hugo-site/themes/freenet/layouts/shortcodes/stripe-donation-form.html`): +`hugo-site/themes/freenet/layouts/shortcodes/stripe-donation-form.html`). + +Note that `/create-donation` has no dry-run mode: each call creates a real PaymentIntent in +the live Stripe account. Nothing is charged and no card is attached, so these are harmless +abandoned intents, exactly what a visitor clicking between the amount radios produces. Run +the loop once after a deploy; do not wrap it in a retry-until-success script. ```bash for a in 1 5 20 50 100 500 2500 10000; do @@ -84,16 +109,26 @@ done ### Rollback +Substitute `` with the timestamp step 4 printed. + ```bash ssh vega ' - sudo cp /home/gkapi/bin/ghostkey-api.rollback- /home/gkapi/bin/ghostkey-api - sudo chown gkapi:gkapi /home/gkapi/bin/ghostkey-api - sudo setcap cap_net_bind_service+ep /home/gkapi/bin/ghostkey-api + set -e + sudo cp /home/gkapi/bin/ghostkey-api.rollback- /home/gkapi/bin/ghostkey-api.staged + sudo chown gkapi:gkapi /home/gkapi/bin/ghostkey-api.staged + sudo chmod 775 /home/gkapi/bin/ghostkey-api.staged + sudo setcap cap_net_bind_service+ep /home/gkapi/bin/ghostkey-api.staged + sudo getcap /home/gkapi/bin/ghostkey-api.staged + sudo mv /home/gkapi/bin/ghostkey-api.staged /home/gkapi/bin/ghostkey-api sudo systemctl reset-failed gkapi && sudo systemctl restart gkapi ' ``` -The `setcap` line is required on the way back too, for the reason above. +The `setcap` line is required on the way back too, and this is the part that is genuinely +easy to get wrong under pressure: copying a backup *into place* produces a file with no +capability, so the service carries on panicking and it reads as "the new build is broken" +rather than "the copy dropped a capability". If a rollback does not fix the panic, run +`sudo getcap` on the live binary before concluding anything about the build. ## letsencrypt