node: /healthz, /readyz, and Prometheus /metrics on --health-listen

[bkeroack]

Implements docs/peer-clients.md "Ranked if we spend time later" rank 4:
"/healthz (and maybe /readyz) on the node listen; Prometheus later as a
flag". That table says a row is implemented only when you name it, so treat
this as a proposal. The calls I made are listed under Open questions;
any of them is easy to change.

Why a separate listener

RPC, Electrum and Esplora bind only after the store opens, catch-up finishes
and the scripthash index materializes. On a synced mainnet node:

run_p2p step	Wall	Listening
run_node store open (schema 24→26 input.* backfill)	41 min 52 s	nothing
run_ibd_or_skip (full IBD, full script checks)	8 h 58 min	nothing
enter_tip_mode (scripthash materialize)	~33 min	nothing
Electrum / Esplora / RPC bind

A liveness probe on any of those listeners fails for the whole window, so k8s
(or any restart-on-unhealthy supervisor) would restart the node in the middle
of the migration or the IBD, and it would start over. The health listener binds
at the top of run_p2p, before run_node.

What

--health-listen [ADDR] (conf health_listen; bare → 127.0.0.1:9332)

Route	Answer
GET /healthz	200 ok in every phase
GET /readyz	200 ok, or 503 not ready: <reason> for the first failing check
GET /metrics	with --metrics only; otherwise 404

Other paths 404, other methods 405, no body, no auth. The same always-on
concurrency / body / timeout layers as Esplora. A non-loopback bind logs a
WARN (a k8s httpGet probe connects to the pod IP). A bind failure stops the
node before it touches the datadir.

/readyz checks, in order:

1. The bring-up phase (opening, starting, catch-up, indexing,
   stopping), written by run_p2p at each transition.
2. A configured RPC / Electrum / Esplora listener that did not bind
   (rpc not listening). Their start only warns, so without this the node
   follows the tip while a load balancer routes wallets to a dead port.
3. initial block download: ChainHub::in_ibd, the same call
   getblockchaininfo.initialblockdownload makes.
4. tip stale (last block Ns ago): the tip is older than --max-tip-age
   (default 24 h). in_ibd latches off after the first exit, so without this
   a node that later loses every peer would stay ready on a stale tip.
5. tip N blocks behind headers (over 6).
6. With --sh-index: scripthash index N blocks behind tip (over 6).

--metrics (conf metrics; refused without --health-listen) adds
Prometheus text on the same listener. To avoid a second metrics dialect, each
gauge is a value the node already publishes, named after its source, and the
journey checks it against RPC at the same moment:

• rbitcoin_blocks, _headers, _tip_time_seconds,
  _initial_block_download = getblockchaininfo
• rbitcoin_connections{direction="in"|"out"} = getnetworkinfo
connections_in / _out (both now count through one
  rbitcoin_net::connection_counts)
• rbitcoin_mempool_transactions, _bytes = getmempoolinfo size, bytes
• rbitcoin_scripthash_lag_blocks = tip: accept sh_lag=
• process_resident_memory_bytes = ibd: sizes rss=; process_start_time_seconds
• rbitcoin_build_info, rbitcoin_phase, rbitcoin_ready = /readyz

Counters are the tip: perf meters: Esplora and Electrum requests and
seconds, historical block serves and bytes, mempool accepts and rejects.
Those meters were sample-and-reset (swap(0)), which a counter cannot
read. They are now running totals (PerfCounter), and the 5 s perf tick takes
the change since its previous sample, so the DEBUG line prints the same
numbers. The window max stays a reset (PerfMax); it is not exported.
Esplora and Electrum share one RequestMeter in place of two copies of the
same three statics and CAS loop.

Cost (principle 9)

• /healthz: no state read.
• /readyz: a phase load; in following also in_ibd (an atomic once
  latched), best_header_height (read lock over the header tips), tip height
  and sh_lag_heights. It runs on the blocking pool because the chain reads
  can touch the store.
• /metrics: the above plus one PeerHub::snapshot and one fold over the
  mempool for bytes (the same fold as getmempoolinfo), on the blocking
  pool. At a 15 s scrape interval that is noise next to the tip path.
• Meters: no extra work per request. Each counter gains one window-mark
  atomic that only the 5 s tick touches.
• Nothing is on lookup, load, scripts or write, so there is no new ibd: perf
  timer.

Tests

Journeys (rbitcoin-test --test cross_surface):

• esplora_broadcast_visible_in_rpc_and_electrum adds --health-listen and
  --metrics to the existing session. It checks /healthz 200 / 405 / 404;
  /readyz is 503 not ready: initial block download while RPC reports
  initialblockdownload: true, then 200 ok after the mined block. It
  scrapes /metrics once in IBD (mempool non-empty) and once after, and
  compares every gauge above with RPC at that moment. The counters are
  present, ≥ 1 where the journey used the surface, and do not go down.
• readyz_names_a_listener_that_did_not_bind: RPC on a taken port →
  503 not ready: rpc not listening while the node follows the tip, and
  /metrics is 404 without --metrics.
• health_listen_bind_failure_stops_before_store_open: a taken health port
  → run_p2p errors and store/ does not exist. This pins that the listener
  binds before the store opens.

The binary smoke journey (scenarios::node_cli_and_surface_smoke) refuses
--metrics without --health-listen (exit 1) and --health-listen bad
(exit 2). The consolidated kebab-flag test checks the bare --health-listen
default and --metrics.

Units only where a session cannot reach the case: the readiness table (every
phase, the lag boundaries 6 and 7, the tip-age boundary, SH off), the listener
answering opening (a real store opens in milliseconds), and PerfCounter /
PerfMax / RequestMeter windows versus totals.

Commits follow the steps: /healthz, /readyz, gauges, counters, docs,
then the stale-tip gate and the /metrics mention in the non-loopback WARN.
Docs: docs/operator/operations.md (flags, a "Health probes and metrics"
section with the reason strings, a k8s probe snippet, the metric table and a
scrape config), a SECURITY.md scope bullet, peer-clients.md rank 4
marked landed, and changelog.d/health-endpoints.md.

Open questions

1. Placement. A separate --health-listen socket, for the reasons above,
   rather than routes on RPC or Esplora.
2. Names and port. --health-listen / --metrics; bare port 9332.
3. Readiness gates. 6 blocks for both lags (satd uses 6), and the tip-age
   gate reuses --max-tip-age. Should SH lag gate readiness at all?
4. Non-loopback binds are allowed with a WARN rather than refused.
5. Q-72. I marked the peer-clients.md row landed but did not add a
   quality.md row; say if you want one.
6. Liveness heartbeat. /healthz is "the process answers HTTP". Failing it
   when the main loop stops ticking would catch a wedged loop, but it would
   have to stay off during opening.
7. NixOS module. No health / metrics options yet; I can't run
   nixos-module-eval locally.
8. Bind failure is fatal, unlike RPC's warn-and-continue.

Local results

Rebased onto 60e03731 (after #800 and #802; the CLI and validate checks
moved into #802's smoke journey). cargo fmt --check,
cargo clippy --workspace --all-targets -D warnings, ./scripts/ast-grep.sh
and cargo deny check pass. cargo test --workspace --no-fail-fast ran on a
host at load average ~165, where only wall-clock tests failed:
integration_multinode (20 s budgets) and
tip_accept::tests::drop_join_does_not_park_worker. With
--test-threads=1, integration_multinode passes 15 of 16, and the 16th
(p2p_compact_hb_getblocktxn_and_orphan, a wall timeout) passes alone;
drop_join_does_not_park_worker passes alone. cross_surface, scenarios
and the node lib pass. Before the rebase the required CI jobs were green.

Side note, not changed here: on a heavily loaded host,
tip_accept::tests::drop_join_does_not_park_worker can miss its 2 s
"job never started" deadline. It then panics without releasing the job it
queued. When that job runs later it blocks the shared tip-accept lane, and
every later lane test in that process hangs.

🤖 Generated with Claude Code