From 2a56f0c8b44fe6e3f5bd74647d5e2708b85b92dd Mon Sep 17 00:00:00 2001 From: Gary Krause Date: Wed, 23 Sep 2026 14:45:21 -0500 Subject: [PATCH 1/7] docs: roadmap the SV2 template provider (Q-64) Split Q-64 into three plans, one PR each: A (caller-budgeted template selection behind GBT), B (mine a block through the TP), C (fee-delta push). B stays whole: a TP without tip push or SubmitSolution strands miners on stale tips or drops found blocks. Step 0 spike: reuse the crates.io stratum-core wire crates (noise_sv2 2.0.0, codec_sv2 7.0.0, framing_sv2 8.0.0, binary_sv2 7.0.0, template_distribution_sv2 7.0.0, parsers_sv2 0.6.0, common_messages_sv2 9.0.0). cargo deny passes; no second bitcoin; secp256k1 0.28 comes in via noise_sv2 only and links alongside 0.29 in a static musl binary. A Noise NX roundtrip of SetupConnection, NewTemplate, and a 3 MB RequestTransactionData.Success decodes field-equal. Co-Authored-By: Claude Opus 5.5 --- docs/README.md | 1 + docs/quality.md | 2 +- docs/sv2-template-provider.md | 373 ++++++++++++++++++++++++++++++++++ 3 files changed, 375 insertions(+), 1 deletion(-) create mode 100644 docs/sv2-template-provider.md diff --git a/docs/README.md b/docs/README.md index 6f41b22e4..cb64fa1f2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -52,6 +52,7 @@ into `AGENTS.md`. | [`overlay-functional.md`](./overlay-functional.md) | Private Tor / i2pd / cjdns mesh harness (labeled / nightly). | | [`how-we-plan.md`](./how-we-plan.md) | Agent contract (cycle, Agent RAM, keep-compiling), then human rationale. | | [`personal-node-plans/`](./personal-node-plans/) | Home-node implementation plan group (SOCKS, overlays, wallet onions, ephemeral broadcast, seqsigwit prune / `NETWORK_LIMITED`). Live flags stay OPERATOR/COMPAT when a slice ships. Do not copy step lists into quality.md until scheduled. | +| [`sv2-template-provider.md`](./sv2-template-provider.md) | SV2 Template Distribution Protocol server roadmap (**Q-64**): Step 0 finding and plans A–C. Live flags/COMPAT rows land in OPERATOR/COMPAT with the plan that ships them. | | [`releases.md`](./releases.md) | Tag `vX.Y.Z`, `vX.Y.x` patch line, `.99` bump, Highlights / GitHub notes. | | [`code-shape.md`](./code-shape.md) | Control flow, types, naming, composition (CONTRIBUTING principle 10). Named extracts: quality.md **Q-61** (Completed). Clippy: no workspace `allow` list; leftover lints are site-local with a reason. | | [`quality.md`](./quality.md) | Living quality roadmap (Open + Won't-fix + Parked + Protect). | diff --git a/docs/quality.md b/docs/quality.md index 8a5168fd8..0b4b46eee 100644 --- a/docs/quality.md +++ b/docs/quality.md @@ -36,6 +36,7 @@ Counts and procedures stay in the owner doc. A row is the rank and the outcome. | 6 | **Q-69** | CLN / LDK chain backend | Operator can point CLN `bcli` and ldk-node Esplora/Electrum at this node. Owner: [`lightning.md`](./lightning.md). | | 7 | **Q-70** | Batch lookup-path hit counters if a profile names them | `head_resolve_stats::add_hit_rank` does two relaxed `fetch_add`s per resolved txid (`head_resolve_denserels`, `tx_table`). SH extract collect flushes output and hit counts once per fk batch, not per output. Still once per event: `accepted_wb` (`SeqCst`, per confirmed block), `serve_perf::note_serve` (per historical getdata), `page_ios` (per SH page read). Done: a profile shows `add_hit_rank` on confirm lookup and those counters flush per batch, or the profile shows the line is noise and this row moves to Won't-fix. `add_hit_ages` is already a batched flush; `add_hit_age` remains the one-bucket path. | | 8 | **Q-71** | Talip review remediations | Every row in [`external_findings/024-talip-review-index.md`](./external_findings/024-talip-review-index.md) is fixed, rejected, or won't-fix, and each fixed row names a regression. | +| 9 | **Q-64** | SV2 template provider (TDP server) | Node serves Noise-encrypted Template Distribution Protocol in-process: `CoinbaseOutputConstraints` → pushed `NewTemplate` / `SetNewPrevHash` on tip change and fee delta, `RequestTransactionData`, `SubmitSolution` → `accept_block`. Roadmap (plans A → B → C, one PR each): [`sv2-template-provider.md`](./sv2-template-provider.md). | R-ids were the 2026-08-12 slice. Canonical id is **bold**. Do not start **R-11+**. Next unused Q-id is **Q-72**. @@ -84,7 +85,6 @@ just not the current product. COMPAT/OPERATOR stay the shipped contract. | ID | Item | Why parked | Reopen when | |----|------|------------|-------------| | **Q-63** | Electrum TLS (50002) + Tor onion **in the binary** | Home Sparrow/phone off-LAN today uses nginx (`OPERATOR.md`). Node stays plain TCP. | Operators refuse a reverse proxy, or a first-class onion listener is the 1.0 install. | -| **Q-64** | GBT longpoll / `waitNext` (then Sv2 template provider) | Opt-in `getblocktemplate` + Esplora `/block-template` with 15 s cache is the mining extra. No stratum/pool. | DATUM / Bitaxe / mkpool users need push templates; IPC mining interface is the Core shape. | | **Q-65** | BIP157/158 compact block filters | Shipped as optional `--block-filter-index` (basic / type 0). `NODE_COMPACT_FILTERS` is advertised once filters first reach the tip. Serving follows the filter watermark, not the scripthash watermark. | — | --- diff --git a/docs/sv2-template-provider.md b/docs/sv2-template-provider.md new file mode 100644 index 000000000..ed1abcb39 --- /dev/null +++ b/docs/sv2-template-provider.md @@ -0,0 +1,373 @@ +# SV2 template provider (TDP server) + +Roadmap for **Q-64**: three plans, each one PR +([`how-we-plan.md`](./how-we-plan.md#checklist-for-authors-and-agents)). +Cycle and step shape: [agent contract](./how-we-plan.md#agent-contract). Do +not start a plan before the previous one is merged, or a step before the +previous slice is committed. + +| Plan | Outcome | Ships flags | +|------|---------|-------------| +| **A** | GBT builds from a caller-budgeted selection that returns fee and sigop cost with each tx | none | +| **B** | A Job Declarator Client mines a block through the node's TP | `--sv2-tp-listen`, `--sv2-tp-authority-sec`, `--sv2-tp-cert-validity`, `--sv2-tp-stale-grace` | +| **C** | Templates refresh on fee gain with the tip unchanged | `--sv2-tp-fee-delta`, `--sv2-tp-template-interval` | + +B is not split further: a listener that serves templates without +tip-change push or `SubmitSolution` makes miners work stale tips or lose +found blocks. The smallest safe operator surface is B whole. + +## Goal + +Operator runs `rbitcoin-node --sv2-tp-listen 127.0.0.1:8442 +--sv2-tp-authority-sec `. A Stratum v2 Job Declarator Client or pool +connects over Noise_NX TCP, completes `SetupConnection(protocol=2)`, sends +`CoinbaseOutputConstraints`, and from then on is **pushed** `NewTemplate` / +`SetNewPrevHash` when the tip changes and when template fees rise by a +configured delta. `RequestTransactionData` returns the retained template's +transactions; `SubmitSolution` assembles the full block and submits it +through the normal accept path. TDP replaces `getblocktemplate` polling for +these clients (sv2-spec 07). + +Reference state machine: sv2-apps `bitcoin-core-sv2` (Bitcoin Core IPC → +TDP). This TP is in-process: the template source is the node's own mempool +and tip, no IPC hop. + +## Step 0 finding (spike, 2026-09-25): reuse stratum-core + +Throwaway workspace member depending on the crates.io wire crates: +`noise_sv2` 2.0.0, `codec_sv2` 7.0.0 (`noise_sv2` feature), `framing_sv2` +8.0.0, `binary_sv2` 7.0.0, `template_distribution_sv2` 7.0.0, +`parsers_sv2` 0.6.0, `common_messages_sv2` 9.0.0. + +- `cargo deny check`: advisories, bans, licenses, sources ok. No new + license allow entry. +- No second `bitcoin`: the wire crates do not depend on it. New duplicate + versions (warn only): `secp256k1` 0.28.2 + `secp256k1-sys` 0.9.2 + (`noise_sv2` only), `bitcoin_hashes` 0.13, `hex-conservative` 0.1, + `cpufeatures`, `crypto-common`. Lock delta: 29 packages (AEAD stack: + `chacha20poly1305`, `aes-gcm`, …, plus unused `mining_sv2` / + `job_declaration_sv2` / `extensions_sv2` pulled by `parsers_sv2`). +- `x86_64-unknown-linux-musl` release link: static-pie; both libsecp + builds coexist under distinct symbol prefixes (`rustsecp256k1_v0_9_2_*`, + `rustsecp256k1_v0_10_0_*`). +- In-memory roundtrip: NX handshake with the authority keypair built from + the **workspace** `secp256k1` 0.29 (`Responder::from_authority_kp` / + `Initiator::from_raw_k` take raw bytes, so 0.28 types never leave + `noise_sv2`), then `SetupConnection`, a 12-deep `NewTemplate`, and a + 3 MB `RequestTransactionData.Success` across Noise chunks, each decoded + back field-equal. +- Published TDP field set matches this roadmap: `CoinbaseOutputConstraints + {max_additional_size: u32, max_additional_sigops: u16}`, no + `coinbase_witness` on `NewTemplate`. + +Decision: **reuse**. Hand-rolling stays the fallback if a later bump fails +`cargo deny`. `parsers_sv2` is optional: B may decode with `binary_sv2:: +from_bytes` on the known TDP / common message types and drop it (and the +three unused subprotocol crates) if it adds nothing. + +## Constraints (all plans) + +- New crate `crates/rbitcoin-sv2` (Plan B), service pattern of + electrum/esplora: depends on `rbitcoin-query` / `rbitcoin-net` / + `rbitcoin-mempool` / `rbitcoin-consensus`; wired in `rbitcoin-node` + `run.rs` behind flags. Nothing starts without `--sv2-tp-listen`. +- `binary_sv2` byte-buffer types and `noise_sv2`'s `secp256k1` 0.28 stay + inside `rbitcoin-sv2`; consensus decode uses the workspace + `rust-bitcoin`. No sv2 types in other crates' APIs. +- Reactor rule: template builds and solution assembly run in a blocking + region, never on tokio workers. `MempoolHub` accessors assert + not-reactor. +- Named RAM trade (CONTRIBUTING 9): each session retains, per live + template, the full witness-serialized non-coinbase txs (≤ ~4 MB × ~3 + templates × sessions). Retention is required: the mempool may evict a tx + before `RequestTransactionData` or `SubmitSolution` arrives. Stale grace + (default 10 s) after a tip change, then drop (mirrors sv2-tp). +- Per-session budget: weight `MAX_BLOCK_WEIGHT − max(1168 + + 4·coinbase_output_max_additional_size, 2000)` WU (sv2-spec 07 §7.1); + sigops start at `coinbase_output_max_additional_sigops` (Core + `BlockAssembler` semantics: the client's value replaces the 400 default + reserve). Each client sizes its own templates. +- Noise is the only mode (mandatory for remote TDP). No plaintext operator + flag; tests drive the shipped Noise path. +- TDP defines no `SetupConnection` flags: nonzero `flags` → + `SetupConnection.Error` echoing the full unsupported set; `protocol != 2` + or no version-2 overlap → Error and close. +- `template_id` strictly increasing per session. +- Coinbase split (sv2-spec 07 §7.2): `coinbase_prefix` is the BIP34 height + push (≤ 8 bytes, start of scriptSig); `coinbase_tx_value_remaining` = + subsidy + Σ fees; `coinbase_tx_outputs` is the raw concatenation (no + CompactSize prefix) with the witness-commitment OP_RETURN **last**, from + `rbitcoin_consensus::witness_commitment_script` (already the one owner, + used by GBT) with a 32-byte zero reserved value. +- `SetNewPrevHash.target` == nBits target here (no weak blocks). +- No templates before sync: same refusal gate as `getblocktemplate` during + IBD. +- `SubmitSolution` has no error message in TDP: undecodable or + unknown-template solutions are logged and dropped; decodable ones are + always attempted through `ChainHub::accept_block` (the TP MUST try to + broadcast work on its templates). +- `SubmitSolution.header_timestamp` pre-check: ≥ the sent + `SetNewPrevHash.header_timestamp` and ≤ that plus wall-clock elapsed + (sv2-spec 07 §7.7). +- OPERATOR / COMPAT / NixOS options land in the plan that ships the flag + (same PR). + +## Out of scope + +Mining Protocol server (channels), Job Declaration **Server**, SV1↔SV2 +translator proxy, Job Declarator Client, weak-block targets below nBits, +extension negotiation, per-IP metering/rate limits. None of these ship; +nothing here precludes a later JD-server plan. + +--- + +## Plan A — Caller-budgeted template selection + +**Goal:** `getblocktemplate` and regtest `generate` build from one +`MempoolHub` call that takes the weight and sigop budget and returns each +selected tx with the fee and sigop cost read under the same lock. Today +`select_block_txs(min_sat_kvb)` fixes the weight at +`template_tx_weight()`, the sigop reserve is graph-global +(`set_block_reserved_sigops` at open), and GBT re-reads `fee` / `sigops` +per tx via `get_live_meta` / `get_live_sigop_cost` after the read lock +drops. A tx evicted in between reports `fee: 0` / `sigops: 0` and +understates `coinbasevalue`. + +No new flag; GBT output for an unchanged mempool is unchanged. + +### A1 — Budget and per-tx meta through the graph + +- **Contract:** `TxGraph::select_block_template(budget, delta)` with + `SelectBudget { max_weight_wu, reserved_sigops, min_sat_kvb }` returns + `Vec` in mining order. At + `reserved_sigops = block_reserved_sigops()` and `max_weight_wu = + template_tx_weight()` it equals today's `select_block_txids_delta`. A + larger `reserved_sigops` skips a chunk that fit before and still takes a + later one; a smaller `max_weight_wu` likewise. `fee_sat` is the base fee + (not the delta-modified fee). +- **Red:** `cargo test -p rbitcoin-mempool select_budget_` — reuse the + `CreateBigSigOpsCluster` / skip-and-continue fixtures from the sigop + budget work; reserved-sigops edge and weight edge. +- **Green:** thread the budget through the existing selection loop; + `select_block_txids_delta` becomes a thin wrapper (or goes away if its + callers move). +- **Refactor:** one selection loop; drop the graph-global reserve's + template use if every template caller now passes it. +- **Verify:** `cargo test -p rbitcoin-mempool select_` + +### A2 — Hub and GBT on the budgeted call + +- **Contract:** `MempoolHub::select_block_template(budget)` returns + `Vec<(Transaction, fee_sat, sigop_cost)>` from one read lock plus the + `prioritisetransaction` deltas. GBT `transactions[].fee` / `sigops` and + `coinbasevalue` come from it; `generate` uses the same call. A tx + removed after selection still reports its selected fee. +- **Red:** `cargo test -p rbitcoin-rpc gbt_` — existing GBT tests stay + green; new case: evict a selected tx between selection and JSON build + (hook or fixture), fee and sigops still match selection. +- **Green:** hub method; `mempool_block_txs` returns the triples; GBT + drops the per-tx `get_live_meta` / `get_live_sigop_cost` reads. +- **Refactor:** `select_block_txs` callers left over move to the new call + or keep a one-line wrapper. +- **Verify:** `cargo test -p rbitcoin-rpc --lib`, + `cargo test -p rbitcoin-net select_` + +--- + +## Plan B — Mine a block through the TP + +**Goal:** the operator outcome in [Goal](#goal), minus fee-delta pushes. +Ships the listener, bootstrap, tip push, transaction data, and +`SubmitSolution`. Requires Plan A. + +### B1 — Crate skeleton, Noise responder, SetupConnection + +- **Contract:** a `noise_sv2` initiator completing the NX handshake against + the listener and sending `SetupConnection{protocol=2, min_version=2, + max_version=2, flags=0}` receives `SetupConnection.Success{used_version=2, + flags=0}`. Nonzero flags → `SetupConnection.Error` echoing them. + `protocol != 2` or no version-2 overlap → Error and the connection + closes. +- **Red:** `cargo test -p rbitcoin-sv2 setup_connection_` — loopback TCP, + in-crate test initiator; success, bad-flags, bad-protocol cases. +- **Green:** `crates/rbitcoin-sv2` (workspace member) with the wire crates + pinned to the Step 0 set; authority-keypair config, listener task, + per-connection session task driving the `codec_sv2` handshake then the + common-message branch. +- **Refactor:** session state as an enum (`Handshake`, + `AwaitingConstraints`, `Active`), not nested ifs. +- **Verify:** `cargo test -p rbitcoin-sv2 setup_` + +### B2 — Merkle path helper in consensus + +- **Contract:** `coinbase_merkle_path(txids)` returns the leftmost-branch + hashes deepest-first; folding them with the coinbase txid reproduces + `merkle_root_bytes` for the same list. Edge cases: single tx (empty + path), odd counts at every level. +- **Red:** `cargo test -p rbitcoin-consensus merkle_path_` — small known + vectors. +- **Green:** helper next to `merkle_root_bytes` + (`crates/rbitcoin-consensus/src/block/mod.rs`). +- **Refactor:** share the level-pairing loop with the root computation if + it dedupes without obscuring. +- **Verify:** `cargo test -p rbitcoin-consensus merkle_path_` + +### B3 — Template builder with TDP coinbase + +- **Contract:** `build(hub, tip, constraints) -> TemplateRecord` calls + `MempoolHub::select_block_template` with the per-session budget + ([Constraints](#constraints-all-plans)); `coinbase_prefix` is the BIP34 + height push; `value_remaining` = subsidy + Σ selected fees (from the + selection, not a re-read); outputs = witness commitment last; + `merkle_path` from B2; the record carries the serialized non-coinbase + txs in selection order. +- **Red:** `cargo test -p rbitcoin-sv2 template_` — synthetic mempool + (reuse `rbitcoin-mempool` accept fixtures): weight bound at the reserved + edge, sigops at a large `max_additional_sigops`, fee sum, prefix bytes, + commitment, tx order. +- **Green:** builder module in `rbitcoin-sv2`; subsidy/params from + `rbitcoin-consensus`. +- **Refactor:** none expected (commitment and selection already have one + owner). +- **Verify:** `cargo test -p rbitcoin-sv2 template_` + +### B4 — Node wiring + bootstrap flow + +- **Contract:** with `--sv2-tp-listen` set, the node serves the listener; a + client completing setup and sending `CoinbaseOutputConstraints` + immediately receives `NewTemplate{future_template: true}` then + `SetNewPrevHash` with the same `template_id`, the current tip as + `prev_hash`, and matching nBits/target. While the GBT sync gate says + not-synced, the session holds the constraints and sends the first + template when the gate clears. +- **Red:** `cargo test -p rbitcoin-test sv2_tp_bootstrap` — one regtest + node, full handshake → setup → constraints; assert NewTemplate fields + against the node tip and mempool, and SetNewPrevHash consistency. Gate + predicate unit in `rbitcoin-sv2`. +- **Green:** `run.rs` service start behind `--sv2-tp-listen` / + `--sv2-tp-authority-sec` / `--sv2-tp-cert-validity`; session loop calls + the builder on first constraints; per-session template map. +- **Refactor:** flag plumbing follows the `esplora_block_template` config + pattern. +- **Verify:** `cargo test -p rbitcoin-test sv2_tp_bootstrap`, + `cargo test -p rbitcoin-node sv2_tp_` + +### B5 — RequestTransactionData + +- **Contract:** a live `template_id` → + `RequestTransactionData.Success{template_id, excess_data: "", + transaction_list}` with the witness-serialized txs in template order; + unknown id → `RequestTransactionData.Error{error_code: + "template-id-not-found"}`. +- **Red:** extend the B4 journey: request the served template's data, + assert count/order/bytes against the mempool txs; unknown-id error. +- **Green:** session cache read path. +- **Refactor:** none expected. +- **Verify:** same journey filter. + +### B6 — Tip-change push + stale grace + +- **Contract:** after the node accepts a new tip block, every active + session receives `NewTemplate{future_template: true}` then + `SetNewPrevHash` on the new `prev_hash`, `template_id` still increasing. + During `--sv2-tp-stale-grace` the old template still answers + `RequestTransactionData`; after the grace it answers + `"stale-template-id"`. Future templates for the old prev hash retire the + same way. +- **Red:** journey: generate a block via the harness RPC, assert the push + pair arrives without client polling; stale-id behavior before and after + the grace (the harness sets it small). +- **Green:** `ChainHub::subscribe_tips()` consumer; rebuild per session + with its constraints; retire on the grace timer. +- **Refactor:** one "publish template" path shared by bootstrap and tip + (Plan C adds the fee trigger to it). +- **Verify:** journey filter; `cargo test -p rbitcoin-sv2 tip_` + +### B7 — SubmitSolution → accept_block + +- **Contract:** a client solving the served template sends + `SubmitSolution{template_id, version, header_timestamp, header_nonce, + coinbase_tx}` (full witness coinbase); the node assembles header (prev + + recomputed merkle + message fields) + coinbase + retained txs and runs + `ChainHub::accept_block`; the tip advances. Unknown/stale template or + undecodable coinbase → log and drop. Timestamp-window pre-check per + [Constraints](#constraints-all-plans). +- **Red:** journey: grind a regtest nonce on the served template, submit, + assert the new tip hash; a garbage-coinbase submission leaves tip and + session healthy. +- **Green:** assembly + pre-checks in a blocking region; accept via + ChainHub. +- **Refactor:** solution assembly shares the B2 merkle fold. +- **Verify:** journey filter. + +### B8 — Operator surface + +- **Contract:** [`OPERATOR.md`](../OPERATOR.md) documents + `--sv2-tp-listen`, `--sv2-tp-authority-sec`, `--sv2-tp-cert-validity`, + `--sv2-tp-stale-grace`. [`COMPAT.md`](../COMPAT.md) gains the SV2 TDP + row (the "no stratum" row stays; that row is v1 stratum/pool). + First-class `services.rbitcoin.sv2.tp.*` options in + [`nix/modules/rbitcoin.nix`](../nix/modules/rbitcoin.nix) with argv + asserts in `nixos-module-eval.nix`. +- **Red:** eval assert for the flags; docs need no test. +- **Green:** options + docs. +- **Refactor:** `extraArgs` still appends last. +- **Verify:** `nix build .#checks.x86_64-linux.nixos-module-eval --no-link` + +--- + +## Plan C — Fee-delta push + +**Goal:** with the tip unchanged, connected clients get a fresh template +when fees rise enough to matter, throttled. Requires Plan B. + +### C1 — Fee-delta push + +- **Contract:** with the tip unchanged, when `MempoolHub::template_updates` + advances and a rebuilt template's total fees exceed the last sent by + `--sv2-tp-fee-delta` sats, and at least `--sv2-tp-template-interval` + seconds passed since the last push, the session sends + `NewTemplate{future_template: false}` with **no** `SetNewPrevHash`. + Below the delta or inside the interval: nothing. +- **Red:** extend the B journey: submit higher-fee txs via the harness + RPC, assert the push; submit a fee-trivial tx, assert silence; assert + the interval throttle. +- **Green:** watch task on the counter; per-session last-sent fee/instant; + feeds the B6 publish path. +- **Refactor:** share the throttle predicate between the decision and its + unit. +- **Verify:** journey filter. + +### C2 — Operator surface + +- **Contract:** OPERATOR documents `--sv2-tp-fee-delta` and + `--sv2-tp-template-interval`; `services.rbitcoin.sv2.tp.*` gains both + with argv asserts. +- **Red / Green / Refactor / Verify:** as B8. + +--- + +## Test budget + +Units in `rbitcoin-mempool` (budgeted selection), `rbitcoin-consensus` +(merkle path), and `rbitcoin-sv2` (builder, throttle, gate). **One** +regtest integration journey in `rbitcoin-test`, opened in B4 and extended +by B5–B7 and C1 — one node open, per [`TESTING.md`](../TESTING.md) budgets. +No live pool/JDC, no mainnet datadir, no plaintext mode. + +## Risks / follow-ups + +- stratum-core publishes the wire crates on crates.io while sv2-apps + git-pins the meta-crate; crates.io versions may lag sv2-tp behavior. B1 + pins the Step 0 set. Interop against the sv2-apps integration-tests + (their JDC against this TP) is a host follow-up, not default CI. +- A bump of the wire crates that fails `cargo deny` reopens the hand-roll + fallback (name the dep that forced it). +- `secp256k1` 0.28 + `secp256k1-sys` 0.9 compile a second libsecp: compile + time and binary size, not correctness (distinct symbol prefixes). Drops + when `noise_sv2` moves to 0.29. +- Authority-cert rotation: certs are short-lived + (`--sv2-tp-cert-validity`); rotation is restart-with-new-cert in + OPERATOR. Hot rotation is a follow-up. +- After ship: JD-server mode, weak blocks, extension negotiation, per-IP + connection limits → quality.md rows, not this roadmap. From 493256478eee7c3e627dc1918f3fd455e6304f6c Mon Sep 17 00:00:00 2001 From: Gary Krause Date: Mon, 28 Sep 2026 14:02:13 -0400 Subject: [PATCH 2/7] mempool: select block templates against a caller budget TxGraph::select_block_template takes a SelectBudget (weight, sigop reserve, -blockmintxfee floor) and returns each pick with its base fee and sigop cost, read under the same graph borrow as the selection. The graph-global reserve now serves admission only; ActiveMempool passes it for the existing template path until the hub moves to the new call. select_block_txids / select_block_txids_delta go away. The sigop skip-and-continue unit now drives the reserve through the budget and absorbs the configured-reserve unit. Co-Authored-By: Claude Opus 5.5 --- crates/rbitcoin-mempool/src/accept.rs | 11 +- crates/rbitcoin-mempool/src/graph.rs | 191 ++++++++++++++++++-------- crates/rbitcoin-mempool/src/lib.rs | 2 +- docs/sv2-template-provider.md | 21 ++- 4 files changed, 154 insertions(+), 71 deletions(-) diff --git a/crates/rbitcoin-mempool/src/accept.rs b/crates/rbitcoin-mempool/src/accept.rs index d471cbe56..0700c873a 100644 --- a/crates/rbitcoin-mempool/src/accept.rs +++ b/crates/rbitcoin-mempool/src/accept.rs @@ -1,7 +1,7 @@ //! Single-tx accept: Libre policy + cluster limits + durable slot write. use crate::error::MempoolError; -use crate::graph::{sigops_adjusted_weight, TxEntry, TxGraph, MAX_BLOCK_SIGOPS_COST}; +use crate::graph::{sigops_adjusted_weight, SelectBudget, TxEntry, TxGraph, MAX_BLOCK_SIGOPS_COST}; use crate::orphanage::Orphanage; use crate::packed::VinAux; use crate::store::Mempool; @@ -1837,10 +1837,15 @@ impl ActiveMempool { min_sat_kvb: u64, delta: impl Fn(Txid) -> i64, ) -> Vec { + let budget = SelectBudget { + max_weight_wu, + reserved_sigops: self.graph.block_reserved_sigops(), + min_sat_kvb, + }; self.graph - .select_block_txids_delta(max_weight_wu, min_sat_kvb, delta) + .select_block_template(budget, delta) .into_iter() - .filter_map(|id| self.get_tx(&id).cloned()) + .filter_map(|s| self.get_tx(&s.txid).cloned()) .collect() } } diff --git a/crates/rbitcoin-mempool/src/graph.rs b/crates/rbitcoin-mempool/src/graph.rs index ef5c7d128..33271f2b2 100644 --- a/crates/rbitcoin-mempool/src/graph.rs +++ b/crates/rbitcoin-mempool/src/graph.rs @@ -46,6 +46,28 @@ fn meets_block_min_feerate(modified_sat: i128, adj_weight_wu: u64, min_sat_kvb: modified_sat.saturating_mul(1000) >= i128::from(min_sat_kvb) * i128::from(vsize) } +/// Caller's block budget for [`TxGraph::select_block_template`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct SelectBudget { + /// Raw weight (WU) available to mempool txs. + pub max_weight_wu: u64, + /// Sigop cost held back for the coinbase (Core `nBlockSigOpsCost` start). + pub reserved_sigops: u64, + /// `-blockmintxfee` chunk floor on modified fee (sat/kvB). + pub min_sat_kvb: u64, +} + +/// One tx picked by [`TxGraph::select_block_template`], read under the same +/// graph borrow as the selection. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Selected { + pub txid: Txid, + /// Base fee (sat), not the `prioritisetransaction`-modified fee. + pub fee_sat: u64, + /// Full BIP16 + BIP141 sigop cost recorded at admission. + pub sigop_cost: u64, +} + /// One live mempool entry (RAM index; body lives on disk). #[derive(Debug, Clone)] pub struct TxEntry { @@ -224,7 +246,7 @@ impl TxGraph { self.bytes_per_sigop } - /// Set the shared admission and block-template sigop reserve. + /// Set the admission sigop reserve (a tx must fit a block beside it). pub(crate) fn set_block_reserved_sigops(&mut self, reserved: u64) { self.block_reserved_sigops = reserved; } @@ -951,28 +973,29 @@ impl TxGraph { Self::MAX_BLOCK_WEIGHT.saturating_sub(Self::DEFAULT_BLOCK_RESERVED_WEIGHT) } - /// Mining-order txids that fit in `max_weight_wu` (best chunks first). + /// Mining-order txs that fit `budget` (best chunks first), ranking by + /// `base_fee + delta(txid)`. /// - /// Empty pool or zero cap → `[]`. A high-feerate child chunk pulls in + /// Empty pool or zero weight → `[]`. A high-feerate child chunk pulls in /// still-unselected in-mempool ancestors so the block is topological. - /// A chunk (plus those ancestors) that would overflow the weight cap or the - /// block sigop budget (80_000 less `block_reserved_sigops`) is skipped; - /// later chunks are still tried. - pub fn select_block_txids(&self, max_weight_wu: u64) -> Vec { - self.select_block_txids_delta(max_weight_wu, 0, |_| 0) - } - - /// Like [`Self::select_block_txids`], ranking by `base_fee + delta(txid)`. + /// A chunk (plus those ancestors) that would overflow + /// `budget.max_weight_wu` or the block sigop limit (80_000 less + /// `budget.reserved_sigops`) is skipped; later chunks are still tried. /// Chunks whose modified fee is **negative** are skipped. A chunk whose - /// modified feerate is under `min_sat_kvb` (`-blockmintxfee`) is skipped - /// whole (Core `BlockAssembler` chunk floor), so a low-fee parent and its - /// CPFP child go in or out together. `0` admits zero-fee chunks. - pub fn select_block_txids_delta( + /// modified feerate is under `budget.min_sat_kvb` (`-blockmintxfee`) is + /// skipped whole (Core `BlockAssembler` chunk floor), so a low-fee parent + /// and its CPFP child go in or out together. `0` admits zero-fee chunks. + /// Each [`Selected`] carries the base fee (not the modified fee). + pub fn select_block_template( &self, - max_weight_wu: u64, - min_sat_kvb: u64, + budget: SelectBudget, delta: impl Fn(Txid) -> i64, - ) -> Vec { + ) -> Vec { + let SelectBudget { + max_weight_wu, + reserved_sigops, + min_sat_kvb, + } = budget; if max_weight_wu == 0 { return Vec::new(); } @@ -994,7 +1017,7 @@ impl TxGraph { let mut selected = HashSet::new(); let mut out = Vec::new(); let mut used = 0u64; - let mut sigops = self.block_reserved_sigops; + let mut sigops = reserved_sigops; for (_, _, ch) in scored { let mut add = Vec::new(); for t in &ch.txids { @@ -1026,7 +1049,13 @@ impl TxGraph { sigops = sigops.saturating_add(extra_sigops); for t in add { selected.insert(t); - out.push(t); + if let Some(e) = self.entries.get(&t) { + out.push(Selected { + txid: t, + fee_sat: e.fee_sat, + sigop_cost: e.sigop_cost, + }); + } } } out @@ -1294,10 +1323,8 @@ mod tests { #[test] fn select_block_txids_empty_parent_before_child_and_weight_cap() { let g = TxGraph::new(); - assert!(g - .select_block_txids(TxGraph::template_tx_weight()) - .is_empty()); - assert!(g.select_block_txids(0).is_empty()); + assert!(select_ids(&g, budget(TxGraph::template_tx_weight()), |_| 0).is_empty()); + assert!(select_ids(&g, budget(0), |_| 0).is_empty()); let mut g = TxGraph::new(); let parent = spend_op([1u8; 32], 50_000, 40_000); @@ -1321,7 +1348,7 @@ mod tests { // Child pays more than parent so its chunk ranks first — still emit parent first. g.insert(entry_for(&parent, 1_000, 0), &parent); g.insert(entry_for(&child, 10_000, 1), &child); - let order = g.select_block_txids(TxGraph::template_tx_weight()); + let order = select_ids(&g, budget(TxGraph::template_tx_weight()), |_| 0); assert_eq!( order, vec![parent.compute_txid(), child.compute_txid()], @@ -1335,15 +1362,15 @@ mod tests { let wl = lo.weight().to_wu(); g.insert(entry_for(&hi, 10_000, 0), &hi); g.insert(entry_for(&lo, 1_000, 1), &lo); - let only_hi = g.select_block_txids(wh); + let only_hi = select_ids(&g, budget(wh), |_| 0); assert_eq!(only_hi, vec![hi.compute_txid()]); - let both = g.select_block_txids(wh.saturating_add(wl)); + let both = select_ids(&g, budget(wh.saturating_add(wl)), |_| 0); assert_eq!(both, vec![hi.compute_txid(), lo.compute_txid()]); - assert!(g.select_block_txids(wh.saturating_sub(1)).is_empty()); + assert!(select_ids(&g, budget(wh.saturating_sub(1)), |_| 0).is_empty()); let hid = hi.compute_txid(); let lid = lo.compute_txid(); - let depri = g.select_block_txids_delta(TxGraph::template_tx_weight(), 0, |id| { + let depri = select_ids(&g, budget(TxGraph::template_tx_weight()), |id| { if id == hid { -10_000 } else { @@ -1355,7 +1382,7 @@ mod tests { vec![lid, hid], "zero modified fee stays selectable; hotter lid ranks first" ); - let depri_neg = g.select_block_txids_delta(TxGraph::template_tx_weight(), 0, |id| { + let depri_neg = select_ids(&g, budget(TxGraph::template_tx_weight()), |id| { if id == hid { -10_001 } else { @@ -1363,7 +1390,7 @@ mod tests { } }); assert_eq!(depri_neg, vec![lid], "negative modified fee is not mined"); - let bump = g.select_block_txids_delta(TxGraph::template_tx_weight(), 0, |id| { + let bump = select_ids(&g, budget(TxGraph::template_tx_weight()), |id| { if id == lid { 86 * 100_000_000 } else { @@ -1371,6 +1398,14 @@ mod tests { } }); assert_eq!(bump[0], lid, "i64-sized delta reorders selection"); + let bumped = g.select_block_template(budget(TxGraph::template_tx_weight()), |id| { + if id == lid { + 86 * 100_000_000 + } else { + 0 + } + }); + assert_eq!(bumped[0].fee_sat, 1_000, "selection reports the base fee"); // Child deprioritised to 0 stays out even when it shares a package with parent. let mut g = TxGraph::new(); @@ -1396,7 +1431,7 @@ mod tests { g.insert(entry_for(&child, 1_000, 1), &child); let cid = child.compute_txid(); let pid = parent.compute_txid(); - let only_p = g.select_block_txids_delta(TxGraph::template_tx_weight(), 0, |id| { + let only_p = select_ids(&g, budget(TxGraph::template_tx_weight()), |id| { if id == cid { -1_000 } else { @@ -1408,7 +1443,7 @@ mod tests { vec![pid, cid], "zero-modified child stays selectable with parent" ); - let only_p_neg = g.select_block_txids_delta(TxGraph::template_tx_weight(), 0, |id| { + let only_p_neg = select_ids(&g, budget(TxGraph::template_tx_weight()), |id| { if id == cid { -1_001 } else { @@ -1436,13 +1471,14 @@ mod tests { let cap = hot.weight().to_wu() + cold.weight().to_wu(); assert!(big.weight().to_wu() > cold.weight().to_wu()); assert_eq!( - g.select_block_txids(cap), + select_ids(&g, budget(cap), |_| 0), vec![hot.compute_txid(), cold.compute_txid()] ); } - /// Sigop budget starts at the 400 coinbase reserve; a chunk reaching - /// 80_000 is skipped (Core `>=`) and a later, cheaper chunk still fits. + /// Sigop budget starts at the caller's reserve; a chunk reaching 80_000 + /// is skipped (Core `>=`) and a later, cheaper chunk still fits. Each pick + /// carries the base fee and sigop cost it was budgeted with. #[test] fn select_budgets_sigops_skip_and_continue() { let heavy = spend_op([8u8; 32], 50_000, 40_000); @@ -1458,29 +1494,55 @@ mod tests { let mut e = entry_for(&light, 1_000, 1); e.sigop_cost = 1; g.insert(e, &light); - g.select_block_txids(TxGraph::template_tx_weight()) + g }; - assert_eq!(pool(79_600), vec![lid], "400 + 79_600 hits the limit"); - assert_eq!(pool(79_599), vec![hid], "79_999 fits; +1 reaches 80_000"); - assert_eq!(pool(79_598), vec![hid, lid], "80_000 - 1 total fits"); - assert_eq!(pool(u64::MAX), vec![lid], "unknown cost never selected"); - } + let at = |g: &TxGraph, reserved_sigops| { + let b = SelectBudget { + reserved_sigops, + ..budget(TxGraph::template_tx_weight()) + }; + select_ids(g, b, |_| 0) + }; + let full = |heavy_cost| at(&pool(heavy_cost), COINBASE_SIGOPS_RESERVE); + assert_eq!(full(79_600), vec![lid], "400 + 79_600 hits the limit"); + assert_eq!(full(79_599), vec![hid], "79_999 fits; +1 reaches 80_000"); + assert_eq!(full(79_598), vec![hid, lid], "80_000 - 1 total fits"); + assert_eq!(full(u64::MAX), vec![lid], "unknown cost never selected"); - #[test] - fn selection_uses_configured_sigop_reserve() { - let tx = spend_op([0x18u8; 32], 50_000, 49_000); - let mut g = TxGraph::new(); - let mut e = entry_for(&tx, 10_000, 0); - e.sigop_cost = 79_999; - g.insert(e, &tx); - assert!(g - .select_block_txids(TxGraph::template_tx_weight()) - .is_empty()); - g.set_block_reserved_sigops(0); + let g = pool(79_598); assert_eq!( - g.select_block_txids(TxGraph::template_tx_weight()), - vec![tx.compute_txid()] + g.select_block_template(budget(TxGraph::template_tx_weight()), |_| 0), + vec![ + Selected { + txid: hid, + fee_sat: 10_000, + sigop_cost: 79_598 + }, + Selected { + txid: lid, + fee_sat: 1_000, + sigop_cost: 1 + }, + ] ); + assert_eq!(at(&g, 401), vec![hid], "a larger reserve drops the tail"); + assert_eq!(at(&g, 402), vec![lid], "skips heavy, still takes light"); + assert_eq!(at(&g, 0), vec![hid, lid]); + } + + fn budget(max_weight_wu: u64) -> SelectBudget { + SelectBudget { + max_weight_wu, + reserved_sigops: COINBASE_SIGOPS_RESERVE, + min_sat_kvb: 0, + } + } + + fn select_ids(g: &TxGraph, budget: SelectBudget, delta: impl Fn(Txid) -> i64) -> Vec { + g.select_block_template(budget, delta) + .into_iter() + .map(|s| s.txid) + .collect() } fn spend_op(seed: [u8; 32], _inv: u64, outv: u64) -> Transaction { @@ -1789,7 +1851,7 @@ mod tests { he.sigop_cost = 1_000; // 20_000 WU at 20 B/sigop g.insert(he, &heavy); g.insert(entry_for(&light, 1_000, 1), &light); - let order = |g: &TxGraph| g.select_block_txids(TxGraph::template_tx_weight()); + let order = |g: &TxGraph| select_ids(g, budget(TxGraph::template_tx_weight()), |_| 0); assert_eq!(order(&g), vec![lid, hid]); assert_eq!(g.worst_chunk().unwrap().1.txids, vec![hid]); assert_eq!(g.mining_chunks_best_first()[1].weight, 20_000); @@ -1829,7 +1891,16 @@ mod tests { g.insert(entry_for(&p, 0, 0), &p); g.insert(entry_for(&c, 100_000, 1), &c); g.insert(entry_for(&lone, lv, 2), &lone); - let sel = |min| g.select_block_txids_delta(TxGraph::template_tx_weight(), min, |_| 0); + let sel = |min| { + select_ids( + &g, + SelectBudget { + min_sat_kvb: min, + ..budget(TxGraph::template_tx_weight()) + }, + |_| 0, + ) + }; let (cid, lid) = (c.compute_txid(), lone.compute_txid()); assert_eq!(sel(1_000), vec![pid, cid, lid]); assert_eq!(sel(1_001), vec![pid, cid]); @@ -1849,7 +1920,7 @@ mod tests { } let cap = a.weight().to_wu() + b.weight().to_wu(); assert!(cap < 20_000); - assert_eq!(g.select_block_txids(cap).len(), 2); + assert_eq!(select_ids(&g, budget(cap), |_| 0).len(), 2); } /// Post-migrate recompute: filling an unknown (`u64::MAX`) cost re-ranks @@ -1866,13 +1937,13 @@ mod tests { g.insert(entry_for(&poor, 100, 1), &poor); assert_eq!(g.worst_chunk().unwrap().1.txids, vec![rid]); assert_eq!( - g.select_block_txids(TxGraph::template_tx_weight()), + select_ids(&g, budget(TxGraph::template_tx_weight()), |_| 0), vec![pid] ); g.set_sigop_cost(&rid, 0); assert_eq!(g.worst_chunk().unwrap().1.txids, vec![pid]); assert_eq!( - g.select_block_txids(TxGraph::template_tx_weight()), + select_ids(&g, budget(TxGraph::template_tx_weight()), |_| 0), vec![rid, pid] ); } diff --git a/crates/rbitcoin-mempool/src/lib.rs b/crates/rbitcoin-mempool/src/lib.rs index 9147f6ce3..32cb61b8a 100644 --- a/crates/rbitcoin-mempool/src/lib.rs +++ b/crates/rbitcoin-mempool/src/lib.rs @@ -55,7 +55,7 @@ pub use fee_est::{ pub use fee_flow::FeeFlowMeter; pub use graph::{ frontier_feerate_from_chunks, weight_above_from_chunks, Chunk, Cluster, MempoolGraphStats, - TxEntry, TxGraph, + SelectBudget, Selected, TxEntry, TxGraph, }; pub use orphanage::{OrphanSnapshot, Orphanage}; pub use packed::VinAux; diff --git a/docs/sv2-template-provider.md b/docs/sv2-template-provider.md index ed1abcb39..58252ce1a 100644 --- a/docs/sv2-template-provider.md +++ b/docs/sv2-template-provider.md @@ -145,14 +145,21 @@ No new flag; GBT output for an unchanged mempool is unchanged. larger `reserved_sigops` skips a chunk that fit before and still takes a later one; a smaller `max_weight_wu` likewise. `fee_sat` is the base fee (not the delta-modified fee). -- **Red:** `cargo test -p rbitcoin-mempool select_budget_` — reuse the - `CreateBigSigOpsCluster` / skip-and-continue fixtures from the sigop - budget work; reserved-sigops edge and weight edge. +- **Red:** `cargo test -p rbitcoin-mempool select_` — the graph selector + is a pure unit (no session passes a non-default budget before Plan B). + Extend the existing `select_budgets_sigops_skip_and_continue` (per-tx + base fee and sigop cost; reserve 401 drops the light tail, 402 skips + the heavy chunk and still takes the light one) and pin base fee + under a delta in the existing delta unit; fold + `selection_uses_configured_sigop_reserve` into it. The weight edge is + the existing `select_skips_overweight_chunk_and_continues` on the + budget. No new `select_budget_` twin. - **Green:** thread the budget through the existing selection loop; - `select_block_txids_delta` becomes a thin wrapper (or goes away if its - callers move). -- **Refactor:** one selection loop; drop the graph-global reserve's - template use if every template caller now passes it. + `select_block_txids` / `select_block_txids_delta` go away. + `ActiveMempool::select_block_txs_delta` passes the admission reserve + until A2 moves its caller. +- **Refactor:** one selection loop; the graph-global reserve is admission + only, templates take the caller's. - **Verify:** `cargo test -p rbitcoin-mempool select_` ### A2 — Hub and GBT on the budgeted call From 45d4ae55c0d056fefd908c1fcf61eb96ca6dabf6 Mon Sep 17 00:00:00 2001 From: Gary Krause Date: Mon, 28 Sep 2026 14:36:50 -0400 Subject: [PATCH 3/7] rpc: build getblocktemplate from the budgeted hub selection MempoolHub::select_block_template(budget) returns each selected tx with its base fee and sigop cost from one read lock; template_budget(min) is the node's own budget (template weight, configured sigop reserve). GBT and generate use it, so GBT no longer re-reads fee and sigops per tx after the lock drops: a tx evicted in between used to report fee 0 and understate coinbasevalue. Plan B passes per-client budgets to the same call. select_block_txs (hub and ActiveMempool) and get_live_sigop_cost have no callers left and go away; the hub sigop tests read through the selection instead, and the chain-ops GBT beat pins the reported fee. Co-Authored-By: Claude Opus 5.5 --- TESTING.md | 2 +- changelog.d/gbt-selection-meta.md | 7 ++ crates/rbitcoin-mempool/src/accept.rs | 35 +++++----- .../src/accept_life_journey.rs | 4 +- crates/rbitcoin-net/src/lib.rs | 2 +- crates/rbitcoin-net/src/tx_relay.rs | 68 +++++++++++++------ crates/rbitcoin-rpc/src/methods/mine.rs | 35 ++++------ crates/rbitcoin-rpc/src/methods/mod.rs | 2 +- .../src/regtest_chain_ops_journey.rs | 4 ++ docs/rpc.md | 4 +- docs/sv2-template-provider.md | 35 ++++++---- 11 files changed, 119 insertions(+), 79 deletions(-) create mode 100644 changelog.d/gbt-selection-meta.md diff --git a/TESTING.md b/TESTING.md index 5b5642adb..ea9893009 100644 --- a/TESTING.md +++ b/TESTING.md @@ -313,7 +313,7 @@ Prefer **one high-level scenario** per behavior cluster. Delete lower-level test | `three_stage_confirm_and_parent_pin_surface` | Consensus+query | Split load→scripts→write of pad+spend from genesis (header-plan BIP68 MTP); parent pin; load ready timeout/cancel; instance-owned `last_write` / `last_pin` / `take_window` meters (a second engine's window stays empty); txstat fee / size / weight from the load assemble, restamp and its refuses; same-run create then spend; 546-shaped 2-vout merge + same-block chain + cross-batch head resolve; an already-at-height retry finishes a spend annotate | | `mempool_under_pressure` | Mempool + RPC (crate) | One entry in `orphanage`, `accept`, `tx_relay`, and `methods_tests`: orphan reserve and expiry, sigops before script, rolling fee floor, cluster cap, parked min-relay orphan, and the package RPC rejects (unsorted, missing inputs, conflict, min-relay parent with maxfeerate child). | | `mempool_accept_life` | Mempool (crate) | One `ActiveMempool` against one chain view that blocks move. Sigop-adjusted vsize (boundary, min relay, full-pool floor, RBF, package and 1p1c), the raw-weight cluster limit, the shared block sigop budget, and sigop cost plus bytes-per-sigop and reserve overlays across reopen and compact. Orphan parks and re-announce, dry run not parked, parent promotes the child; missing vout, invalid parent, and block-spent coin reject without parking. Full RBF and no return, the staged commit failing closed on a conflict that landed after prepare, pure RBFR unpinning a child, replaced txs out of the cluster count; a ~30 kvB single tx under the vsize cap and the ten-way merge over it. Package order, CPFP, child fail restoring the RBF victim. A block evicts double-spent txs with descendants; a reorg readmits the parent and evicts the BIP68 and coinbase-maturity spends. Raised `-minrelaytxfee`: 1p1c needs a paying child, an unrelated tx does not ride the waiver, child fail takes a promoted spender down. Full pool: a protected lone worst chunk evicts nothing and leaves the floor, the next arrival evicts the CPFP pair together. | -| `rpc_regtest_chain_ops` | RPC (crate) | One regtest hub from genesis through `dispatch`. At genesis: `size_on_disk` is the store walk, IBD comes from the hub and not the stale atomic, buried deployments, `generateblock submit=false` connects nothing, and the priority, mocktime, mockscheduler, submitheader decode, and not-found refuses. The first block pays a p2wpkh address: display-order hashes and txids, raw `getblock` and header, a headers-only child at progress 0.5. Mocktime stamps `generate` and makes a far block `time-too-new`. Coinbase-only blocks: verbosity 1 without a seqsigwit zip, `getnetworkhashps` over chainwork, the empty template and proposal needles, a `time-too-old` header, an invalid parent body that marks its branch, and the `submitblock` merkle, length, coinbase, duplicate, value, and missing-input rejects. After a 120-block pad: the `nblocks=0` window, GBT sigops (bare, P2SH, P2WSH), sigop-adjusted mempool vsize (`getmempoolentry`, package retry, `blockmintxfee`; weight and the Esplora/Electrum histogram stay raw) and a big-sigops cluster under the block budget, a non-DER spend with Core `reject-details`, deprioritise, `generateblock` reject shapes then parent-first mining, a premature coinbase, proposal spend/value/final needles against the chain, default and explicit `maxfeerate`, `testmempoolaccept` known vs mempool vs archived, invalidate and reconsider, and a parked sibling (held `getblock`, `preciousblock`). Last, a mainnet view of the same hub refuses the regtest-only methods and still takes `submitblock`. | +| `rpc_regtest_chain_ops` | RPC (crate) | One regtest hub from genesis through `dispatch`. At genesis: `size_on_disk` is the store walk, IBD comes from the hub and not the stale atomic, buried deployments, `generateblock submit=false` connects nothing, and the priority, mocktime, mockscheduler, submitheader decode, and not-found refuses. The first block pays a p2wpkh address: display-order hashes and txids, raw `getblock` and header, a headers-only child at progress 0.5. Mocktime stamps `generate` and makes a far block `time-too-new`. Coinbase-only blocks: verbosity 1 without a seqsigwit zip, `getnetworkhashps` over chainwork, the empty template and proposal needles, a `time-too-old` header, an invalid parent body that marks its branch, and the `submitblock` merkle, length, coinbase, duplicate, value, and missing-input rejects. After a 120-block pad: the `nblocks=0` window, GBT fee and sigops (bare, P2SH, P2WSH), sigop-adjusted mempool vsize (`getmempoolentry`, package retry, `blockmintxfee`; weight and the Esplora/Electrum histogram stay raw) and a big-sigops cluster under the block budget, a non-DER spend with Core `reject-details`, deprioritise, `generateblock` reject shapes then parent-first mining, a premature coinbase, proposal spend/value/final needles against the chain, default and explicit `maxfeerate`, `testmempoolaccept` known vs mempool vs archived, invalidate and reconsider, and a parked sibling (held `getblock`, `preciousblock`). Last, a mainnet view of the same hub refuses the regtest-only methods and still takes `submitblock`. | | `block_cache_and_mempool_hub_surface` | Net | BlockCache locator/eviction + MempoolHub accept/remove/reorg on mature chain. `DEFAULT_BODY_DEPTH == 16` stays a unit. | | `store_error_and_corrupt_paths` | Store | Error/corrupt surfaces | | `store_table_header_and_idx_corrupt` | Store | Table header/head corrupt open | diff --git a/changelog.d/gbt-selection-meta.md b/changelog.d/gbt-selection-meta.md new file mode 100644 index 000000000..80659fbaf --- /dev/null +++ b/changelog.d/gbt-selection-meta.md @@ -0,0 +1,7 @@ +Fixed + +- **`getblocktemplate` fees come from the selection.** Each + transaction's `fee` and `sigops`, and the `coinbasevalue`, are read + under the same mempool lock that selected it. A transaction evicted + while the template was built no longer reports `fee: 0` and + understates `coinbasevalue`. diff --git a/crates/rbitcoin-mempool/src/accept.rs b/crates/rbitcoin-mempool/src/accept.rs index 0700c873a..2ef70d58c 100644 --- a/crates/rbitcoin-mempool/src/accept.rs +++ b/crates/rbitcoin-mempool/src/accept.rs @@ -1,7 +1,9 @@ //! Single-tx accept: Libre policy + cluster limits + durable slot write. use crate::error::MempoolError; -use crate::graph::{sigops_adjusted_weight, SelectBudget, TxEntry, TxGraph, MAX_BLOCK_SIGOPS_COST}; +use crate::graph::{ + sigops_adjusted_weight, SelectBudget, Selected, TxEntry, TxGraph, MAX_BLOCK_SIGOPS_COST, +}; use crate::orphanage::Orphanage; use crate::packed::VinAux; use crate::store::Mempool; @@ -502,7 +504,7 @@ impl ActiveMempool { self.graph.set_bytes_per_sigop(bytes_per_sigop); } - /// Set the sigop reserve shared by admission and block-template selection. + /// Set the sigop reserve shared by admission and [`Self::template_budget`]. pub fn set_block_reserved_sigops(&mut self, reserved_sigops: u64) { self.graph.set_block_reserved_sigops(reserved_sigops); } @@ -1824,28 +1826,27 @@ impl ActiveMempool { .collect() } - /// Mining-order live txs that fit in `max_weight_wu` (best chunks first). - pub fn select_block_txs(&self, max_weight_wu: u64) -> Vec { - self.select_block_txs_delta(max_weight_wu, 0, |_| 0) + /// This node's own block budget (GBT / `generate`): template weight and + /// the admission sigop reserve, with a `-blockmintxfee` floor. + pub fn template_budget(&self, min_sat_kvb: u64) -> SelectBudget { + SelectBudget { + max_weight_wu: TxGraph::template_tx_weight(), + reserved_sigops: self.graph.block_reserved_sigops(), + min_sat_kvb, + } } - /// Like [`Self::select_block_txs`] with `prioritisetransaction` fee deltas - /// and a `-blockmintxfee` chunk floor (sat/kvB). - pub fn select_block_txs_delta( + /// Mining-order live txs that fit `budget` (best chunks first) with + /// `prioritisetransaction` deltas, each with its [`Selected`] meta. + pub fn select_block_template( &self, - max_weight_wu: u64, - min_sat_kvb: u64, + budget: SelectBudget, delta: impl Fn(Txid) -> i64, - ) -> Vec { - let budget = SelectBudget { - max_weight_wu, - reserved_sigops: self.graph.block_reserved_sigops(), - min_sat_kvb, - }; + ) -> Vec<(Transaction, Selected)> { self.graph .select_block_template(budget, delta) .into_iter() - .filter_map(|s| self.get_tx(&s.txid).cloned()) + .filter_map(|s| self.get_tx(&s.txid).map(|tx| (tx.clone(), s))) .collect() } } diff --git a/crates/rbitcoin-mempool/src/accept_life_journey.rs b/crates/rbitcoin-mempool/src/accept_life_journey.rs index 96f3df681..ce7136681 100644 --- a/crates/rbitcoin-mempool/src/accept_life_journey.rs +++ b/crates/rbitcoin-mempool/src/accept_life_journey.rs @@ -630,7 +630,7 @@ fn sigop_block_budget(life: &mut Life) { .expect("79,520 fits beside the default reserve"); assert_eq!( life.mp - .select_block_txs(TxGraph::template_tx_weight()) + .select_block_template(life.mp.template_budget(0), |_| 0) .len(), 1 ); @@ -648,7 +648,7 @@ fn sigop_block_budget(life: &mut Life) { .expect("79,920 fits when the template reserve is zero"); assert_eq!( life.mp - .select_block_txs(TxGraph::template_tx_weight()) + .select_block_template(life.mp.template_budget(0), |_| 0) .len(), 1 ); diff --git a/crates/rbitcoin-net/src/lib.rs b/crates/rbitcoin-net/src/lib.rs index 5df6500ed..6a72f6479 100644 --- a/crates/rbitcoin-net/src/lib.rs +++ b/crates/rbitcoin-net/src/lib.rs @@ -60,8 +60,8 @@ pub use peers::{ parse_peer_addr, parse_peer_addr_with_port, parse_peer_net, pick_stale_follow_evict, DialRequest, DialTarget, LivePeer, PeerConnType, PeerHub, PeerInfo, PeerOut, PingAction, }; -pub use rbitcoin_mempool::AcceptError; pub(crate) use rbitcoin_mempool::MempoolGraphStats; +pub use rbitcoin_mempool::{AcceptError, Selected}; pub use reactor::BlockingRegion; pub use seeds::{ default_port, default_rpc_port, dns_seeds, fixed_seed_hosts, resolve_all_seeds, diff --git a/crates/rbitcoin-net/src/tx_relay.rs b/crates/rbitcoin-net/src/tx_relay.rs index 0db9a9918..0dc1b9651 100644 --- a/crates/rbitcoin-net/src/tx_relay.rs +++ b/crates/rbitcoin-net/src/tx_relay.rs @@ -13,8 +13,8 @@ use rbitcoin_mempool::{ blend_sat_kvb, block_p10_sat_kvb, fine_candidate_rates, flow_for_depth, frontier_feerate_from_chunks, historical_far_sat_kvb, hold_defined_then_monotone, min_rate_for_capacity, percentile_sat, weight_above_from_chunks, AcceptError, AcceptResult, - ActiveMempool, ChainPrevout, ChainTipCtx, Chunk, Coin, FeeFlowMeter, UtxoProvider, - BLOCK_WEIGHT_WU, MAX_PACKAGE_COUNT, + ActiveMempool, ChainPrevout, ChainTipCtx, Chunk, Coin, FeeFlowMeter, SelectBudget, Selected, + UtxoProvider, BLOCK_WEIGHT_WU, MAX_PACKAGE_COUNT, }; use rbitcoin_primitives::{Fk, Height}; use rbitcoin_query::Query; @@ -2745,17 +2745,20 @@ impl MempoolHub { } } - /// Block template / generate selection: mining-order live txs that fit - /// in a block (best chunks first), skipping chunks under `min_sat_kvb` - /// (`-blockmintxfee`) on modified fee. - pub fn select_block_txs(&self, min_sat_kvb: u64) -> Vec { + /// This node's own block budget (GBT / `generate`): template weight and + /// the configured sigop reserve, with a `-blockmintxfee` floor. + pub fn template_budget(&self, min_sat_kvb: u64) -> SelectBudget { + self.lock_read().template_budget(min_sat_kvb) + } + + /// Block template selection: mining-order live txs that fit `budget` + /// (best chunks first, `prioritisetransaction` deltas applied). Base fee + /// and sigop cost come from the same read lock as the selection, so a tx + /// evicted afterwards still reports what it was selected with. + pub fn select_block_template(&self, budget: SelectBudget) -> Vec<(Transaction, Selected)> { let deltas = self.fee_deltas.lock().unwrap().clone(); let g = self.lock_read(); - g.select_block_txs_delta( - rbitcoin_mempool::TxGraph::template_tx_weight(), - min_sat_kvb, - |id| deltas.get(&id).copied().unwrap_or(0), - ) + g.select_block_template(budget, |id| deltas.get(&id).copied().unwrap_or(0)) } /// Additive `prioritisetransaction` delta (sat). Zero total drops the entry. @@ -2854,11 +2857,6 @@ impl MempoolHub { g.graph.get(txid).map(|e| e.adjusted_weight(bps)) } - /// Full BIP16 + BIP141 sigop cost recorded at admission (GBT `sigops`). - pub fn get_live_sigop_cost(&self, txid: &Txid) -> Option { - self.lock_read().graph.get(txid).map(|e| e.sigop_cost) - } - /// Fee + sigop-adjusted weight for the feefilter announce gate (Core /// `txinfo.vsize` is `GetTxSize`). `None` if a writer holds `inner`. pub fn try_get_live_meta(&self, txid: &Txid) -> Option<(u64, u64)> { @@ -4100,10 +4098,14 @@ mod tests { store.flush().unwrap(); } let hub = MempoolHub::open(&mp, Arc::clone(&q)).unwrap(); - assert_eq!(hub.get_live_sigop_cost(&ok.compute_txid()), Some(4)); - assert_eq!( - hub.get_live_sigop_cost(&gone.compute_txid()), - None, + let picked: Vec<_> = hub + .select_block_template(hub.template_budget(0)) + .into_iter() + .map(|(_, s)| (s.txid, s.sigop_cost)) + .collect(); + assert_eq!(picked, vec![(ok.compute_txid(), 4)]); + assert!( + !hub.contains(&gone.compute_txid()), "unresolvable input evicted" ); let _ = std::fs::remove_dir_all(&mp); @@ -4163,7 +4165,13 @@ mod tests { }], }; hub.accept_tx(&tx).expect("16004 sigop cost fits a block"); - assert_eq!(hub.get_live_sigop_cost(&tx.compute_txid()), Some(16_004)); + let picked = hub.select_block_template(hub.template_budget(0)); + assert_eq!(picked.len(), 1); + assert_eq!(picked[0].0, tx); + assert_eq!( + (picked[0].1.fee_sat, picked[0].1.sigop_cost), + (100_000, 16_004) + ); let _ = std::fs::remove_dir_all(&mp); } @@ -4736,7 +4744,23 @@ mod tests { Some(0), ) .unwrap(); - assert_eq!(hub.get_live_sigop_cost(&txid), Some(79_920)); + // GBT's budget carries the configured reserve; each pick carries the + // fee and sigop cost read under the selection's lock. + let budget = hub.template_budget(0); + assert_eq!(budget.reserved_sigops, 0); + let picked = hub.select_block_template(budget); + assert_eq!(picked.len(), 1); + assert_eq!(picked[0].0, tx); + assert_eq!( + (picked[0].1.fee_sat, picked[0].1.sigop_cost), + (4_999_999_001, 79_920) + ); + // A caller reserving Core's 400 for its coinbase cannot fit it. + let core_reserve = rbitcoin_mempool::SelectBudget { + reserved_sigops: 400, + ..budget + }; + assert!(hub.select_block_template(core_reserve).is_empty()); assert!(hub.contains(&txid)); let _ = std::fs::remove_dir_all(&mempool_dir); let _ = std::fs::remove_dir_all(&store_dir); diff --git a/crates/rbitcoin-rpc/src/methods/mine.rs b/crates/rbitcoin-rpc/src/methods/mine.rs index 8bbff2513..1cddc3878 100644 --- a/crates/rbitcoin-rpc/src/methods/mine.rs +++ b/crates/rbitcoin-rpc/src/methods/mine.rs @@ -7,6 +7,7 @@ use bitcoin::{ Address, Amount, Block, BlockHash, Network as BtcNetwork, OutPoint, ScriptBuf, Transaction, Txid, }; +use rbitcoin_net::Selected; use rbitcoin_primitives::{Height, Network}; use serde_json::{json, Value}; use std::str::FromStr; @@ -107,7 +108,7 @@ pub(crate) fn hashes_json(hashes: &[BlockHash]) -> Value { json!(hashes.iter().map(|h| h.to_string()).collect::>()) } -pub(crate) fn mempool_block_txs(ctx: &RpcContext) -> Vec { +pub(crate) fn mempool_block_txs(ctx: &RpcContext) -> Vec<(Transaction, Selected)> { let min = ctx .chain .as_ref() @@ -115,7 +116,7 @@ pub(crate) fn mempool_block_txs(ctx: &RpcContext) -> Vec { .unwrap_or(1); ctx.mempool .as_ref() - .map(|mp| mp.select_block_txs(min)) + .map(|mp| mp.select_block_template(mp.template_budget(min))) .unwrap_or_default() } @@ -133,7 +134,10 @@ pub(crate) fn generate_with_mempool( script: ScriptBuf, ) -> Result { let miner = require_regtest_miner(ctx, "generate")?; - let extras = mempool_block_txs(ctx); + let extras: Vec = mempool_block_txs(ctx) + .into_iter() + .map(|(tx, _)| tx) + .collect(); let hashes = miner .generate_to_script(nblocks, script, extras.clone()) .map_err(|e| rpc_error(ERR_MISC, e))?; @@ -624,16 +628,9 @@ pub fn gbt_template(ctx: &RpcContext) -> Result { let selected = mempool_block_txs(ctx); let mut fees = 0u64; let mut tx_json = Vec::with_capacity(selected.len()); - let ids: Vec = selected.iter().map(Transaction::compute_txid).collect(); - for (i, tx) in selected.iter().enumerate() { - let txid = ids[i]; - let fee = ctx - .mempool - .as_ref() - .and_then(|mp| mp.get_live_meta(&txid)) - .map(|(f, _)| f) - .unwrap_or(0); - fees = fees.saturating_add(fee); + let ids: Vec = selected.iter().map(|(_, s)| s.txid).collect(); + for (tx, sel) in &selected { + fees = fees.saturating_add(sel.fee_sat); let mut depends = Vec::new(); for inp in &tx.input { if let Some(pos) = ids.iter().position(|t| *t == inp.previous_output.txid) { @@ -642,15 +639,11 @@ pub fn gbt_template(ctx: &RpcContext) -> Result { } tx_json.push(json!({ "data": serialize_hex(tx), - "txid": txid.to_string(), + "txid": sel.txid.to_string(), "hash": tx.compute_wtxid().to_string(), "depends": depends, - "fee": fee, - "sigops": ctx - .mempool - .as_ref() - .and_then(|mp| mp.get_live_sigop_cost(&txid)) - .unwrap_or(0), + "fee": sel.fee_sat, + "sigops": sel.sigop_cost, "weight": tx.weight().to_wu(), })); } @@ -662,7 +655,7 @@ pub fn gbt_template(ctx: &RpcContext) -> Result { } let wtxids: Vec<[u8; 32]> = selected .iter() - .map(|tx| tx.compute_wtxid().to_byte_array()) + .map(|(tx, _)| tx.compute_wtxid().to_byte_array()) .collect(); let witness_commit = rbitcoin_consensus::witness_commitment_script(wtxids, &[0u8; 32]); Ok(json!({ diff --git a/crates/rbitcoin-rpc/src/methods/mod.rs b/crates/rbitcoin-rpc/src/methods/mod.rs index 405e0ed7f..23d45858b 100644 --- a/crates/rbitcoin-rpc/src/methods/mod.rs +++ b/crates/rbitcoin-rpc/src/methods/mod.rs @@ -767,7 +767,7 @@ const NAMED_HELP: &[(&str, &str)] = &[ ( "getblocktemplate", "getblocktemplate (template_request)\n\ - All networks. Template from select_block_txs; proposal validates \ + All networks. Template from select_block_template; proposal validates \ without connecting. rules must include segwit. longpollid waits \ for a new tip or mempool/priority change. No BIP9 testdummy.", ), diff --git a/crates/rbitcoin-rpc/src/regtest_chain_ops_journey.rs b/crates/rbitcoin-rpc/src/regtest_chain_ops_journey.rs index f779bf628..c23667f6f 100644 --- a/crates/rbitcoin-rpc/src/regtest_chain_ops_journey.rs +++ b/crates/rbitcoin-rpc/src/regtest_chain_ops_journey.rs @@ -608,6 +608,7 @@ fn chain_ops_template_sigops_and_script_reject(ctx: &RpcContext, cbs: &mut TrueC assert_eq!(txs.len(), 1); assert_eq!(txs[0]["txid"], tid); assert_eq!(txs[0]["sigops"], 4); + assert_eq!(txs[0]["fee"], 1_000); let lp = tmpl["longpollid"].clone(); let again = dispatch(ctx, "getblocktemplate", vec![json!({"rules": ["segwit"]})]).unwrap(); assert_eq!(again["longpollid"], lp); @@ -786,6 +787,9 @@ fn chain_ops_sigop_adjusted_entry_and_min_fee(ctx: &RpcContext, cbs: &mut TrueCo .unwrap() .set_block_min_tx_fee_sat_kvb(min); crate::methods::mine::mempool_block_txs(ctx) + .into_iter() + .map(|(tx, _)| tx) + .collect::>() }; // 2_000 sat is 0.5 sat/vB at 4_000 vB. assert_eq!(keep(500), vec![tx.clone()]); diff --git a/docs/rpc.md b/docs/rpc.md index 58706c463..68bcfd8cc 100644 --- a/docs/rpc.md +++ b/docs/rpc.md @@ -117,8 +117,8 @@ still wait for durable SH when shindex is on. | `estimatesmartfee` | **10-minute inclusion frontier** — not Core historical multi-horizon. Core's result shape: `{feerate, blocks}`, or `{errors, blocks}` with no `feerate` when there is no estimate. Like Core, `feerate` is at least `mempoolminfee`. See [`mempool-fee-estimation.md`](./mempool-fee-estimation.md). | | `estimaterawfee` | Same 10-minute frontier product as `estimatesmartfee` (Core RPC name for harness scripts). Not Core historical `estimaterawfee` buckets. | | `getnetworkhashps` | Core `GetNetworkHashPS`: `chainwork(end) − chainwork(start)` over `(maxTime − minTime)` in the lookup window. Default `nblocks` 120; `nblocks<=0` uses `height % difficulty_adjustment_interval + 1` (capped to height). `height<0` or past tip → tip. Genesis / zero dt → `0.0`. | -| `generatetoaddress` / `generatetodescriptor` / `generateblock` / `generate` | **Regtest only.** Mine through `ChainHub::accept_block` (same confirm as P2P). First generated block includes `select_block_txs`, then `remove_for_block`. `generatetodescriptor` accepts `raw(HEX)`, `addr(ADDRESS)`, or a bare address. | -| `getblocktemplate` / `getmininginfo` | All networks. Template from `select_block_txs`; `-blockmintxfee` skips whole chunks under the floor (Core chunk feerate). `rules` must include `segwit`. Proposal validates without connecting and returns Core reject needles (`bad-cb-missing`, `bad-diffbits`, `time-too-old`, …). Version is `VERSIONBITS_TOP_BITS` only (no testdummy). `longpollid` waits until the tip or mempool update counter changes. `getmininginfo.blockmintxfee` is 8-decimal BTC/kvB (`sat_btc_json`, same helper as mempool fees). | +| `generatetoaddress` / `generatetodescriptor` / `generateblock` / `generate` | **Regtest only.** Mine through `ChainHub::accept_block` (same confirm as P2P). First generated block includes `select_block_template`, then `remove_for_block`. `generatetodescriptor` accepts `raw(HEX)`, `addr(ADDRESS)`, or a bare address. | +| `getblocktemplate` / `getmininginfo` | All networks. Template from `select_block_template`; `-blockmintxfee` skips whole chunks under the floor (Core chunk feerate). `rules` must include `segwit`. Proposal validates without connecting and returns Core reject needles (`bad-cb-missing`, `bad-diffbits`, `time-too-old`, …). Version is `VERSIONBITS_TOP_BITS` only (no testdummy). `longpollid` waits until the tip or mempool update counter changes. `getmininginfo.blockmintxfee` is 8-decimal BTC/kvB (`sat_btc_json`, same helper as mempool fees). | | `prioritisetransaction` / `getprioritisedtransactions` | All networks. Local mining fee delta (sat). Dummy must be 0. Selector honors modified fee. | | `getmempoolcluster` | All networks. Cluster weight / chunks from the live graph (modified fees). Same prefix-maximal chunks as mining selection. `clusterweight` and `chunkweight` are sigop-adjusted (Core); the cluster size limit itself counts raw weight (`COMPAT.md`). | | `getmempoolancestors` / `getmempooldescendants` | All networks. Exclusive walks of the live cluster graph. `verbose` reuses `getmempoolentry` fields. | diff --git a/docs/sv2-template-provider.md b/docs/sv2-template-provider.md index 58252ce1a..74bcf7330 100644 --- a/docs/sv2-template-provider.md +++ b/docs/sv2-template-provider.md @@ -165,19 +165,30 @@ No new flag; GBT output for an unchanged mempool is unchanged. ### A2 — Hub and GBT on the budgeted call - **Contract:** `MempoolHub::select_block_template(budget)` returns - `Vec<(Transaction, fee_sat, sigop_cost)>` from one read lock plus the - `prioritisetransaction` deltas. GBT `transactions[].fee` / `sigops` and - `coinbasevalue` come from it; `generate` uses the same call. A tx - removed after selection still reports its selected fee. -- **Red:** `cargo test -p rbitcoin-rpc gbt_` — existing GBT tests stay - green; new case: evict a selected tx between selection and JSON build - (hook or fixture), fee and sigops still match selection. -- **Green:** hub method; `mempool_block_txs` returns the triples; GBT - drops the per-tx `get_live_meta` / `get_live_sigop_cost` reads. -- **Refactor:** `select_block_txs` callers left over move to the new call - or keep a one-line wrapper. + `Vec<(Transaction, Selected)>` from one read lock plus the + `prioritisetransaction` deltas; `MempoolHub::template_budget(min)` is + the node's own budget (template weight, configured reserve). GBT + `transactions[].fee` / `sigops` and `coinbasevalue` come from it; + `generate` uses the same call. A tx removed after selection still + reports its selected fee. +- **Red:** `cargo test -p rbitcoin-net sigop` plus `hub_live_journey` — + extend the existing hub sigop tests to read fee and sigop cost through + `select_block_template` (configured reserve 0 fits a 79,920-cost tx; a + caller reserving 400 gets nothing), replacing `get_live_sigop_cost`. + A non-default budget is Plan B's path, so these stay hub units. + The evict-between-selection-and-JSON race is not reachable from a real + session without a hook; it is made structurally impossible (GBT no + longer re-reads after the lock drops) and not pinned by a hook test. + The `rpc_regtest_chain_ops` GBT beat gains `fee == 1_000` beside its + `sigops` asserts; the existing sigop-budget GBT test pins + `coinbasevalue` via block accept. +- **Green:** hub `template_budget` / `select_block_template`; + `mempool_block_txs` returns `(tx, Selected)`; GBT drops the per-tx + `get_live_meta` / `get_live_sigop_cost` reads. +- **Refactor:** `select_block_txs` (hub and `ActiveMempool`) and + `get_live_sigop_cost` go away; no callers left. - **Verify:** `cargo test -p rbitcoin-rpc --lib`, - `cargo test -p rbitcoin-net select_` + `cargo test -p rbitcoin-net --lib sigop` --- From 6cc3016712fb04cf80b235b951bc3a0f119fe094 Mon Sep 17 00:00:00 2001 From: Gary Krause Date: Mon, 28 Sep 2026 16:18:06 -0400 Subject: [PATCH 4/7] docs: sv2 roadmap session cap, waitNext out of scope Cap concurrent TP sessions (Electrum max_connections pattern) so the retained-template RAM trade has a real bound, with a B1 contract and red case. Record Core's IPC waitNext as out of scope in place of the old Q-64 backlog row; GBT longpoll already ships. B1 adds the CRATES.md row with the new crate. Co-Authored-By: Claude Opus 5.5 --- docs/sv2-template-provider.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/sv2-template-provider.md b/docs/sv2-template-provider.md index 74bcf7330..02b96b5df 100644 --- a/docs/sv2-template-provider.md +++ b/docs/sv2-template-provider.md @@ -79,9 +79,13 @@ three unused subprotocol crates) if it adds nothing. not-reactor. - Named RAM trade (CONTRIBUTING 9): each session retains, per live template, the full witness-serialized non-coinbase txs (≤ ~4 MB × ~3 - templates × sessions). Retention is required: the mempool may evict a tx + templates × capped sessions; ≤ ~96 MB at the default cap of 8). + Retention is required: the mempool may evict a tx before `RequestTransactionData` or `SubmitSolution` arrives. Stale grace (default 10 s) after a tip change, then drop (mirrors sv2-tp). +- Session cap (always on): the listener accepts at most 8 concurrent + sessions and closes the next one after accept, like Electrum's + `max_connections` semaphore. Per-IP metering stays out of scope. - Per-session budget: weight `MAX_BLOCK_WEIGHT − max(1168 + 4·coinbase_output_max_additional_size, 2000)` WU (sv2-spec 07 §7.1); sigops start at `coinbase_output_max_additional_sigops` (Core @@ -119,6 +123,11 @@ translator proxy, Job Declarator Client, weak-block targets below nBits, extension negotiation, per-IP metering/rate limits. None of these ship; nothing here precludes a later JD-server plan. +Core's IPC mining interface (`waitNext`) is also out: TDP push covers +these clients, and polling clients already have GBT longpoll and the +Esplora `/block-template` 15 s cache. This replaces the old Q-64 backlog +row ("GBT longpoll / `waitNext`, then Sv2"). + --- ## Plan A — Caller-budgeted template selection @@ -205,13 +214,17 @@ Ships the listener, bootstrap, tip push, transaction data, and max_version=2, flags=0}` receives `SetupConnection.Success{used_version=2, flags=0}`. Nonzero flags → `SetupConnection.Error` echoing them. `protocol != 2` or no version-2 overlap → Error and the connection - closes. + closes. With the session cap reached, the next connection is closed + before the handshake and the existing sessions stay up. - **Red:** `cargo test -p rbitcoin-sv2 setup_connection_` — loopback TCP, - in-crate test initiator; success, bad-flags, bad-protocol cases. + in-crate test initiator; success, bad-flags, bad-protocol, and + (cap + 1)th-connection cases. - **Green:** `crates/rbitcoin-sv2` (workspace member) with the wire crates pinned to the Step 0 set; authority-keypair config, listener task, per-connection session task driving the `codec_sv2` handshake then the - common-message branch. + common-message branch; session-cap semaphore on accept. Add the + `rbitcoin-sv2` row to [`CRATES.md`](./CRATES.md) in this commit + ([`README.md`](./README.md) rule: row with the new file). - **Refactor:** session state as an enum (`Handshake`, `AwaitingConstraints`, `Active`), not nested ifs. - **Verify:** `cargo test -p rbitcoin-sv2 setup_` From 0474653a949884c2d49eeca2a3732ea6650f6102 Mon Sep 17 00:00:00 2001 From: "rearden-grok[bot]" <317016512+rearden-grok[bot]@users.noreply.github.com> Date: Tue, 29 Sep 2026 10:26:08 -0700 Subject: [PATCH 5/7] mempool: allow a block sigop cost of exactly 80_000 Consensus rejects a cost above 80_000, so a running total of exactly 80_000 (the configured reserve included) is a valid template. Admission and selection both used `>=` and left that block out. --- changelog.d/sigop-cap-inclusive.md | 6 +++ crates/rbitcoin-mempool/src/accept.rs | 2 +- .../src/accept_life_journey.rs | 31 +++++++++++++--- crates/rbitcoin-mempool/src/graph.rs | 37 ++++++++++++------- docs/external_findings/048-standard-sigops.md | 6 +-- docs/operator/operations.md | 2 +- 6 files changed, 60 insertions(+), 24 deletions(-) create mode 100644 changelog.d/sigop-cap-inclusive.md diff --git a/changelog.d/sigop-cap-inclusive.md b/changelog.d/sigop-cap-inclusive.md new file mode 100644 index 000000000..32f32b6a7 --- /dev/null +++ b/changelog.d/sigop-cap-inclusive.md @@ -0,0 +1,6 @@ +Fixed + +- **Block sigop cap includes 80,000.** Admission and template selection + allow a running cost of exactly 80,000, with `--block-reserved-sigops` + counted in that total. A cost that would pass 80,000 is still + `bad-txns-too-many-sigops`. diff --git a/crates/rbitcoin-mempool/src/accept.rs b/crates/rbitcoin-mempool/src/accept.rs index 2ef70d58c..c16df9be9 100644 --- a/crates/rbitcoin-mempool/src/accept.rs +++ b/crates/rbitcoin-mempool/src/accept.rs @@ -346,7 +346,7 @@ fn block_fit_sigop_cost( .map(|o| o.script_pubkey.as_bytes()) .collect(); let cost = rbitcoin_consensus::tx_sigop_cost(tx, &spks, true, true); - if reserved_sigops.saturating_add(cost) >= MAX_BLOCK_SIGOPS_COST { + if reserved_sigops.saturating_add(cost) > MAX_BLOCK_SIGOPS_COST { return Err(AcceptError::TooManySigops { cost }); } Ok(cost) diff --git a/crates/rbitcoin-mempool/src/accept_life_journey.rs b/crates/rbitcoin-mempool/src/accept_life_journey.rs index ce7136681..a7bf79a14 100644 --- a/crates/rbitcoin-mempool/src/accept_life_journey.rs +++ b/crates/rbitcoin-mempool/src/accept_life_journey.rs @@ -637,15 +637,35 @@ fn sigop_block_budget(life: &mut Life) { life.mp .remove_for_block(&[fits_default.compute_txid()]) .unwrap(); + life.mp + .accept_tx(&exact_default_budget, &utxos, TIP_OK) + .expect("79,600 beside the default reserve is exactly 80,000"); + assert_eq!( + life.mp + .select_block_template(life.mp.template_budget(0), |_| 0) + .len(), + 1 + ); + life.mp + .remove_for_block(&[exact_default_budget.compute_txid()]) + .unwrap(); assert!(matches!( - life.mp.accept_tx(&exact_default_budget, &utxos, TIP_OK), - Err(AcceptError::TooManySigops { cost: 79_600 }) + life.mp + .accept_tx(&multisig_outputs_tx(op, 996), &utxos, TIP_OK), + Err(AcceptError::TooManySigops { cost: 79_680 }) )); life.mp.set_block_reserved_sigops(0); let fits_without_reserve = multisig_outputs_tx(op, 999); + let exact_block = multisig_outputs_tx(op, 1000); life.mp .accept_tx(&fits_without_reserve, &utxos, TIP_OK) .expect("79,920 fits when the template reserve is zero"); + life.mp + .remove_for_block(&[fits_without_reserve.compute_txid()]) + .unwrap(); + life.mp + .accept_tx(&exact_block, &utxos, TIP_OK) + .expect("80,000 fits when the template reserve is zero"); assert_eq!( life.mp .select_block_template(life.mp.template_budget(0), |_| 0) @@ -653,12 +673,11 @@ fn sigop_block_budget(life: &mut Life) { 1 ); life.mp - .remove_for_block(&[fits_without_reserve.compute_txid()]) + .remove_for_block(&[exact_block.compute_txid()]) .unwrap(); assert!(matches!( - life.mp - .accept_tx(&multisig_outputs_tx(op, 1000), &utxos, TIP_OK), - Err(AcceptError::TooManySigops { cost: 80_000 }) + life.mp.accept_tx(&multisig_outputs_tx(op, 1001), &utxos, TIP_OK), + Err(AcceptError::TooManySigops { cost: 80_080 }) )); assert_eq!(life.mp.live_count(), 0); restore_sigop_knobs(&mut life.mp); diff --git a/crates/rbitcoin-mempool/src/graph.rs b/crates/rbitcoin-mempool/src/graph.rs index 33271f2b2..cb22420e9 100644 --- a/crates/rbitcoin-mempool/src/graph.rs +++ b/crates/rbitcoin-mempool/src/graph.rs @@ -978,9 +978,10 @@ impl TxGraph { /// /// Empty pool or zero weight → `[]`. A high-feerate child chunk pulls in /// still-unselected in-mempool ancestors so the block is topological. - /// A chunk (plus those ancestors) that would overflow - /// `budget.max_weight_wu` or the block sigop limit (80_000 less - /// `budget.reserved_sigops`) is skipped; later chunks are still tried. + /// A chunk (plus those ancestors) that would exceed + /// `budget.max_weight_wu` or the block sigop limit is skipped; later + /// chunks are still tried. The limit is 80_000 including + /// `budget.reserved_sigops`; a total of exactly 80_000 fits. /// Chunks whose modified fee is **negative** are skipped. A chunk whose /// modified feerate is under `budget.min_sat_kvb` (`-blockmintxfee`) is /// skipped whole (Core `BlockAssembler` chunk floor), so a low-fee parent @@ -1039,9 +1040,10 @@ impl TxGraph { .fold((0u64, 0u64), |(w, s), e| { (w.saturating_add(e.weight), s.saturating_add(e.sigop_cost)) }); - // Core `TestChunkBlockLimits`: skip this chunk, keep trying smaller ones. + // Core `TestChunkBlockLimits`: skip this chunk, keep trying + // smaller ones. Consensus allows a cost of exactly 80_000. if used.saturating_add(extra_w) > max_weight_wu - || sigops.saturating_add(extra_sigops) >= MAX_BLOCK_SIGOPS_COST + || sigops.saturating_add(extra_sigops) > MAX_BLOCK_SIGOPS_COST { continue; } @@ -1476,9 +1478,10 @@ mod tests { ); } - /// Sigop budget starts at the caller's reserve; a chunk reaching 80_000 - /// is skipped (Core `>=`) and a later, cheaper chunk still fits. Each pick - /// carries the base fee and sigop cost it was budgeted with. + /// Sigop budget starts at the caller's reserve. A running cost of + /// exactly 80_000 fits; a chunk that would pass 80_000 is skipped and a + /// later, cheaper chunk still fits. Each pick carries the base fee and + /// sigop cost it was budgeted with. #[test] fn select_budgets_sigops_skip_and_continue() { let heavy = spend_op([8u8; 32], 50_000, 40_000); @@ -1504,9 +1507,17 @@ mod tests { select_ids(g, b, |_| 0) }; let full = |heavy_cost| at(&pool(heavy_cost), COINBASE_SIGOPS_RESERVE); - assert_eq!(full(79_600), vec![lid], "400 + 79_600 hits the limit"); - assert_eq!(full(79_599), vec![hid], "79_999 fits; +1 reaches 80_000"); - assert_eq!(full(79_598), vec![hid, lid], "80_000 - 1 total fits"); + assert_eq!(full(79_601), vec![lid], "400 + 79_601 passes 80_000"); + assert_eq!( + full(79_600), + vec![hid], + "400 + 79_600 equals 80_000 and fits" + ); + assert_eq!( + full(79_599), + vec![hid, lid], + "light's +1 lands on 80_000 and fits" + ); assert_eq!(full(u64::MAX), vec![lid], "unknown cost never selected"); let g = pool(79_598); @@ -1525,8 +1536,8 @@ mod tests { }, ] ); - assert_eq!(at(&g, 401), vec![hid], "a larger reserve drops the tail"); - assert_eq!(at(&g, 402), vec![lid], "skips heavy, still takes light"); + assert_eq!(at(&g, 402), vec![hid], "a larger reserve drops the tail"); + assert_eq!(at(&g, 403), vec![lid], "skips heavy, still takes light"); assert_eq!(at(&g, 0), vec![hid, lid]); } diff --git a/docs/external_findings/048-standard-sigops.md b/docs/external_findings/048-standard-sigops.md index deb39b400..973f36bd8 100644 --- a/docs/external_findings/048-standard-sigops.md +++ b/docs/external_findings/048-standard-sigops.md @@ -7,9 +7,9 @@ Mempool admission counted no standard sigop cap before the script interpreter, and an invalid script was logged without a ban score. -A transaction whose sigop cost does not fit the configured template budget -(80_000 minus `--block-reserved-sigops`, default 400) is rejected as -`bad-txns-too-many-sigops` before `verify_tx_scripts_detached`. This is a +A transaction whose sigop cost, plus `--block-reserved-sigops` (default +400), would pass 80_000 is rejected as `bad-txns-too-many-sigops` before +`verify_tx_scripts_detached`. A total of exactly 80_000 fits. This is a local admission/template policy, not a consensus-invalidity test. Core's 16_000 standard cap (`MAX_BLOCK_SIGOPS_COST / 5`) is not applied. The cost is up to ~5x more signature checks per rejected tx; the ban score below bounds diff --git a/docs/operator/operations.md b/docs/operator/operations.md index 1b377d5e4..5d1db6d15 100644 --- a/docs/operator/operations.md +++ b/docs/operator/operations.md @@ -126,7 +126,7 @@ Clean smoke: | `--block-version N` | `block_version=` | unset — generate/template version overlay | | `--block-min-tx-fee BTC` | `block_min_tx_fee=` | unset — template min tx fee; garbage/negatives fail start | | `--bytes-per-sigop N` | `bytes_per_sigop=` | 20 — policy size is `max(weight, sigops*N)/4` for feerate; `0` disables | -| `--block-reserved-sigops N` | `block_reserved_sigops=` | 400 — sigop budget held for coinbase/template overhead; admission and template selection use the same strict limit; range 0–80000 | +| `--block-reserved-sigops N` | `block_reserved_sigops=` | 400 — sigop budget held for coinbase/template overhead; admission and template selection allow a running cost of exactly 80_000 and reject one that would pass it; range 0–80000 | | `--alert-notify CMD` | `alert_notify=` | unset — `%s` = warning; fires once | | `--startup-notify CMD` | `startup_notify=` | unset | | `--test-activation-height name@HEIGHT` | `test_activation_height=` | empty — buried deployment overlay | From d59fa0516f77584cb03e33c72969bf0572130a75 Mon Sep 17 00:00:00 2001 From: "rearden-grok[bot]" <317016512+rearden-grok[bot]@users.noreply.github.com> Date: Tue, 29 Sep 2026 10:26:52 -0700 Subject: [PATCH 6/7] docs: describe Plan A as the selection that shipped The owner doc still named the removed template helpers as the current GBT path. GBT and generate already call MempoolHub::select_block_template. --- docs/sv2-template-provider.md | 95 +++++++++++------------------------ 1 file changed, 28 insertions(+), 67 deletions(-) diff --git a/docs/sv2-template-provider.md b/docs/sv2-template-provider.md index 02b96b5df..078607bd8 100644 --- a/docs/sv2-template-provider.md +++ b/docs/sv2-template-provider.md @@ -8,7 +8,7 @@ previous slice is committed. | Plan | Outcome | Ships flags | |------|---------|-------------| -| **A** | GBT builds from a caller-budgeted selection that returns fee and sigop cost with each tx | none | +| **A** | Landed. GBT and `generate` build from `MempoolHub::select_block_template` | none | | **B** | A Job Declarator Client mines a block through the node's TP | `--sv2-tp-listen`, `--sv2-tp-authority-sec`, `--sv2-tp-cert-validity`, `--sv2-tp-stale-grace` | | **C** | Templates refresh on fee gain with the tip unchanged | `--sv2-tp-fee-delta`, `--sv2-tp-template-interval` | @@ -132,72 +132,33 @@ row ("GBT longpoll / `waitNext`, then Sv2"). ## Plan A — Caller-budgeted template selection -**Goal:** `getblocktemplate` and regtest `generate` build from one -`MempoolHub` call that takes the weight and sigop budget and returns each -selected tx with the fee and sigop cost read under the same lock. Today -`select_block_txs(min_sat_kvb)` fixes the weight at -`template_tx_weight()`, the sigop reserve is graph-global -(`set_block_reserved_sigops` at open), and GBT re-reads `fee` / `sigops` -per tx via `get_live_meta` / `get_live_sigop_cost` after the read lock -drops. A tx evicted in between reports `fee: 0` / `sigops: 0` and -understates `coinbasevalue`. - -No new flag; GBT output for an unchanged mempool is unchanged. - -### A1 — Budget and per-tx meta through the graph - -- **Contract:** `TxGraph::select_block_template(budget, delta)` with - `SelectBudget { max_weight_wu, reserved_sigops, min_sat_kvb }` returns - `Vec` in mining order. At - `reserved_sigops = block_reserved_sigops()` and `max_weight_wu = - template_tx_weight()` it equals today's `select_block_txids_delta`. A - larger `reserved_sigops` skips a chunk that fit before and still takes a - later one; a smaller `max_weight_wu` likewise. `fee_sat` is the base fee - (not the delta-modified fee). -- **Red:** `cargo test -p rbitcoin-mempool select_` — the graph selector - is a pure unit (no session passes a non-default budget before Plan B). - Extend the existing `select_budgets_sigops_skip_and_continue` (per-tx - base fee and sigop cost; reserve 401 drops the light tail, 402 skips - the heavy chunk and still takes the light one) and pin base fee - under a delta in the existing delta unit; fold - `selection_uses_configured_sigop_reserve` into it. The weight edge is - the existing `select_skips_overweight_chunk_and_continues` on the - budget. No new `select_budget_` twin. -- **Green:** thread the budget through the existing selection loop; - `select_block_txids` / `select_block_txids_delta` go away. - `ActiveMempool::select_block_txs_delta` passes the admission reserve - until A2 moves its caller. -- **Refactor:** one selection loop; the graph-global reserve is admission - only, templates take the caller's. -- **Verify:** `cargo test -p rbitcoin-mempool select_` - -### A2 — Hub and GBT on the budgeted call - -- **Contract:** `MempoolHub::select_block_template(budget)` returns - `Vec<(Transaction, Selected)>` from one read lock plus the - `prioritisetransaction` deltas; `MempoolHub::template_budget(min)` is - the node's own budget (template weight, configured reserve). GBT - `transactions[].fee` / `sigops` and `coinbasevalue` come from it; - `generate` uses the same call. A tx removed after selection still - reports its selected fee. -- **Red:** `cargo test -p rbitcoin-net sigop` plus `hub_live_journey` — - extend the existing hub sigop tests to read fee and sigop cost through - `select_block_template` (configured reserve 0 fits a 79,920-cost tx; a - caller reserving 400 gets nothing), replacing `get_live_sigop_cost`. - A non-default budget is Plan B's path, so these stay hub units. - The evict-between-selection-and-JSON race is not reachable from a real - session without a hook; it is made structurally impossible (GBT no - longer re-reads after the lock drops) and not pinned by a hook test. - The `rpc_regtest_chain_ops` GBT beat gains `fee == 1_000` beside its - `sigops` asserts; the existing sigop-budget GBT test pins - `coinbasevalue` via block accept. -- **Green:** hub `template_budget` / `select_block_template`; - `mempool_block_txs` returns `(tx, Selected)`; GBT drops the per-tx - `get_live_meta` / `get_live_sigop_cost` reads. -- **Refactor:** `select_block_txs` (hub and `ActiveMempool`) and - `get_live_sigop_cost` go away; no callers left. -- **Verify:** `cargo test -p rbitcoin-rpc --lib`, - `cargo test -p rbitcoin-net --lib sigop` +**Landed.** `getblocktemplate` and regtest `generate` call +`MempoolHub::select_block_template(budget)`. `template_budget(min)` is +this node's budget: template weight, the configured sigop reserve, and +the `-blockmintxfee` floor. The call returns each selected transaction +with its base fee and admission sigop cost from that same read. +GBT `transactions[].fee` / `sigops` and `coinbasevalue` come from it, so +a transaction removed after selection still reports the fee it was +selected with. + +`TxGraph::select_block_template(budget, delta)` takes +`SelectBudget { max_weight_wu, reserved_sigops, min_sat_kvb }` and +returns `Vec` in mining order. +`fee_sat` is the base fee, not the `prioritisetransaction` delta. +`MempoolHub::select_block_template` applies the node's deltas under the +same lock and returns `Vec<(Transaction, Selected)>`. A larger +`reserved_sigops` or a smaller `max_weight_wu` drops what no longer fits +and still takes a later chunk. A running sigop cost of exactly 80_000 +fits; a chunk that would pass 80_000 is skipped. + +No new flag. + +`select_budgets_sigops_skip_and_continue` pins the budget, the base fee +under a delta, and the exact-80_000 edge. `mempool_accept_life` pins +admission against the same cap. `hub_live_journey` reads fee and sigop +cost through the hub call (reserve 0 fits a 79,920-cost tx; a caller +reserving 400 does not). `rpc_regtest_chain_ops` pins GBT `fee` beside +`sigops`. --- From 8a6b46f554f37c64429028bac28e4d0ab6f18324 Mon Sep 17 00:00:00 2001 From: "rearden-grok[bot]" <317016512+rearden-grok[bot]@users.noreply.github.com> Date: Tue, 29 Sep 2026 10:26:52 -0700 Subject: [PATCH 7/7] net: drop the restated template-selection comment The asserts already pin the reserve and the selected fee and sigop cost. --- crates/rbitcoin-net/src/tx_relay.rs | 2 -- 1 file changed, 2 deletions(-) diff --git a/crates/rbitcoin-net/src/tx_relay.rs b/crates/rbitcoin-net/src/tx_relay.rs index 0dc1b9651..ef6755403 100644 --- a/crates/rbitcoin-net/src/tx_relay.rs +++ b/crates/rbitcoin-net/src/tx_relay.rs @@ -4744,8 +4744,6 @@ mod tests { Some(0), ) .unwrap(); - // GBT's budget carries the configured reserve; each pick carries the - // fee and sigop cost read under the selection's lock. let budget = hub.template_budget(0); assert_eq!(budget.reserved_sigops, 0); let picked = hub.select_block_template(budget);