diff --git a/.agents/skills/ship-pr/SKILL.md b/.agents/skills/ship-pr/SKILL.md index c1dd0860f..f92c00620 100644 --- a/.agents/skills/ship-pr/SKILL.md +++ b/.agents/skills/ship-pr/SKILL.md @@ -84,6 +84,9 @@ version-bump PR. Do not label ordinary net or RPC PRs. Label `overlay-functional` when the PR touches that harness, and on every ship version-bump PR. Poll with `--interest overlay-functional`. It is not a required check. +Label `warnet` to run the two-tank Docker example +([`docs/core-functional.md`](../../../docs/core-functional.md)). Not a +required check. Do not label ordinary PRs. Label `nixos-module-runtime` when the NixOS module VM test should run (not eval). Poll with `--interest nixos-module-runtime`. It is not a required check. diff --git a/.github/workflows/warnet-example.yml b/.github/workflows/warnet-example.yml new file mode 100644 index 000000000..6b981281f --- /dev/null +++ b/.github/workflows/warnet-example.yml @@ -0,0 +1,43 @@ +# Two-tank Warnet lab image. Label `warnet` or workflow_dispatch. +# Not a required check. Not kind/Helm miner_std. +name: warnet-example + +on: + workflow_dispatch: + pull_request: + types: [opened, synchronize, reopened, labeled, unlabeled] + +concurrency: + group: warnet-example-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + warnet-example: + name: warnet-example + runs-on: ubuntu-latest + timeout-minutes: 45 + if: >- + github.event_name == 'workflow_dispatch' || + contains(github.event.pull_request.labels.*.name, 'warnet') + env: + CARGO_TARGET_DIR: target/dev + RUSTFLAGS: -D warnings + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + submodules: false + - name: Build a bookworm node and run two tanks + run: | + set -euo pipefail + docker run --rm -v "$PWD":/src -w /src \ + -e CARGO_TARGET_DIR=/src/target/docker \ + rust:1.95.0-bookworm \ + cargo build -p rbitcoin-node + ./scripts/core-functional/init-submodule.sh + ./scripts/core-functional/warnet/Dockerfile.test.sh + NODE_BIN="$PWD/target/docker/debug/rbitcoin-node" \ + ./scripts/core-functional/warnet/example.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index a5c15a7af..a5e8f62f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,6 +46,18 @@ before 1.0). ### Changed +- **Hostname `--connect` / `addnode`:** clearnet names resolve at each dial + (`localhost` and a missing port use the network P2P default) on a blocking + thread, and retry until a live session exists. `--connect` does not enable + seed redial. If the first catch-up accepts nothing and the tip is still 0, + the node enters tip-follow (16 blocks in flight, no return to the IBD + window) so a late short-chain peer can attach. A non-zero tip stays in IBD. + Relay stays gated below `--min-chain-work`. + Label `warnet` runs a two-tank Docker example that starts the connecting + tank first and waits until its height matches the miner and `getpeerinfo` + shows the resolved address + ([`docs/core-functional.md`](docs/core-functional.md)). + - **Tor cookie HMAC uses `hmac` 0.13 and `sha2` 0.11** (digest 0.11). Node `getrandom` is 0.4, matching the rest of the workspace. `bitcoin` requirement is 0.32.102. Compatible lock bumps include `bitflags` 2.13.2, diff --git a/COMPAT.md b/COMPAT.md index 014372b5e..9c78764af 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -113,7 +113,7 @@ Per-method notes, auth, and the shindex matrix live in |--------------|--------| | Control (`help`, `uptime`, `stop`, `getrpcinfo`, `echo`) | done (`syncwithvalidationinterfacequeue` omitted; functional proxy no-op for Core `sync_mempools`) | | Blockchain (`getblockchaininfo`, `getblockcount`, `getbestblockhash`, `getblockhash`, `getblock`/`header`, `getdifficulty`, `getblockstats`) | done (`getblockstats` from `txstat` when stamped; size and count fields match Core, including `utxo_increase_actual`; omit coins-DB `utxo_size_*`) | -| Network (`getnetworkinfo`, `getconnectioncount`, `getpeerinfo`, `addnode`, `disconnectnode`, `addconnection`) | done (BIP324 v2-only; peer `timeoffset` / `synced_*` from session state) | +| Network (`getnetworkinfo`, `getconnectioncount`, `getpeerinfo`, `addnode`, `disconnectnode`, `addconnection`) | done (BIP324 v2-only; peer `timeoffset` / `synced_*` from session state; hostname `addnode` / `--connect` resolve at dial and retry until live) | | Mempool / rawtx (`getmempool*`, `getrawtransaction`, `sendrawtransaction`, `testmempoolaccept`) | done (Libre; RPC `maxfeerate` / `maxburnamount` / `"version"` only) | | Coin / MiniWallet (`gettxout`, `scantxoutset` `raw(HEX)`) | done (Class A unspent walk — not a coins-DB) | | Index / tips (`getindexinfo`, `getchaintips`, `waitforblock*`) | done (`txindex` = Class A reconstruct) | diff --git a/OPERATOR.md b/OPERATOR.md index 78fd3d1cc..a756e1c6a 100644 --- a/OPERATOR.md +++ b/OPERATOR.md @@ -853,6 +853,16 @@ Do **not** wipe `store/` for mempool slot/full/schema errors. groups; the node still starts. `--connect` is operator-pinned and skips the filter. We do not ship a mainnet map. Core publishes maps from the same `ip_asn.dat` used by `bitcoind -asmap`. +- **Genesis `--connect` after a failed first catch-up.** If that attempt + accepts no blocks and the tip is still height 0, the process enters + tip-follow instead of staying in IBD. The peer that appears later is a + follow session: `getheaders`, then at most 16 blocks in flight, confirmed + on the tip index path. The stagnant-tip loop does not start the IBD + scheduler again. A catch-up that finishes is unchanged, and a non-zero tip + that has not finished catch-up stays in IBD. This is for a peer you expect + to show up with a short chain. A fresh mainnet or signet datadir pointed at + a full node stays on this slower path for the rest of the process whenever + the first dial accepts nothing. - Tx inv/getdata/tx relay is **off during IBD**; enabled in tip mode after catch-up. - **BIP152 compact blocks v2:** `sendcmpct` high-bandwidth; mempool/orphan/`extra_compact` short-id fill + `getblocktxn` / `blocktxn`; full witness getdata fallback. We also **serve** `getblocktxn`. diff --git a/crates/rbitcoin-net/src/lib.rs b/crates/rbitcoin-net/src/lib.rs index ae519b86f..09177ba0b 100644 --- a/crates/rbitcoin-net/src/lib.rs +++ b/crates/rbitcoin-net/src/lib.rs @@ -56,8 +56,8 @@ pub use peer::{ }; pub use peer_dos::DEFAULT_MAX_INBOUND; pub use peers::{ - parse_peer_addr, parse_peer_net, pick_stale_follow_evict, DialRequest, DialTarget, LivePeer, - PeerConnType, PeerHub, PeerInfo, PeerOut, PingAction, + parse_peer_addr, parse_peer_addr_with_port, parse_peer_net, pick_stale_follow_evict, + DialRequest, DialTarget, LivePeer, PeerConnType, PeerHub, PeerInfo, PeerOut, PingAction, }; pub use rbitcoin_mempool::AcceptError; pub(crate) use rbitcoin_mempool::MempoolGraphStats; diff --git a/crates/rbitcoin-net/src/peers.rs b/crates/rbitcoin-net/src/peers.rs index cdb51dbbb..f167b654e 100644 --- a/crates/rbitcoin-net/src/peers.rs +++ b/crates/rbitcoin-net/src/peers.rs @@ -8,7 +8,7 @@ use bitcoin::p2p::ServiceFlags; use bitcoin::{BlockHash, Wtxid}; use std::collections::{HashMap, HashSet, VecDeque}; use std::hash::Hash; -use std::net::{IpAddr, SocketAddr}; +use std::net::{IpAddr, SocketAddr, ToSocketAddrs}; use std::sync::atomic::{AtomicBool, AtomicU16, AtomicU32, AtomicU64, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex, RwLock, Weak}; use tokio::sync::mpsc; @@ -1123,6 +1123,13 @@ pub struct PeerHub { next_id: AtomicU64, live: RwLock>>, added: Mutex>, + /// Raw `addnode add` strings. Re-resolved on each redial so a Warnet name + /// that is not in DNS yet is kept. + manual_hosts: Mutex>, + /// `--connect` hostnames that are not a [`crate::NetAddr`] (clearnet DNS). + connect_hosts: Mutex>, + /// Network P2P port used when a remembered host omits `:port`. `0` = unset. + connect_default_port: AtomicU16, dial_tx: Mutex>>, /// Peers we asked to send us compact (BIP152 HB, max 3, prefer outbound). hb_selected: Mutex>, @@ -1230,6 +1237,9 @@ impl PeerHub { next_id: AtomicU64::new(0), live: RwLock::new(HashMap::new()), added: Mutex::new(HashSet::new()), + manual_hosts: Mutex::new(HashSet::new()), + connect_hosts: Mutex::new(Vec::new()), + connect_default_port: AtomicU16::new(0), dial_tx: Mutex::new(None), hb_selected: Mutex::new(Vec::new()), mock_now: AtomicU64::new(0), @@ -2119,6 +2129,172 @@ impl PeerHub { } } + /// `addnode` with the operator string. IP, onion, I2P, and CJDNS parse as + /// [`crate::NetAddr`]. Anything else is clearnet DNS, resolved at dial. + /// `add` keeps the string when DNS fails so a later redial can succeed. + /// `onetry` fails if the name does not resolve now. + pub fn addnode_host(&self, node: &str, cmd: &str, default_port: u16) -> Result<(), String> { + self.note_default_port(default_port); + match cmd { + "onetry" => self.dial_resolved(node, PeerConnType::Manual), + "add" => { + self.manual_hosts + .lock() + .unwrap_or_else(|e| e.into_inner()) + .insert(node.to_string()); + let _ = self.dial_resolved(node, PeerConnType::Manual); + Ok(()) + } + "remove" => { + self.manual_hosts + .lock() + .unwrap_or_else(|e| e.into_inner()) + .remove(node); + if let Ok(target) = self.host_dial_target(node) { + self.disconnect_target(&target); + } + Ok(()) + } + other => Err(format!("unknown addnode command {other}")), + } + } + + pub fn set_connect_hosts(&self, hosts: Vec, default_port: u16) { + self.note_default_port(default_port); + *self.connect_hosts.lock().unwrap_or_else(|e| e.into_inner()) = hosts; + } + + fn note_default_port(&self, default_port: u16) { + if default_port != 0 && self.connect_default_port.load(Ordering::Relaxed) == 0 { + self.connect_default_port + .store(default_port, Ordering::Relaxed); + } + } + + fn default_port_opt(&self) -> Option { + let port = self.connect_default_port.load(Ordering::Relaxed); + (port != 0).then_some(port) + } + + fn host_dial_target(&self, node: &str) -> Result { + let with_port = ensure_host_port(node, self.default_port_opt())?; + if let Ok(net) = parse_peer_net(&with_port) { + return Ok(DialTarget::from_net(net)); + } + let addr = + parse_peer_addr_with_port(node, self.default_port_opt()).map_err(|e| e.to_string())?; + Ok(DialTarget::Socket(addr)) + } + + fn dial_resolved(&self, node: &str, typ: PeerConnType) -> Result<(), String> { + let target = self.host_dial_target(node)?; + match &target { + DialTarget::Socket(addr) => { + self.added + .lock() + .unwrap_or_else(|e| e.into_inner()) + .insert(crate::NetAddr::from_socket(*addr)); + } + DialTarget::Domain { .. } => { + self.added + .lock() + .unwrap_or_else(|e| e.into_inner()) + .insert(target.net_addr()); + } + } + self.dial_target(target, typ) + } + + fn dial_target(&self, target: DialTarget, typ: PeerConnType) -> Result<(), String> { + match target { + DialTarget::Socket(addr) => self.dial(addr, typ), + DialTarget::Domain { host, port } => self.dial_domain(host, port, typ), + } + } + + fn disconnect_target(&self, target: &DialTarget) -> bool { + match target { + DialTarget::Socket(addr) => { + self.added + .lock() + .unwrap_or_else(|e| e.into_inner()) + .remove(&crate::NetAddr::from_socket(*addr)); + self.disconnect_addr(*addr) + } + DialTarget::Domain { .. } => { + let net = target.net_addr(); + self.added + .lock() + .unwrap_or_else(|e| e.into_inner()) + .remove(&net); + self.disconnect_net(net) + } + } + } + + fn is_target_live(&self, target: &DialTarget) -> bool { + let peers = self.snapshot(); + match target { + DialTarget::Socket(addr) => peers.iter().any(|p| p.addr == *addr), + DialTarget::Domain { host, port } => peers + .iter() + .any(|p| p.net.host_str() == *host && p.net.port() == *port), + } + } + + fn remembered_redials(&self) -> Vec<(String, PeerConnType)> { + let mut out = Vec::new(); + for host in self + .manual_hosts + .lock() + .unwrap_or_else(|e| e.into_inner()) + .iter() + { + out.push((host.clone(), PeerConnType::Manual)); + } + for host in self + .connect_hosts + .lock() + .unwrap_or_else(|e| e.into_inner()) + .iter() + { + out.push((host.clone(), PeerConnType::OutboundFullRelay)); + } + out + } + + /// One dial per resolved endpoint in this pass. `addnode add` and + /// `--connect` of the same host share that dial (Manual wins). A later + /// pass dials again when the session is still not live. DNS lookup is + /// synchronous; callers on a Tokio worker use + /// [`Self::redial_remembered_off_runtime`]. + pub fn redial_remembered_with(&self, resolve: impl Fn(&str) -> Result) { + let mut seen = HashSet::::new(); + for (host, typ) in self.remembered_redials() { + let Ok(target) = resolve(&host) else { + continue; + }; + if !seen.insert(target.to_string()) { + continue; + } + if self.is_target_live(&target) { + continue; + } + let _ = self.dial_target(target, typ); + } + } + + pub fn redial_remembered(&self) { + self.redial_remembered_with(|node| self.host_dial_target(node)); + } + + /// `ToSocketAddrs` on the blocking pool so a slow resolver cannot stall + /// the Tokio worker that owns the retry interval. + pub async fn redial_remembered_off_runtime(self: &Arc) { + let peers = Arc::clone(self); + let _ = tokio::task::spawn_blocking(move || peers.redial_remembered()).await; + } + /// Select `id` as a BIP152 high-bandwidth peer (we send them sendcmpct(1)). /// Evicts the oldest inbound if we already have 3; never evict the last outbound /// when adding an inbound. @@ -2391,12 +2567,57 @@ pub fn parse_peer_net(s: &str) -> Result { .map_err(|_| NetError::Encode(format!("bad peer address {s}"))) } +fn ensure_host_port(node: &str, default_port: Option) -> Result { + if node.parse::().is_ok() { + return Ok(node.to_string()); + } + if let Some((host, port_s)) = node.rsplit_once(':') { + if !host.is_empty() && !host.starts_with('[') && port_s.parse::().is_ok() { + return Ok(node.to_string()); + } + } + let port = default_port.ok_or_else(|| format!("bad peer address {node}"))?; + if node.is_empty() || node.contains(char::is_whitespace) { + return Err(format!("bad peer address {node}")); + } + Ok(format!("{node}:{port}")) +} + +/// Parse `ip:port`, `[v6]:port`, `host:port`, or `host` (uses `default_port`). +/// +/// Hostnames resolve at call time (`ToSocketAddrs`). Dual-stack names prefer +/// IPv4 so `localhost` reaches a `127.0.0.1` listener. +pub fn parse_peer_addr_with_port( + s: &str, + default_port: Option, +) -> Result { + let bad = || NetError::Encode(format!("bad peer address {s}")); + if let Ok(addr) = s.parse::() { + return Ok(addr); + } + if let Ok(ip) = s.parse::() { + let port = default_port.ok_or_else(bad)?; + return Ok(SocketAddr::new(ip, port)); + } + let with_port = ensure_host_port(s, default_port).map_err(|_| bad())?; + let addrs: Vec = with_port.to_socket_addrs().map_err(|_| bad())?.collect(); + addrs + .iter() + .copied() + .find(|a| a.is_ipv4()) + .or_else(|| addrs.first().copied()) + .ok_or_else(bad) +} + #[cfg(test)] mod tests { use super::*; use bitcoin::hashes::Hash; use bitcoin::p2p::address::Address; use std::net::{IpAddr, Ipv4Addr}; + use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::Arc; + use tokio::sync::mpsc; fn ver(ua: &str) -> VersionMessage { VersionMessage { @@ -3127,6 +3348,91 @@ mod tests { assert!(hub.addnode(a, "nope").is_err()); } + fn take_dials(rx: &mut mpsc::UnboundedReceiver) -> Vec { + let mut out = Vec::new(); + while let Ok(req) = rx.try_recv() { + out.push(req); + } + out + } + + #[test] + fn parse_peer_addr_localhost_and_default_port() { + let with_port = parse_peer_addr_with_port("localhost:18444", None).expect("localhost:port"); + assert!(with_port.is_ipv4()); + assert_eq!(with_port.port(), 18444); + let bare = parse_peer_addr_with_port("localhost", Some(18444)).expect("localhost default"); + assert!(bare.is_ipv4()); + assert_eq!(bare.port(), 18444); + let lit = "127.0.0.1:18444".parse::().unwrap(); + assert_eq!( + parse_peer_addr_with_port("127.0.0.1:18444", None).unwrap(), + lit + ); + assert!(parse_peer_addr_with_port("not-a-real-host.invalid", Some(18444)).is_err()); + } + + #[test] + fn addnode_add_keeps_unresolved_host_for_redial() { + let hub = PeerHub::new(); + let (tx, mut rx) = mpsc::unbounded_channel(); + hub.set_dialer(tx); + hub.addnode_host("not-a-real-host.invalid", "add", 18444) + .expect("unresolved add is remembered"); + assert!(take_dials(&mut rx).is_empty()); + let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 9); + hub.redial_remembered_with(|_| Ok(DialTarget::Socket(addr))); + let got = take_dials(&mut rx); + assert_eq!(got.len(), 1, "later resolve must dial: {got:?}"); + assert!(matches!(got[0].target, DialTarget::Socket(a) if a == addr)); + assert!(hub + .addnode_host("not-a-real-host.invalid", "onetry", 18444) + .is_err()); + } + + #[test] + fn redial_same_endpoint_in_addnode_and_connect_dials_once() { + let hub = PeerHub::new(); + let (tx, mut rx) = mpsc::unbounded_channel(); + hub.set_dialer(tx); + hub.addnode_host("127.0.0.1:18444", "add", 18444).unwrap(); + assert_eq!(take_dials(&mut rx).len(), 1); + hub.set_connect_hosts(vec!["127.0.0.1:18444".into()], 18444); + hub.redial_remembered(); + let got = take_dials(&mut rx); + assert_eq!( + got.len(), + 1, + "addnode and --connect of one endpoint share one dial per pass: {got:?}" + ); + assert!(matches!(got[0].typ, PeerConnType::Manual)); + } + + #[tokio::test] + async fn slow_redial_resolve_does_not_stall_runtime() { + let hub = PeerHub::new(); + hub.set_connect_hosts(vec!["slow.example".into()], 18444); + let flag = Arc::new(AtomicBool::new(false)); + let flag2 = Arc::clone(&flag); + let progress = tokio::spawn(async move { + tokio::time::sleep(std::time::Duration::from_millis(40)).await; + flag2.store(true, Ordering::SeqCst); + }); + let hub2 = Arc::clone(&hub); + let slow = tokio::task::spawn_blocking(move || { + hub2.redial_remembered_with(|_| { + std::thread::sleep(std::time::Duration::from_millis(150)); + Err("slow".into()) + }); + }); + progress.await.unwrap(); + assert!( + flag.load(Ordering::SeqCst), + "a blocking resolve must not stall other Tokio tasks" + ); + slow.await.unwrap(); + } + #[test] fn ping_pong_logs_core_needles() { let hub = PeerHub::new(); diff --git a/crates/rbitcoin-net/src/service.rs b/crates/rbitcoin-net/src/service.rs index cb0a7994a..3456031a3 100644 --- a/crates/rbitcoin-net/src/service.rs +++ b/crates/rbitcoin-net/src/service.rs @@ -216,6 +216,24 @@ impl P2PNode { }); tasks.push(dial_task); + // DNS for remembered `--connect` / `addnode add` runs on the blocking + // pool (`PeerHub::redial_remembered_off_runtime`). + let retry_peers = peers.clone(); + let retry_shutdown = shutdown.clone(); + let retry_task = tokio::spawn(async move { + let mut interval = tokio::time::interval(Duration::from_secs(2)); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + interval.tick().await; + loop { + interval.tick().await; + if retry_shutdown.load(Ordering::SeqCst) { + break; + } + retry_peers.redial_remembered_off_runtime().await; + } + }); + tasks.push(retry_task); + Ok(Self { cache, query, @@ -422,6 +440,22 @@ impl P2PNode { } } +impl Drop for P2PNode { + fn drop(&mut self) { + // The connect-retry interval runs until this flag or an abort. + // Dropping the handle without `shutdown` must not pin the test runtime. + self.shutdown.store(true, Ordering::SeqCst); + for t in &self.tasks { + t.abort(); + } + if let Ok(g) = self.session_tasks.lock() { + for t in g.iter() { + t.abort(); + } + } + } +} + fn push_session_task(bag: &Mutex>>, h: JoinHandle<()>) { if let Ok(mut g) = bag.lock() { g.retain(|t| !t.is_finished()); diff --git a/crates/rbitcoin-node/src/cli.rs b/crates/rbitcoin-node/src/cli.rs index 9939b061e..7040d9125 100644 --- a/crates/rbitcoin-node/src/cli.rs +++ b/crates/rbitcoin-node/src/cli.rs @@ -139,6 +139,7 @@ fn apply_operator_kvs(config: &mut NodeConfig, kvs: Vec<(String, String)>) -> Re } if key == "connect" && !saw_connect { config.listen.connect.clear(); + config.listen.connect_dns.clear(); saw_connect = true; } if key == "seed_node" && !saw_seednode { @@ -960,6 +961,14 @@ mod tests { )); } + #[test] + fn connect_bare_label_stays_config() { + let n = ready_config(["rbitcoin-node", "--connect", "bad"]); + assert_eq!(n.listen.connect_dns, vec!["bad".to_string()]); + assert!(n.listen.connect.is_empty()); + assert!(operator_config_from_args(["rbitcoin-node", "--connect", "bad host"]).is_err()); + } + #[test] fn only_net_onion_without_proxy_is_config_error() { let mut c = NodeConfig::default(); diff --git a/crates/rbitcoin-node/src/config.rs b/crates/rbitcoin-node/src/config.rs index 86080de44..fefcbfbd3 100644 --- a/crates/rbitcoin-node/src/config.rs +++ b/crates/rbitcoin-node/src/config.rs @@ -86,6 +86,8 @@ pub struct ListenOpts { pub electrum: Option, pub esplora: Option, pub connect: Vec, + /// `--connect` names that are not a `NetAddr` (clearnet DNS, Warnet tanks). + pub connect_dns: Vec, pub seednodes: Vec, pub use_seeds: bool, pub max_outbound: u32, @@ -121,6 +123,7 @@ impl Default for ListenOpts { electrum: None, esplora: None, connect: Vec::new(), + connect_dns: Vec::new(), seednodes: Vec::new(), use_seeds: true, max_outbound: 16, @@ -142,6 +145,10 @@ impl Default for ListenOpts { } impl ListenOpts { + pub fn has_pinned_connect(&self) -> bool { + !self.connect.is_empty() || !self.connect_dns.is_empty() + } + pub fn dialer(&self) -> rbitcoin_net::Dialer { rbitcoin_net::Dialer::with_proxies(self.proxy, self.onion, self.proxy_randomize) } @@ -814,10 +821,18 @@ impl NodeConfig { ); } "connect" => { - self.listen.connect.push( - val.parse() - .map_err(|e| NodeError::Config(format!("conf connect: {e}")))?, - ); + push_connect(&mut self.listen, val)?; + } + "head_scale" => { + self.head_scale = match val { + "tiny" => HeadScale::Tiny, + "mainnet" => HeadScale::Mainnet, + other => { + return Err(NodeError::Config(format!( + "conf head_scale must be tiny or mainnet, got {other}" + ))) + } + }; } "proxy" => { self.listen.proxy = Some(parse_required_socket(val, "proxy")?); @@ -1264,6 +1279,35 @@ pub fn parse_minimum_chain_work(spec: &str) -> Result<[u8; 32], String> { Ok(out) } +fn push_connect(listen: &mut ListenOpts, val: &str) -> Result<(), NodeError> { + let val = val.trim(); + if val.is_empty() { + return Err(NodeError::Config( + "conf connect requires host[:port]".into(), + )); + } + if let Ok(addr) = val.parse::() { + listen.connect.push(addr); + return Ok(()); + } + let host = match val.rsplit_once(':') { + Some((host, port)) if port.parse::().is_ok() => host, + _ => val, + }; + let lower = host.to_ascii_lowercase(); + if lower.ends_with(".onion") + || lower.ends_with(".b32.i2p") + || val.contains(char::is_whitespace) + || val.contains('/') + { + return Err(NodeError::Config(format!( + "conf connect: bad peer address {val}" + ))); + } + listen.connect_dns.push(val.to_string()); + Ok(()) +} + impl NodeConfig { /// True when tip work meets `--min-chain-work` (or the flag is unset). pub fn meets_minimum_chain_work(&self, tip_work_be: [u8; 32]) -> bool { @@ -1673,6 +1717,7 @@ mod tests { Some(std::path::Path::new("/tmp/ip_asn.dat")) ); assert_eq!(cfg.listen.connect.len(), 1); + assert!(cfg.listen.connect_dns.is_empty()); assert_eq!( cfg.datadir.cold.as_deref(), Some(std::path::Path::new("/mnt/hdd/rbtc-cold")) @@ -2128,4 +2173,30 @@ mod tests { above[31] = 0x66; assert!(cfg.meets_minimum_chain_work(above)); } + + #[test] + fn connect_hostname_is_dns_not_netaddr() { + let mut cfg = NodeConfig::default(); + cfg.apply_kv("head_scale", "tiny").unwrap(); + assert_eq!(cfg.head_scale, HeadScale::Tiny); + cfg.apply_kv("connect", "tank-0001:18444").unwrap(); + cfg.apply_kv("connect", "bad").unwrap(); + cfg.apply_kv("connect", "tank-0").unwrap(); + cfg.apply_kv("connect", "127.0.0.1:18444").unwrap(); + assert!(cfg + .listen + .connect_dns + .iter() + .any(|h| h == "tank-0001:18444")); + assert!(cfg.listen.connect_dns.iter().any(|h| h == "bad")); + assert!(cfg.listen.connect_dns.iter().any(|h| h == "tank-0")); + assert_eq!(cfg.listen.connect.len(), 1); + assert!(cfg.apply_kv("connect", "").is_err()); + assert!(cfg + .apply_kv("connect", "short.onion:8333") + .unwrap_err() + .to_string() + .contains("bad")); + assert!(cfg.listen.has_pinned_connect()); + } } diff --git a/crates/rbitcoin-node/src/run.rs b/crates/rbitcoin-node/src/run.rs index 5b592d1c8..f61f52bb7 100644 --- a/crates/rbitcoin-node/src/run.rs +++ b/crates/rbitcoin-node/src/run.rs @@ -483,9 +483,17 @@ pub async fn run_p2p(config: NodeConfig) -> Result<(), NodeError> { .set_cjdns_reachable(config.listen.cjdns_reachable); node.peers.set_pruned(config.prune_seqsigwit); node.peers.set_asmap(asmap); + node.peers.set_connect_hosts( + config.listen.connect_dns.clone(), + config.network.default_p2p_port(), + ); + let dns_resolved = resolve_connect_dns(&config.listen.connect_dns, config.network).await; for c in &config.listen.connect { addrman.add_addr(*c); } + for c in &dns_resolved { + addrman.add_addr(*c); + } if should_resolve_default_seeds(&config) { info!( "ibd: resolving DNS/fixed seeds for {}…", @@ -502,7 +510,7 @@ pub async fn run_p2p(config: NodeConfig) -> Result<(), NodeError> { let n = queue_proxy_seed_addrfetch(&node.peers, config.network); info!("ibd: SOCKS proxy set — queued {n} seed hostnames via SOCKS addrfetch"); } else if config.signet_challenge.is_some() - && config.listen.connect.is_empty() + && !config.listen.has_pinned_connect() && addrman.is_empty() { warn!("custom signet has no peers; use --connect ADDR or reuse a datadir with known peers"); @@ -522,8 +530,10 @@ pub async fn run_p2p(config: NodeConfig) -> Result<(), NodeError> { let max_out = config.listen.max_outbound.max(1) as usize; let candidate_n = max_out.saturating_mul(2).clamp(16, 48); let occupied = node.peers.live_outbound_full_relay_nets(); - let targets = follow_dial_targets(&config.listen.connect, &addrman, max_out, &occupied); - let ibd_targets = follow_dial_targets(&config.listen.connect, &addrman, candidate_n, &occupied); + let mut pinned = config.listen.connect.clone(); + pinned.extend(dns_resolved); + let targets = follow_dial_targets(&pinned, &addrman, max_out, &occupied); + let ibd_targets = follow_dial_targets(&pinned, &addrman, candidate_n, &occupied); let catch_up = run_ibd_or_skip( &node, &ibd_targets, @@ -535,6 +545,13 @@ pub async fn run_p2p(config: NodeConfig) -> Result<(), NodeError> { ) .await; + let catch_up = catch_up_with_connect( + catch_up, + config.listen.has_pinned_connect(), + shutdown.requested(), + node.tip_height().unwrap_or(0), + ); + // Still enter tip-follow when work is below `--min-chain-work` so later // blocks can raise the tip. Relay / getheaders stay gated on the hub floor. if catch_up.is_complete() && !tip_meets_min_work(&config, &node.hub) { @@ -560,10 +577,11 @@ pub async fn run_p2p(config: NodeConfig) -> Result<(), NodeError> { Arc::clone(&shutdown.flag), ); } - if !config.mempool.blocksonly - && tip_meets_min_work(&config, &node.hub) - && !node.hub.in_ibd() - { + if relay_while_following( + tip_meets_min_work(&config, &node.hub), + config.mempool.blocksonly, + node.hub.in_ibd(), + ) { mempool_blocking(&mempool, |mp| mp.set_relay_enabled(true)).await?; } let mp_live = mempool_blocking(&mempool, MempoolHub::live_count).await?; @@ -991,7 +1009,7 @@ pub async fn run_p2p(config: NodeConfig) -> Result<(), NodeError> { } let stagnant = last_tip_change.elapsed() >= Duration::from_secs(STALE_TIP_SECS); - if !stagnant || !config.listen.connect.is_empty() || !config.listen.use_seeds { + if !stagnant || config.listen.has_pinned_connect() || !config.listen.use_seeds { continue; } if addrman.is_empty() || shutdown.requested() { @@ -1156,11 +1174,53 @@ fn tip_meets_min_work(config: &NodeConfig, hub: &rbitcoin_net::ChainHub) -> bool fn should_resolve_default_seeds(config: &NodeConfig) -> bool { config.listen.use_seeds - && config.listen.connect.is_empty() + && !config.listen.has_pinned_connect() && config.signet_challenge.is_none() && config.listen.proxy.is_none() } +/// Genesis `--connect` whose first catch-up accepts nothing is treated as +/// finished so the process stays up. The later dial is a follow session +/// (16 blocks in flight on the tip index path), not a return to the IBD +/// window. A non-zero tip that has not finished catch-up stays in IBD. +pub(crate) fn catch_up_with_connect( + catch_up: CatchUp, + has_connect: bool, + shutdown: bool, + tip: u32, +) -> CatchUp { + if catch_up.is_complete() || shutdown || !has_connect || tip > 0 { + catch_up + } else { + CatchUp::complete_dial_failed() + } +} + +/// Tip-follow may run below `--min-chain-work`. Relay stays off until the floor. +pub(crate) fn relay_while_following(meets_min_work: bool, blocks_only: bool, in_ibd: bool) -> bool { + meets_min_work && !blocks_only && !in_ibd +} + +async fn resolve_connect_dns(hosts: &[String], network: Network) -> Vec { + if hosts.is_empty() { + return Vec::new(); + } + let hosts = hosts.to_vec(); + let port = network.default_p2p_port(); + tokio::task::spawn_blocking(move || { + hosts + .iter() + .filter_map(|h| { + rbitcoin_net::parse_peer_addr_with_port(h, Some(port)) + .ok() + .map(rbitcoin_net::NetAddr::from_socket) + }) + .collect() + }) + .await + .unwrap_or_default() +} + fn queue_proxy_seed_addrfetch(peers: &Arc, network: Network) -> usize { let mut n = 0usize; for (host, port) in socks_dns_seed_dests(network) { @@ -2193,6 +2253,46 @@ mod tests { assert_eq!(catch_up_after_err(10, true, true), CatchUp::Incomplete); } + #[test] + fn connect_genesis_incomplete_enters_tip_follow() { + assert_eq!( + catch_up_with_connect(CatchUp::Incomplete, true, false, 0), + CatchUp::complete_dial_failed() + ); + assert_eq!( + catch_up_with_connect(CatchUp::Incomplete, false, false, 0), + CatchUp::Incomplete + ); + assert_eq!( + catch_up_with_connect(CatchUp::Incomplete, true, true, 0), + CatchUp::Incomplete + ); + assert!(catch_up_with_connect(CatchUp::complete(), true, false, 0).is_complete()); + } + + #[test] + fn connect_nongenesis_incomplete_stays_in_ibd() { + assert_eq!( + catch_up_with_connect(CatchUp::Incomplete, true, false, 50), + CatchUp::Incomplete + ); + } + + #[test] + fn relay_requires_min_chain_work_even_when_following() { + assert!(!relay_while_following(false, false, false)); + assert!(relay_while_following(true, false, false)); + assert!(!relay_while_following(true, true, false)); + assert!(!relay_while_following(true, false, true)); + let mut cfg = NodeConfig::default(); + assert!(cfg.meets_minimum_chain_work([0; 32])); + cfg.minimum_chain_work = Some([0xff; 32]); + assert!( + !cfg.meets_minimum_chain_work([0; 32]), + "a non-genesis tip under --min-chain-work follows, but relay stays gated" + ); + } + #[test] fn cjdns_listen_is_advertised_without_external_ip() { use std::net::{IpAddr, Ipv6Addr}; diff --git a/crates/rbitcoin-rpc/src/methods/net.rs b/crates/rbitcoin-rpc/src/methods/net.rs index 8a3b4bd72..bdc9bb695 100644 --- a/crates/rbitcoin-rpc/src/methods/net.rs +++ b/crates/rbitcoin-rpc/src/methods/net.rs @@ -178,9 +178,7 @@ pub(crate) fn addnode(ctx: &RpcContext, params: &RpcParams) -> Result/AGENTS.md` when that file | Minor, patch, or major release | [`../.agents/skills/release/SKILL.md`](../.agents/skills/release/SKILL.md) | | Core functional harness | [`../.agents/skills/core-functional/SKILL.md`](../.agents/skills/core-functional/SKILL.md) | | Overlay functional harness (private Tor / i2pd / cjdns) | [`../.agents/skills/overlay-functional/SKILL.md`](../.agents/skills/overlay-functional/SKILL.md) | +| Warnet two-tank example (label `warnet`) | [`core-functional.md`](./core-functional.md) (Warnet lab) | ## Ask first diff --git a/docs/core-functional.md b/docs/core-functional.md index b0abe492a..6b4fd1b09 100644 --- a/docs/core-functional.md +++ b/docs/core-functional.md @@ -283,3 +283,66 @@ no `store/` (clean-chain starts stay empty). python3 scripts/core-functional/create_cache.py --ensure ./scripts/core-functional/create_cache.test.sh ``` + +## Warnet lab (all-rbitcoin tanks) + +Not an operator musl Release. Image tag `rbitcoin-warnet:local`. Warnet +stays Core Helm (`bitcoin.conf`, `rpcuser`/`rpcpassword`, `addnode=tank-N`, +`pidof bitcoind`). The node is not taught Core conf; the Python shim +translates `addnode=` to `--connect host:18444`, seeds `{datadir}/rpc.token` +from `rpcpassword`, and binds the test proxy on `rpcbind=0.0.0.0`. Basic +auth accepts any username whose password matches that token. + +Hostname `--connect` / `addnode add` resolve at each dial on a blocking +thread and retry every 2 seconds until a live session exists. `--connect` +does not turn DNS seeds back on. If the first catch-up accepts nothing and +the tip is still 0, the tank enters tip-follow (16 blocks in flight, no +return to the IBD window) so it can attach when the other tank appears. +That is the right shape for this one-block example, not for a genesis +mainnet sync. A non-zero tip that has not finished catch-up stays in IBD. +Relay stays off while tip work is below `--min-chain-work`. The operator +note is in [`OPERATOR.md`](../OPERATOR.md). + +### CI example (label `warnet`) + +Two tanks on one Docker network, not kind or Helm. Unlabeled PRs do not +run it. `workflow_dispatch` also runs it. + +```bash +cargo build -p rbitcoin-node +./scripts/core-functional/init-submodule.sh +./scripts/core-functional/warnet/example.sh +``` + +The example conf sets `head_scale=tiny` so a tank does not fallocate mainnet +heads. `example.sh` copies a **bookworm-linked** `rbitcoin-node` into the image +(a nix devshell binary will not exec: its dynamic loader is not in Debian). +It starts `tank1` first (`addnode=tank0`), waits until that RPC is up at +height 0 with no peer, then starts `tank0`, mines one regtest block, and +waits until `tank1` matches that height with a live peer whose +`getpeerinfo` addr is the resolved address. CI builds the binary with +`rust:1.95.0-bookworm`. + +`Dockerfile.test.sh` pins the image text and does not need Docker. + +### Operator kind / miner_std + +Needs Docker + kind on an operator host. Do not open mainnet. Wallet keys +are RAM-only. + +```bash +docker build -t rbitcoin-warnet:local \ + --build-arg NODE_BIN=target/dev/debug/rbitcoin-node \ + -f scripts/core-functional/warnet/Dockerfile . +kind load docker-image rbitcoin-warnet:local +python3 -m venv .venv && source .venv/bin/activate +pip install warnet +warnet setup +warnet new /tmp/rbtc-warnet +``` + +Three tanks, ring `addnode`, unique `rpcpassword`, `pullPolicy: Never`. +Pass when `miner_std.py --interval=10 --mature` leaves all three +`getblockcount` values equal and greater than 0. Fail classes: CrashLoop +`pidof`; 401 on RPC; height only on the miner (DNS or bind); ImagePullBackOff +without `kind load`. diff --git a/docs/releases.md b/docs/releases.md index 86646ddb3..ea48e8605 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -106,6 +106,7 @@ Cargo.toml). Those PRs run Core functional even without a label. | `fmt` `deny` `clippy` `ast-grep` `test` `windows` `macos` `coverage` `nixos-module-eval` | Every PR (`ci.yml`) | | `core-functional` | Nightly, `workflow_dispatch`, label **`core-functional`**, label **`release`**, **or** ship version | | `overlay-functional` | Nightly (`42 6`), `workflow_dispatch`, label **`overlay-functional`**, label **`release`**, **or** ship version. Not required. Not in `release-extra` yet. | +| `warnet-example` | Label **`warnet`** or `workflow_dispatch`. Two-tank Docker lab. Not required. Not a ship gate. | | `nixos-module-runtime` | Label **`nixos-module-runtime`**, `workflow_dispatch`, **or** GitHub Release tags (`release.yml`). Not required. | | `release-extra` | Every PR. **Fails** if the PR is ship and `core-functional` is not success | diff --git a/docs/rpc.md b/docs/rpc.md index 79a810058..b83694d94 100644 --- a/docs/rpc.md +++ b/docs/rpc.md @@ -92,7 +92,7 @@ still wait for durable SH when shindex is on. | `ping` | All networks. Queues a ping on each live session (`null`). | | `addpeeraddress` | Hidden Core name. Inserts `{address,port}` into addrman RAM (does not rewrite `peers` per call). | | `getnodeaddresses` | Sample from addrman (`count=0` → all). Optional `network` filter. | -| `addnode` / `disconnectnode` / `addconnection` | All networks. `addnode onetry` / `add` dial; `disconnectnode` by `nodeid` or address | +| `addnode` / `disconnectnode` / `addconnection` | All networks. Hostnames resolve at dial (network default P2P port if omitted). `addnode add` and `--connect` retry until a live session. `addnode onetry` dials once and errors if DNS fails. `disconnectnode` by `nodeid` or address | | `getmempoolinfo` / `getrawmempool` / `getmempoolentry` | MempoolHub. `maxmempool` is the operator weight budget (`--mempool-size-mb`). `ancestorcount` / `descendantcount` (and size/fee sums) walk the cluster graph. Verbose `fees.{base,modified,ancestor,descendant,chunk}` and `chunkweight` include `prioritisetransaction` deltas; top-level `ancestorfees` / `descendantfees` stay base satoshis. `unbroadcastcount` / `unbroadcast` track `sendrawtransaction` txs until a peer getdata's them. `orphanage.{size,bytes}` is the parked missing-parent side pool (vsize). `permitbaremultisig` is always `true` (Libre has no Core `IsStandard` bare-multisig gate; `--permitbaremultisig` is not a node flag). | | `getorphantxs` | Hidden operator dump. Verbosity 0 txids, 1 details + `from` peer ids, 2 + hex. Not listed by `help` / `getrpcinfo`. Counts-only `getmempoolinfo.orphanage` is not a substitute. | | `getrawtransaction` | Class A + mempool. Optional Core `blockhash` arg is accepted and ignored. Verbose objects share `tx_to_json` with `decoderawtransaction` / `getblock` verbosity 2 (`scriptSig`, `scriptPubKey.type`). Below `pruneheight`: `-8` `Transaction not available (pruned data)` (not `-5`). | diff --git a/scripts/core-functional/bitcoin-cli b/scripts/core-functional/bitcoin-cli index 7a4508dcb..0e3de814d 100755 --- a/scripts/core-functional/bitcoin-cli +++ b/scripts/core-functional/bitcoin-cli @@ -8,6 +8,7 @@ parallel. Core's TestNodeCLI shells out to this binary. from __future__ import annotations import json +import os import sys import urllib.error import urllib.request @@ -85,7 +86,10 @@ def main(argv: list[str]) -> int: print("too few parameters (need at least command)", file=sys.stderr) return 1 if datadir is None: - print("error: -datadir is required", file=sys.stderr) + env = os.environ.get("BITCOIN_DATA") + datadir = Path(env) if env else Path.home() / ".bitcoin" + if not datadir.is_dir(): + print(f"error: -datadir {datadir} is not a directory", file=sys.stderr) return 1 if rpcport is None: @@ -118,6 +122,11 @@ def main(argv: list[str]) -> int: params.append(a) cookie = cookie_auth(datadir) + if cookie is None: + user = conf_value(datadir, "rpcuser") or "" + password = conf_value(datadir, "rpcpassword") + if password: + cookie = f"{user}:{password}" body = json.dumps({"jsonrpc": "1.0", "id": "cli", "method": method, "params": params}).encode() req = urllib.request.Request( f"http://127.0.0.1:{rpcport}/", diff --git a/scripts/core-functional/bitcoind b/scripts/core-functional/bitcoind index 3de1fffb9..b419b69b2 100755 --- a/scripts/core-functional/bitcoind +++ b/scripts/core-functional/bitcoind @@ -150,6 +150,13 @@ IGNORE_FLAGS = frozenset( # Stop-at-height is a later product; ignore so restart after the # future-tip check can proceed to the rest of rpc_blockchain. "rpcthreads", + # Warnet Helm conf. rpcpassword seeds rpc.token; addnode becomes --connect. + "rpcuser", + "rpcpassword", + "rpcbind", + "rpcallowip", + "rpcauth", + "fallbackfee", } ) @@ -339,6 +346,14 @@ def listen_targets( return seen +def proxy_bind_host(rpcbind: str) -> str: + """Helm `rpcbind=0.0.0.0` / `[::]` binds the test proxy all-interfaces.""" + host = (rpcbind or "").strip() + if host in ("0.0.0.0", "::", "[::]"): + return "0.0.0.0" + return "127.0.0.1" + + def maybe_seed_store(node_dir: Path) -> None: """Copy the rbitcoin 199-block store into a Core cache-shaped datadir. @@ -432,6 +447,7 @@ def translate( wipe_mempool = False blocksdir_arg: Path | None = None argv_binds: list[tuple[str, int | None, bool]] = [] + addnodes: list[str] = [] listen_off = False whitebind_vals: list[str] = [] rpc_proxy = True @@ -481,6 +497,7 @@ def translate( "walletdir", "wallet", "checkblocks", + "addnode", ): i += 1 if i >= len(argv): @@ -506,6 +523,8 @@ def translate( # Non-onion bind with an explicit port supplies the P2P listen port. if not onion and bport is not None: p2pport = str(bport) + elif key == "addnode" and val: + addnodes.append(val) elif key == "whitebind" and val: whitebind_vals.append(val) hostport = val.rsplit("@", 1)[-1] @@ -656,6 +675,22 @@ def translate( if log_level: cmd += ["--log-level", log_level] cmd += extra_node_args + for scale in read_conf_keys(datadir / "bitcoin.conf", "head_scale"): + if scale == "tiny" or scale == "mainnet": + cmd += ["--head-scale", scale] + for host in read_conf_keys(datadir / "bitcoin.conf", "addnode") + addnodes: + if not host: + continue + if ":" not in host: + host = f"{host}:{p2pport}" + cmd += ["--connect", host] + + pw = (conf.get("rpcpassword") or "").strip() + if pw: + token_path = node_dir / "rpc.token" + if not token_path.exists(): + token_path.write_text(pw + "\n") + token_path.chmod(0o600) bound_lines = [f"Bound to {addr}" for addr in listens] @@ -817,7 +852,9 @@ def main(argv: list[str] | None = None) -> int: return _child_init_exit() try: - proxy = RpcProxy(("127.0.0.1", public_rpc), node_url, _cookie) + core_datadir = node_dir.parent + rpcbind = read_conf(core_datadir / "bitcoin.conf").get("rpcbind") or "" + proxy = RpcProxy((proxy_bind_host(rpcbind), public_rpc), node_url, _cookie) except OSError: if child.poll() is None: child.terminate() diff --git a/scripts/core-functional/bitcoind.test.sh b/scripts/core-functional/bitcoind.test.sh index 173e450d3..ea2fab3a1 100755 --- a/scripts/core-functional/bitcoind.test.sh +++ b/scripts/core-functional/bitcoind.test.sh @@ -330,6 +330,21 @@ else FAIL=$((FAIL + 1)) fi +# Warnet Helm: addnode=tank name becomes --connect host:port and seeds rpc.token. +WN_DD="$WORKDIR/warnet-conf" +mkdir -p "$WN_DD" +printf 'regtest=1\nrpcuser=warnet\nrpcpassword=secret0\nrpcbind=0.0.0.0\naddnode=tank-0001\nhead_scale=tiny\nport=18444\nrpcport=18443\n' >"$WN_DD/bitcoin.conf" +OUT_WN="$("$SHIM" --print-cmd -datadir="$WN_DD" -regtest 2>/dev/null)" || OUT_WN="" +if printf '%s' "$OUT_WN" | grep -q -- "--connect tank-0001:18444" \ + && printf '%s' "$OUT_WN" | grep -q -- "--head-scale tiny" \ + && [[ "$(cat "$WN_DD/regtest/rpc.token")" == "secret0" ]]; then + echo "ok - addnode= becomes --connect and rpcpassword seeds rpc.token" + PASS=$((PASS + 1)) +else + echo "not ok - warnet conf translate (got: $OUT_WN token=$(cat "$WN_DD/regtest/rpc.token" 2>/dev/null))" + FAIL=$((FAIL + 1)) +fi + assert_fail_msg "port 65536 invalid" "Error: Invalid port specified in -port: '65536'" \ env RBITCOIN_NODE="$FAKE" "$SHIM" --print-cmd -datadir="$DATADIR" -regtest -listen -port=65536 assert_fail_msg "port 0 invalid" "Error: Invalid port specified in -port: '0'" \ diff --git a/scripts/core-functional/nightly.sh b/scripts/core-functional/nightly.sh index 623e9e9f5..adb3724ce 100755 --- a/scripts/core-functional/nightly.sh +++ b/scripts/core-functional/nightly.sh @@ -12,6 +12,7 @@ cd "$ROOT" ./scripts/core-functional/init-submodule.sh "$HERE/bitcoind.test.sh" +"$HERE/warnet/Dockerfile.test.sh" "$HERE/map_debuglog_test.sh" "$HERE/rpc_util_validateaddress.test.sh" "$HERE/check_inventory_test.sh" diff --git a/scripts/core-functional/rpc_proxy.py b/scripts/core-functional/rpc_proxy.py index 1f9521e32..e36de3bf4 100644 --- a/scripts/core-functional/rpc_proxy.py +++ b/scripts/core-functional/rpc_proxy.py @@ -38,6 +38,43 @@ def node_authorization(cookie_line: str) -> str: return f"Bearer {token}" +def token_from_cookie_line(cookie: str) -> str: + prefix = "__cookie__:" + if cookie.startswith(prefix): + return cookie[len(prefix) :] + return cookie + + +def parse_basic_userpass(authorization: str) -> tuple[str, str] | None: + if not authorization.startswith("Basic "): + return None + try: + raw = base64.b64decode(authorization[6:]).decode() + except (ValueError, UnicodeDecodeError): + return None + if ":" not in raw: + return None + user, password = raw.split(":", 1) + return user, password + + +def authorization_ok(authorization: str, cookie_line: str | None) -> bool: + """Cookie Basic, or any username whose password equals the token. + + Warnet sends `rpcuser:rpcpassword`. The username is not Core `rpcuser`. + """ + if not cookie_line: + return True + want = "Basic " + base64.b64encode(cookie_line.encode()).decode() + if authorization == want: + return True + parsed = parse_basic_userpass(authorization) + if parsed is None: + return False + _user, password = parsed + return password == token_from_cookie_line(cookie_line) + + def core_btc_kvb_to_sat_vb(value: Any) -> int: """Core `maxfeerate` BTC/kvB → node sat/vB. `>= 1` is Core `-8`.""" if value is None: @@ -215,10 +252,8 @@ def shutdown(self) -> None: def handle_http(self, raw: bytes, authorization: str) -> tuple[int, bytes]: cookie = self.cookie_line() - if cookie: - want = "Basic " + base64.b64encode(cookie.encode()).decode() - if authorization != want: - return 401, b'{"error":"unauthorized"}\n' + if not authorization_ok(authorization, cookie): + return 401, b'{"error":"unauthorized"}\n' try: payload = json.loads(raw.decode() or "null") except (UnicodeDecodeError, json.JSONDecodeError): diff --git a/scripts/core-functional/rpc_proxy.test.sh b/scripts/core-functional/rpc_proxy.test.sh index 97b63c7d2..1bf756fad 100755 --- a/scripts/core-functional/rpc_proxy.test.sh +++ b/scripts/core-functional/rpc_proxy.test.sh @@ -14,6 +14,7 @@ from http.server import BaseHTTPRequestHandler, HTTPServer from rpc_proxy import ( RpcError, RpcProxy, + authorization_ok, core_btc_kvb_to_sat_vb, esplora_port, node_authorization, @@ -159,6 +160,12 @@ assert node_authorization("__cookie__:secret") == "Bearer secret" assert node_authorization("secret") == "Bearer secret" COOKIE = "__cookie__:secret" +assert authorization_ok("Basic " + base64.b64encode(COOKIE.encode()).decode(), COOKIE) +attacker = "Basic " + base64.b64encode(b"attacker:secret").decode() +assert authorization_ok(attacker, COOKIE), "username is ignored; password must match the token" +assert not authorization_ok( + "Basic " + base64.b64encode(b"attacker:nope").decode(), COOKIE +) class FakeNode(BaseHTTPRequestHandler): diff --git a/scripts/core-functional/rpc_wallet.py b/scripts/core-functional/rpc_wallet.py index 040ebcdb4..d9c02d79a 100644 --- a/scripts/core-functional/rpc_wallet.py +++ b/scripts/core-functional/rpc_wallet.py @@ -133,6 +133,7 @@ def register(self) -> None: p.register("createwallet", self.createwallet) p.register("loadwallet", self.loadwallet) p.register("listwallets", self.listwallets) + p.register("listwalletdir", self.listwalletdir) p.register("getwalletinfo", self.getwalletinfo) p.register("importdescriptors", self.importdescriptors) p.register("importprivkey", self.importprivkey) @@ -206,6 +207,9 @@ def loadwallet(self, params: Any) -> dict[str, Any]: def listwallets(self, _params: Any) -> list[str]: return list(self.wallets.keys()) + def listwalletdir(self, _params: Any) -> dict[str, Any]: + return {"wallets": [{"name": name} for name in self.wallets]} + def getwalletinfo(self, _params: Any) -> dict[str, Any]: w = self._cur() bal = self._balance_sat(w, minconf=0) diff --git a/scripts/core-functional/warnet/Dockerfile b/scripts/core-functional/warnet/Dockerfile new file mode 100644 index 000000000..94dd443bf --- /dev/null +++ b/scripts/core-functional/warnet/Dockerfile @@ -0,0 +1,38 @@ +# Lab image for an all-rbitcoin Warnet tank. Not an operator Release. +# The copied node must use Debian bookworm's dynamic linker. A nix devshell +# binary fails at exec with "No such file or directory". +# +# docker run --rm -v "$PWD":/src -w /src -e CARGO_TARGET_DIR=/src/target/docker \ +# rust:1.95.0-bookworm cargo build -p rbitcoin-node +# ./scripts/core-functional/init-submodule.sh +# docker build -t rbitcoin-warnet:local \ +# --build-arg NODE_BIN=target/docker/debug/rbitcoin-node \ +# -f scripts/core-functional/warnet/Dockerfile . + +FROM debian:bookworm-slim + +ARG NODE_BIN=target/dev/debug/rbitcoin-node + +RUN apt-get update \ + && apt-get install -y --no-install-recommends python3 ca-certificates procps \ + && rm -rf /var/lib/apt/lists/* \ + && cp /usr/bin/python3 /usr/local/bin/bitcoind + +COPY ${NODE_BIN} /usr/local/bin/rbitcoin-node +COPY scripts/core-functional/bitcoind /opt/rbitcoin/shim/bitcoind +COPY scripts/core-functional/bitcoin-cli /usr/local/bin/bitcoin-cli +COPY scripts/core-functional/*.py /opt/rbitcoin/shim/ +COPY scripts/core-functional/debuglog_map.toml /opt/rbitcoin/shim/ +COPY third_party/bitcoin/test/functional/test_framework /opt/rbitcoin/functional/test_framework +COPY scripts/core-functional/warnet/entrypoint.sh /entrypoint.sh + +RUN chmod +x /usr/local/bin/rbitcoin-node /usr/local/bin/bitcoin-cli \ + /opt/rbitcoin/shim/bitcoind /entrypoint.sh \ + && chmod 755 /usr/local/bin/bitcoind + +ENV BITCOIN_DATA=/root/.bitcoin +ENV RBITCOIN_NODE=/usr/local/bin/rbitcoin-node +ENV RBITCOIN_LOG_STDOUT=1 +ENV PYTHONPATH=/opt/rbitcoin/shim:/opt/rbitcoin/functional + +ENTRYPOINT ["/entrypoint.sh"] diff --git a/scripts/core-functional/warnet/Dockerfile.test.sh b/scripts/core-functional/warnet/Dockerfile.test.sh new file mode 100755 index 000000000..b18ae4f3d --- /dev/null +++ b/scripts/core-functional/warnet/Dockerfile.test.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Text contract for the Warnet lab image (no docker required). +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +PASS=0 +FAIL=0 + +ok() { echo "ok - $1"; PASS=$((PASS + 1)); } +bad() { echo "not ok - $1"; FAIL=$((FAIL + 1)); } + +EP="$HERE/entrypoint.sh" +DF="$HERE/Dockerfile" + +if [[ -f "$EP" ]] && grep -q 'BITCOIN_DATA:-/root/.bitcoin' "$EP" \ + && grep -q 'exec' "$EP"; then + ok "entrypoint defaults -datadir to BITCOIN_DATA or /root/.bitcoin" +else + bad "entrypoint defaults -datadir to BITCOIN_DATA or /root/.bitcoin" +fi + +if [[ -f "$DF" ]] \ + && grep -q 'rbitcoin-node' "$DF" \ + && grep -q 'bitcoind' "$DF" \ + && grep -q 'bitcoin-cli' "$DF" \ + && grep -q 'test_framework' "$DF" \ + && grep -q 'RBITCOIN_LOG_STDOUT' "$DF"; then + ok "Dockerfile copies node, shims, test_framework, log tee" +else + bad "Dockerfile copies node, shims, test_framework, log tee" +fi + +echo +echo "$PASS passed, $FAIL failed" +if [[ "$FAIL" -ne 0 ]]; then + exit 1 +fi diff --git a/scripts/core-functional/warnet/entrypoint.sh b/scripts/core-functional/warnet/entrypoint.sh new file mode 100755 index 000000000..5199b03b9 --- /dev/null +++ b/scripts/core-functional/warnet/entrypoint.sh @@ -0,0 +1,12 @@ +#!/bin/sh +set -e +DATADIR="${BITCOIN_DATA:-/root/.bitcoin}" +export BITCOIN_DATA="$DATADIR" +export RBITCOIN_NODE="${RBITCOIN_NODE:-/usr/local/bin/rbitcoin-node}" +export RBITCOIN_LOG_STDOUT="${RBITCOIN_LOG_STDOUT:-1}" +export PYTHONPATH="${PYTHONPATH:-/opt/rbitcoin/shim:/opt/rbitcoin/functional}" +mkdir -p "$DATADIR" +if [ "$#" -eq 0 ]; then + exec /usr/local/bin/bitcoind /opt/rbitcoin/shim/bitcoind -datadir="$DATADIR" +fi +exec /usr/local/bin/bitcoind /opt/rbitcoin/shim/bitcoind "$@" diff --git a/scripts/core-functional/warnet/example-compose.yml b/scripts/core-functional/warnet/example-compose.yml new file mode 100644 index 000000000..00ffaeebc --- /dev/null +++ b/scripts/core-functional/warnet/example-compose.yml @@ -0,0 +1,13 @@ +# Two regtest tanks on one Docker network. Not a kind/Helm Warnet deploy. +# Hostnames are the service names (`addnode=tank0` → `--connect tank0:18444`). +services: + tank0: + image: rbitcoin-warnet:local + hostname: tank0 + volumes: + - ./example/tank0.conf:/root/.bitcoin/bitcoin.conf:ro + tank1: + image: rbitcoin-warnet:local + hostname: tank1 + volumes: + - ./example/tank1.conf:/root/.bitcoin/bitcoin.conf:ro diff --git a/scripts/core-functional/warnet/example.sh b/scripts/core-functional/warnet/example.sh new file mode 100755 index 000000000..02e48b04a --- /dev/null +++ b/scripts/core-functional/warnet/example.sh @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +# Two-tank regtest on the Warnet lab image. Needs Docker. Not kind/Helm. +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/../../.." && pwd)" +HERE="$(cd "$(dirname "$0")" && pwd)" +NODE_BIN="${NODE_BIN:-$ROOT/target/dev/debug/rbitcoin-node}" +IMAGE="${IMAGE:-rbitcoin-warnet:local}" +COMPOSE=(docker compose -f "$HERE/example-compose.yml" -p rbitcoin-warnet-example) + +if ! command -v docker >/dev/null 2>&1; then + echo "example.sh: docker is required" >&2 + exit 1 +fi +if [[ ! -x "$NODE_BIN" ]]; then + echo "example.sh: missing node binary $NODE_BIN" >&2 + exit 1 +fi +if ldd "$NODE_BIN" 2>/dev/null | grep -q '/nix/store/'; then + echo "example.sh: $NODE_BIN links the nix dynamic loader. Build it with rust:1.95.0-bookworm (see the Dockerfile comment)." >&2 + exit 1 +fi + +docker build -t "$IMAGE" \ + --build-arg "NODE_BIN=${NODE_BIN#"$ROOT"/}" \ + -f "$HERE/Dockerfile" \ + "$ROOT" + +cleanup() { + "${COMPOSE[@]}" down -t 5 >/dev/null 2>&1 || true +} +trap cleanup EXIT + +# Genesis --connect enters tip-follow, so tank1 answers RPC while tank0 is +# down. The 2s redial is what connects once that hostname exists. +"${COMPOSE[@]}" up -d --force-recreate --remove-orphans tank1 + +cli() { + local tank="$1" + shift + "${COMPOSE[@]}" exec -T "$tank" bitcoin-cli -regtest "$@" +} + +height() { + cli "$1" getblockcount 2>/dev/null | tr -d '[:space:]' || true +} + +peers() { + cli "$1" getconnectioncount 2>/dev/null | tr -d '[:space:]' || true +} + +# bitcoin-cli uses json.dumps, which writes `"addr": "ip:port"`. +peer_addr() { + local raw + raw="$(cli "$1" getpeerinfo 2>/dev/null || true)" + if [[ "$raw" =~ \"addr\":[[:space:]]*\"([^\"]+)\" ]]; then + printf '%s\n' "${BASH_REMATCH[1]}" + fi +} + +fail() { + echo "example.sh: $*" >&2 + "${COMPOSE[@]}" ps -a >&2 || true + "${COMPOSE[@]}" logs --no-color >&2 || true + exit 1 +} + +deadline=$((SECONDS + 90)) +until [[ "$(height tank1)" == "0" ]]; do + if (( SECONDS > deadline )); then + fail "tank1 did not answer getblockcount (got '$(height tank1)')" + fi + sleep 2 +done + +if [[ "$(peers tank1)" != "0" ]]; then + fail "tank1 had a peer before tank0 existed (peers='$(peers tank1)' addr='$(peer_addr tank1)')" +fi + +"${COMPOSE[@]}" up -d --remove-orphans tank0 + +deadline=$((SECONDS + 90)) +until [[ "$(height tank0)" == "0" ]]; do + if (( SECONDS > deadline )); then + fail "tank0 did not answer getblockcount (got '$(height tank0)')" + fi + sleep 2 +done + +cli tank0 generate 1 >/dev/null +want="$(height tank0)" +if [[ "$want" == "" || "$want" == "0" ]]; then + fail "tank0 generate did not advance (height='$want')" +fi + +deadline=$((SECONDS + 90)) +addr="" +while (( SECONDS <= deadline )); do + a="$(peer_addr tank1)" + if [[ "$(height tank1)" == "$want" && "$(peers tank1)" != "" && "$(peers tank1)" != "0" ]]; then + case "$a" in + ""|0.0.0.0:*) ;; + *) addr="$a"; break ;; + esac + fi + sleep 2 +done +if [[ -z "$addr" ]]; then + fail "tank1 did not follow tank0 (height='$(height tank1)' want='$want' peers='$(peers tank1)' addr='$(peer_addr tank1)')" +fi + +echo "ok - tank0=$want tank1=$(height tank1) peers=$(peers tank1) addr=$addr" diff --git a/scripts/core-functional/warnet/example/tank0.conf b/scripts/core-functional/warnet/example/tank0.conf new file mode 100644 index 000000000..06d37a790 --- /dev/null +++ b/scripts/core-functional/warnet/example/tank0.conf @@ -0,0 +1,8 @@ +regtest=1 +server=1 +head_scale=tiny +rpcuser=warnet +rpcpassword=secret0 +rpcbind=0.0.0.0 +port=18444 +rpcport=18443 diff --git a/scripts/core-functional/warnet/example/tank1.conf b/scripts/core-functional/warnet/example/tank1.conf new file mode 100644 index 000000000..a5a2e5593 --- /dev/null +++ b/scripts/core-functional/warnet/example/tank1.conf @@ -0,0 +1,8 @@ +regtest=1 +server=1 +head_scale=tiny +rpcuser=warnet +rpcpassword=secret1 +port=18444 +rpcport=18443 +addnode=tank0