Repository navigation
[2/?] Local reputation: persistence and in-flight replay across restarts - #11266
GeorgeTsagk wants to merge 9 commits into
Conversation
e2d924b to
0c654c4
Compare
0c654c4 to
9d181ce
Compare
🔴 PR Severity: CRITICAL
🔴 Critical (11 files)
🟡 Medium (9 files)
🟢 Low (36 files)
AnalysisThis PR introduces a new HTLC reputation / channel-jamming-mitigation system and wires it directly into the core forwarding path: Per the classification rules, To override, add a |
1f18d99 to
35d7f93
Compare
35d7f93 to
9cf7ed7
Compare
Add a native SQL table holding the per channel local reputation state, one row per channel_id: the outgoing reputation decaying average and the incoming revenue aggregated average, each stored as the running value plus the timestamp it was last updated at, along with the revenue start time used for the warm-up factor. The channel_id is used as the key because it is stable for the channel's lifetime, unlike the short channel id, which changes when a zero-conf channel confirms or a channel is spliced. The averages decay lazily on read, so storing the value and timestamp verbatim is enough for a restarted node to decay them over its full downtime on the first read. The table is migration 000016 and is applied in production builds. The offers migration, which is only registered under the dev build tags, moves from 000016 to 000017 so that it stays after the last production migration and is not pulled in when production migrates to this schema version.
Add the Store interface through which the manager persists per channel reputation state, a no-op implementation for nodes without a native SQL backend, and the SQL implementation on top of the reputation_channels table. The store round-trips the decaying averages verbatim: running value plus last update time, and the revenue start time for the warm-up factor. Rows are keyed by the channel's channel_id, and a row with a malformed id is reported instead of decoded into a bogus channel.
Give the manager a Store. On Start the persisted channel state is loaded and on Stop, and every minute in between, channels whose averages changed are written back. A store that cannot be read fails Start rather than silently starting from empty state, and a failed write keeps the channels marked so the next flush retries them. The averages are restored with their persisted timestamps, not the load time. Decay is applied lazily on read, so this is what makes a restart transparent: the first read after it decays the value over the whole downtime, and the revenue warm-up factor keeps advancing from the original start. A restarted manager reads exactly the same values as one that never stopped, which the tests assert against a reference manager. Timestamps in the future on load, the clock went backwards, are clamped to now. Rows whose averages decayed to zero are restored like any other, since their revenue start time still drives the warm-up factor. RemoveChannel drops a closed channel from memory and the store, together with the HTLCs still pending with it as the outgoing link and the HTLCs that arrived on it, so that a late settle cannot recreate the channel to credit its fee. Load, flush and removal are serialised by a store mutex the hooks never take, so a flush cannot write a snapshot taken before a removal back after it.
Add benchmarks for the two store paths the persistence introduces, flush and load, over 10, 100 and 1000 channels on the real SQL store, and a forward and resolve loop with the store attached that flushes every 10k forwards, so the per forward cost includes the write cost amortised over the forwards that caused it. The test database helpers take a testing.TB so the benchmarks can share them.
Pending HTLCs are not persisted, so without this a restart wiped the in-flight view: the risk of HTLCs still held on the outgoing channels was no longer counted and their eventual resolution was ignored as unmatched. ReplayInFlight takes the HTLCs the switch still has open and tracks them like live forwards. The original forward time is not recoverable, so they are stamped with the current time and height: the hold time charged on resolution starts at the restart and the remaining worst case hold is measured from the current height. HTLCs that cannot be tracked, expired or already known, are skipped with a warning.
Add ActiveCircuits to the circuit map and the switch, returning a snapshot of the open circuits: the HTLCs forwarded on an outgoing link that are awaiting a settle or fail from the remote peer. The reputation subsystem uses it on startup to rebuild its view of the HTLCs that were in flight across the restart.
Build the SQL reputation store when the native SQL store is in use and hand it to the manager, so channel reputation survives a restart. Nodes on the KV backend get no store and keep the in-memory behaviour. On startup, before the switch starts re-forwarding pending resolutions, the HTLCs that were in flight across the restart are reconstructed and replayed into the manager. The circuit map only retains the circuit keys and amounts, so the incoming cltv expiry and the accountable signal are read back from the live incoming commitment HTLC, and circuits whose incoming HTLC is no longer live are skipped. The fee is the fee the node's own policies require for the forward, computed like the live forward path from the outgoing channel's outbound fee and the incoming channel's inbound fee, so that an over-paying sender cannot inflate reputation through a restart. It is capped at the fee the sender offered, since the policy may have changed since the forward was accepted, and the offered fee is used when a channel has no policy in the graph. Circuits key both ends by short channel id while the manager keys channels by channel_id, so both ends are mapped through the open channels. A circuit whose outgoing channel is no longer open is skipped as well. Closed channels are forwarded from the channel notifier to the manager, by the channel_id of their channel point, so their reputation state is dropped from memory and the store.
Add an itest that restarts the forwarding node with a forward held in flight at the final hop. After the restart the node must report the in-flight HTLC as replayed, and when the hold invoice settles the resolution must be scored rather than ignored as unknown. When the harness runs with native SQL both channels must be reported as loaded from the store and the settle must land on top of the reputation from before the restart. Otherwise nothing is loaded and the channel starts over, which the test asserts as well. The lntest harness gains NodeLogSubmatches to read values out of log lines.
9cf7ed7 to
4d879df
Compare
Description
Second part of the local reputation work started in #10919.
Makes the reputation state survive a restart of the node:
db.use-native-sql, one row perchannel_id. Nodes on the KV backend getno store and keep the in-memory behaviour of [1/?] Local reputation: subsystem core, read only #10919.
running value plus last update time and restored verbatim, so the first
read after a restart decays them over the full gap. A restarted manager
reads exactly what a manager that never stopped would read.
open circuits before it starts re-forwarding resolutions, with the incoming
cltv and the accountable signal read back from the commitment HTLC. Their
in-flight risk is counted again and their settle or fail is scored instead
of being dropped as unknown. They are scored on the fee our own policy
requires, like a live forward, capped at the fee the sender offered. The
original add time and height are not recoverable, so the hold is measured
from the restart.
Still log-only. Nothing here affects forwarding or the wire. The forward and
resolve benchmarks from #10919 are unchanged, and the new persistence
benchmarks put the periodic flush at about 10 ms per 1000 changed channels on
sqlite, once a minute, off the forwarding path.
The itest restarts the forwarding node with two HTLCs held at the final hop,
one of them accountable, and asserts both are replayed, one is scored as
settled and the other as failed, and that reputation carries across the
restart only with the native SQL store.
Not in scope: resource buckets and any enforcement, which come in later parts. This PR does not affect HTLC routing.