Skip to content

[2/?] Local reputation: persistence and in-flight replay across restarts - #11266

Draft
GeorgeTsagk wants to merge 9 commits into
lightningnetwork:masterfrom
GeorgeTsagk:local-reputation-persist
Draft

GeorgeTsagk wants to merge 9 commits into
lightningnetwork:masterfrom
GeorgeTsagk:local-reputation-persist

Conversation

@GeorgeTsagk

@GeorgeTsagk GeorgeTsagk commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Second part of the local reputation work started in #10919.

Makes the reputation state survive a restart of the node:

  • Channel state is persisted in a native SQL table when running with
    db.use-native-sql, one row per channel_id. Nodes on the KV backend get
    no store and keep the in-memory behaviour of [1/?] Local reputation: subsystem core, read only #10919.
  • Decay across downtime comes for free: the decaying averages are stored as
    running value plus last update time and restored verbatim, so the first
    read after a restart decays them over the full gap. A restarted manager
    reads exactly what a manager that never stopped would read.
  • HTLCs that were in flight during the restart are rebuilt from the switch's
    open circuits before it starts re-forwarding resolutions, with the incoming
    cltv and the accountable signal read back from the commitment HTLC. Their
    in-flight risk is counted again and their settle or fail is scored instead
    of being dropped as unknown. They are scored on the fee our own policy
    requires, like a live forward, capped at the fee the sender offered. The
    original add time and height are not recoverable, so the hold is measured
    from the restart.
  • Closed channels are removed from memory and from the store.

Still log-only. Nothing here affects forwarding or the wire. The forward and
resolve benchmarks from #10919 are unchanged, and the new persistence
benchmarks put the periodic flush at about 10 ms per 1000 changed channels on
sqlite, once a minute, off the forwarding path.

The itest restarts the forwarding node with two HTLCs held at the final hop,
one of them accountable, and asserts both are replayed, one is scored as
settled and the other as failed, and that reputation carries across the
restart only with the native SQL store.

Not in scope: resource buckets and any enforcement, which come in later parts. This PR does not affect HTLC routing.

@GeorgeTsagk
GeorgeTsagk force-pushed the local-reputation-persist branch from e2d924b to 0c654c4 Compare September 28, 2026 09:33
@GeorgeTsagk GeorgeTsagk self-assigned this Sep 28, 2026
@GeorgeTsagk GeorgeTsagk added database Related to the database/storage of LND logging Related to the logging / debug output functionality channel jamming Issues related to channel jamming mitigation labels Sep 28, 2026
@GeorgeTsagk
GeorgeTsagk force-pushed the local-reputation-persist branch from 0c654c4 to 9d181ce Compare October 6, 2026 10:57
@github-actions github-actions Bot added the severity-critical Requires expert review - security/consensus critical label Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

🔴 PR Severity: CRITICAL

56 files | 5911 additions / 14 deletions

🔴 Critical (11 files)
  • htlcswitch/circuit_map.go - HTLC circuit map, core forwarding state
  • htlcswitch/interfaces.go - HTLC switch interfaces
  • htlcswitch/link.go - HTLC link forwarding logic
  • htlcswitch/mock.go - htlcswitch test double, same package as forwarding logic
  • htlcswitch/reputation_guard.go - new hook gating HTLC forwarding decisions
  • htlcswitch/switch.go - core HTLC forwarding/routing state machine
  • server.go - core server coordination, wires reputation manager into startup
  • sqldb/migrations.go - registers new DB migration
  • sqldb/sqlc/migrations/000016_reputation.up.sql / .down.sql - new schema migration
  • sqldb/sqlc/models.go / querier.go / reputation.sql.go - generated DB layer for new migration
  • sqldb/sqlc/queries/reputation.sql - new SQL queries backing the migration
🟡 Medium (9 files)
  • config_builder.go - wires new reputation config into the config builder
  • lncfg/db.go, lncfg/routing.go - new config flags
  • log.go - registers new reputation subsystem logger
  • reputation/*.go (channel, config, decaying_average, decision, htlc, log, manager, persist, replay, revenue, saturated, sql_store, store) - new standalone reputation/jamming-mitigation package
  • reputation_wiring.go - wires reputation manager into the server
  • sample-lnd.conf - documents new config options
🟢 Low (36 files)
  • docs/release-notes/release-notes-0.22.0.md - release notes
  • htlcswitch/circuit_test.go, link_test.go, reputation_hooks_test.go - test-only
  • itest/list_on_test.go, itest/lnd_reputation_persistence_test.go, itest/lnd_reputation_test.go - integration tests
  • lntest/harness_assertion.go - test harness helper
  • reputation/*_test.go, reputation/bench_test.go, reputation/testutil_test.go - test-only
  • reputation_wiring_test.go, sqldb/sqlc/sql_store_postgres_test.go, sqldb/sqlc/sql_store_sqlite_test.go - test-only

Analysis

This PR introduces a new HTLC reputation / channel-jamming-mitigation system and wires it directly into the core forwarding path: htlcswitch/switch.go, link.go, circuit_map.go, and interfaces.go are all modified to call into the new reputation_guard.go hook, meaning forwarding decisions are now gated on reputation state. It also adds a new database migration (sqldb/sqlc/migrations/000016_reputation.*) and wires the new manager into server.go startup.

Per the classification rules, htlcswitch/* changes and any sqldb/* migration are always CRITICAL, and this PR touches both, plus server.go directly — so it lands at CRITICAL independent of size. It also independently qualifies for a severity bump (56 files, ~5.9k lines changed, multiple distinct critical packages touched: htlcswitch, sqldb, server.go), though the ceiling is already CRITICAL. This warrants expert review of the HTLC forwarding state machine and the new persistence layer before merge.


To override, add a severity-override-{critical,high,medium,low} label.

@GeorgeTsagk
GeorgeTsagk force-pushed the local-reputation-persist branch 4 times, most recently from 1f18d99 to 35d7f93 Compare October 9, 2026 08:53
@GeorgeTsagk
GeorgeTsagk force-pushed the local-reputation-persist branch from 35d7f93 to 9cf7ed7 Compare October 16, 2026 14:16
@saubyk saubyk added this to lnd v0.22 Oct 16, 2026
@github-project-automation github-project-automation Bot moved this to Backlog in lnd v0.22 Oct 16, 2026
@saubyk saubyk moved this from Backlog to In progress in lnd v0.22 Oct 16, 2026
@saubyk saubyk added this to the v0.22.0 milestone Oct 16, 2026
Add a native SQL table holding the per channel local reputation state, one
row per channel_id: the outgoing reputation decaying average and the
incoming revenue aggregated average, each stored as the running value plus
the timestamp it was last updated at, along with the revenue start time
used for the warm-up factor. The channel_id is used as the key because it
is stable for the channel's lifetime, unlike the short channel id, which
changes when a zero-conf channel confirms or a channel is spliced.

The averages decay lazily on read, so storing the value and timestamp
verbatim is enough for a restarted node to decay them over its full
downtime on the first read.

The table is migration 000016 and is applied in production builds. The
offers migration, which is only registered under the dev build tags, moves
from 000016 to 000017 so that it stays after the last production migration
and is not pulled in when production migrates to this schema version.
Add the Store interface through which the manager persists per channel
reputation state, a no-op implementation for nodes without a native SQL
backend, and the SQL implementation on top of the reputation_channels
table.

The store round-trips the decaying averages verbatim: running value plus
last update time, and the revenue start time for the warm-up factor. Rows
are keyed by the channel's channel_id, and a row with a malformed id is
reported instead of decoded into a bogus channel.
Give the manager a Store. On Start the persisted channel state is loaded
and on Stop, and every minute in between, channels whose averages changed
are written back. A store that cannot be read fails Start rather than
silently starting from empty state, and a failed write keeps the channels
marked so the next flush retries them.

The averages are restored with their persisted timestamps, not the load
time. Decay is applied lazily on read, so this is what makes a restart
transparent: the first read after it decays the value over the whole
downtime, and the revenue warm-up factor keeps advancing from the original
start. A restarted manager reads exactly the same values as one that never
stopped, which the tests assert against a reference manager. Timestamps in
the future on load, the clock went backwards, are clamped to now. Rows
whose averages decayed to zero are restored like any other, since their
revenue start time still drives the warm-up factor.

RemoveChannel drops a closed channel from memory and the store, together
with the HTLCs still pending with it as the outgoing link and the HTLCs
that arrived on it, so that a late settle cannot recreate the channel to
credit its fee. Load, flush and removal are serialised by a store mutex
the hooks never take, so a flush cannot write a snapshot taken before a
removal back after it.
Add benchmarks for the two store paths the persistence introduces, flush
and load, over 10, 100 and 1000 channels on the real SQL store, and a
forward and resolve loop with the store attached that flushes every 10k
forwards, so the per forward cost includes the write cost amortised over
the forwards that caused it. The test database helpers take a testing.TB
so the benchmarks can share them.
Pending HTLCs are not persisted, so without this a restart wiped the
in-flight view: the risk of HTLCs still held on the outgoing channels was
no longer counted and their eventual resolution was ignored as unmatched.

ReplayInFlight takes the HTLCs the switch still has open and tracks them
like live forwards. The original forward time is not recoverable, so they
are stamped with the current time and height: the hold time charged on
resolution starts at the restart and the remaining worst case hold is
measured from the current height. HTLCs that cannot be tracked, expired or
already known, are skipped with a warning.
Add ActiveCircuits to the circuit map and the switch, returning a snapshot
of the open circuits: the HTLCs forwarded on an outgoing link that are
awaiting a settle or fail from the remote peer. The reputation subsystem
uses it on startup to rebuild its view of the HTLCs that were in flight
across the restart.
Build the SQL reputation store when the native SQL store is in use and
hand it to the manager, so channel reputation survives a restart. Nodes
on the KV backend get no store and keep the in-memory behaviour.

On startup, before the switch starts re-forwarding pending resolutions,
the HTLCs that were in flight across the restart are reconstructed and
replayed into the manager. The circuit map only retains the circuit keys
and amounts, so the incoming cltv expiry and the accountable signal are
read back from the live incoming commitment HTLC, and circuits whose
incoming HTLC is no longer live are skipped.

The fee is the fee the node's own policies require for the forward,
computed like the live forward path from the outgoing channel's outbound
fee and the incoming channel's inbound fee, so that an over-paying sender
cannot inflate reputation through a restart. It is capped at the fee the
sender offered, since the policy may have changed since the forward was
accepted, and the offered fee is used when a channel has no policy in the
graph.

Circuits key both ends by short channel id while the manager keys channels
by channel_id, so both ends are mapped through the open channels. A
circuit whose outgoing channel is no longer open is skipped as well.

Closed channels are forwarded from the channel notifier to the manager,
by the channel_id of their channel point, so their reputation state is
dropped from memory and the store.
Add an itest that restarts the forwarding node with a forward held in
flight at the final hop. After the restart the node must report the
in-flight HTLC as replayed, and when the hold invoice settles the
resolution must be scored rather than ignored as unknown.

When the harness runs with native SQL both channels must be reported as
loaded from the store and the settle must land on top of the reputation
from before the restart. Otherwise nothing is loaded and the channel starts
over, which the test asserts as well.

The lntest harness gains NodeLogSubmatches to read values out of log lines.
@GeorgeTsagk
GeorgeTsagk force-pushed the local-reputation-persist branch from 9cf7ed7 to 4d879df Compare October 16, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel jamming Issues related to channel jamming mitigation database Related to the database/storage of LND logging Related to the logging / debug output functionality severity-critical Requires expert review - security/consensus critical

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

2 participants