diff --git a/docs/payment-router-settlement-recovery.md b/docs/payment-router-settlement-recovery.md new file mode 100644 index 0000000..ff239fb --- /dev/null +++ b/docs/payment-router-settlement-recovery.md @@ -0,0 +1,85 @@ +# payment-router: failed settlement and the dead-letter path + +Settlement fails for reasons the contract cannot prevent and cannot see: a +destination account that no longer exists, an asset frozen or clawed back, a +trustline removed after the escrow was opened. This is what happens then. + +## Pull, not push + +Release and payout are already separate. `release_partial` makes funds +claimable and calls no token; the payee pulls with `claim`. Recovery keeps that +shape — the contract never retries a push against a destination that just +failed. Push-based retry burns fees on every attempt, and an attacker who can +make a destination fail can make the contract pay for the attempts. + +So there is no retry mechanism here. The retry *is* `claim`, called again by +the payee once their account works. + +## States + +| State | Meaning | Claim | +| --- | --- | --- | +| healthy | no failures recorded | claimable | +| failed | `failures >= 3` reported | **preserved**, still claimable | +| dead-lettered | swept after the delay | ended | + +`Failed` preserves the claim. It is a signal to operators, not a confiscation. + +## Classification + +A reverted `transfer` takes the whole transaction with it, so there is no +in-band way for the contract to observe a failure and carry on. The payee +reports it with `report_failure`, signing for their own escrow, and classifies +it as `NoDestination`, `NoTrustline`, `AssetRestricted`, or `Other`. + +The classification is not decorative: the operator response differs. A removed +trustline is fixed by the payee in one transaction; a clawed-back asset is not +something the payee can do anything about. An indexer that only sees "failed" +cannot tell those apart. `SettlementFailed` carries the reason and the running +count, and sets `exhausted` on the attempt that tips the escrow into `Failed`. + +`clear_failure` resets the record once the payee's account works, so a later +failure counts from zero rather than from a stale total, and a recovered escrow +stops looking broken on a dashboard. + +## Dead-letter policy + +Three questions have to be answered explicitly, or the path is a liability. + +**When.** `MAX_SETTLEMENT_FAILURES` (3) reported failures, *and* then +`DEAD_LETTER_DELAY` (120,960 ledgers, roughly seven days) after the escrow +entered `Failed`. Three attempts distinguishes a transient problem from a +broken destination. The week exists for the payee's benefit — a removed +trustline is fixable, and value should not be swept from under someone who is +fixing it. `dead_letter_at` reports the deadline. + +**Where the value goes.** Back to the payer. The payee has demonstrably been +unable to receive it across three attempts and a week. Returning it to the +party who put it in is the only destination that requires no new trust +assumption: it is not burned, and it does not accrue to the contract or to any +treasury. This is a return, not a forfeiture. + +**Who can trigger it.** The arbiter. Deliberately *not* the payer, who is the +beneficiary — a payer who could both trigger the sweep and receive it would +have an incentive to grief a payee into the failed state and take the value +back. Not the payee either, for whom it does nothing. + +The sweep advances the escrow's `claimed` to its `released` before transferring, +which is the same accounting a successful pull performs. That is what stops the +payee pulling value that has already been returned, and keeps the contract's +balance equal to what it still owes. + +## Events + +`SettlementFailed`, `SettlementRecovered`, and `DeadLettered` are emitted at +each step so a backend can alert an operator rather than discovering a stuck +escrow by inspection. + +## Storage + +Recovery state lives in a `DataKey::Settlement(id)` side-car rather than as +fields on `Escrow`. Recovery is the exception — most escrows never have one — +and a side-car means those escrows pay nothing for it. It uses the same +persistent TTL policy as the escrow it describes: state that outlived its +escrow would be useless, and state that expired first would silently reset the +failure count. diff --git a/payment-router/src/lib.rs b/payment-router/src/lib.rs index 7d9630a..b641e54 100644 --- a/payment-router/src/lib.rs +++ b/payment-router/src/lib.rs @@ -25,6 +25,20 @@ const PERSISTENT_THRESHOLD: u32 = 259_200; /// making one escrow grow without limit. const MAX_RELEASES: u32 = 32; +/// Failed settlement attempts before an escrow is moved to `Failed`. +/// +/// Three separate attempts is enough to distinguish a transient problem from +/// a destination that is genuinely broken, without leaving a payee who fixed +/// their trustline on the second try stuck in a recovery state. +const MAX_SETTLEMENT_FAILURES: u32 = 3; + +/// Ledgers a `Failed` escrow waits before the dead-letter path opens. +/// +/// Roughly seven days at five seconds a ledger. The window exists for the +/// payee's benefit: a removed trustline or a frozen asset is fixable, and the +/// value should not be swept out from under someone who is fixing it. +const DEAD_LETTER_DELAY: u32 = 120_960; + #[contracterror] #[derive(Copy, Clone, Debug, Eq, PartialEq, PartialOrd, Ord)] #[repr(u32)] @@ -39,6 +53,13 @@ pub enum Error { InsufficientEscrow = 6, ReleaseCapReached = 7, NothingToClaim = 8, + /// A recovery step was asked for on an escrow that has not accumulated + /// enough failed settlement attempts to be in the `Failed` state. + SettlementNotFailed = 20, + /// The escrow is failed, but the dead-letter delay has not yet elapsed. + /// The payee still has time to fix their account and pull. + DeadLetterNotReady = 21, + AlreadyDeadLettered = 22, NoDisputeOpen = 9, DisputeAlreadyOpen = 10, NotTheArbiter = 11, @@ -51,6 +72,7 @@ pub enum DataKey { Arbiter, Escrow(u64), Counter, + Settlement(u64), } #[contracttype] @@ -86,6 +108,47 @@ impl Escrow { } } +/// Why a settlement attempt did not land. +/// +/// Classified rather than swallowed: the three cases have different operator +/// responses, and an indexer that only sees "it failed" cannot tell an asset +/// freeze (nothing the payee can do) from a removed trustline (which the payee +/// fixes themselves in one transaction). +#[contracttype] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum FailureReason { + /// The destination account no longer exists. + NoDestination = 0, + /// No trustline for this asset on the destination. + NoTrustline = 1, + /// The asset is frozen, clawed back, or otherwise not transferable. + AssetRestricted = 2, + /// Anything else. Kept explicit so a new failure mode is recorded rather + /// than being forced into a category it does not belong in. + Other = 3, +} + +/// Recovery state for an escrow whose settlement is not landing. +/// +/// Stored under its own key rather than as fields on `Escrow`: recovery is the +/// exception, most escrows never have one, and a side-car entry means those +/// escrows pay nothing for it. +#[contracttype] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Settlement { + pub id: u64, + /// Consecutive reported failures. + pub failures: u32, + pub last_reason: FailureReason, + /// Ledger at which the escrow entered `Failed`. Only meaningful when + /// `failed` is set — a genesis-adjacent ledger is a valid sequence, so + /// this field is not a flag in disguise. + pub failed_at: u32, + /// `failures >= MAX_SETTLEMENT_FAILURES`. The claim is preserved, not lost. + pub failed: bool, + pub dead_lettered: bool, +} + #[contractevent] #[derive(Clone, Debug, Eq, PartialEq)] pub struct EscrowOpened { @@ -142,6 +205,40 @@ pub struct EscrowClosed { pub refunded: i128, } +#[contractevent] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct SettlementFailed { + #[topic] + pub id: u64, + #[topic] + pub payee: Address, + pub reason: FailureReason, + pub failures: u32, + /// True on the attempt that tipped the escrow into `Failed`. + pub exhausted: bool, +} + +#[contractevent] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct SettlementRecovered { + #[topic] + pub id: u64, + #[topic] + pub payee: Address, + pub amount: i128, +} + +#[contractevent] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct DeadLettered { + #[topic] + pub id: u64, + #[topic] + pub payer: Address, + pub amount: i128, + pub reason: FailureReason, +} + /// Escrow, dispute windows, payout release. #[contract] pub struct PaymentRouter; @@ -378,6 +475,156 @@ impl PaymentRouter { .publish(&env); } + // ----------------------------------------------------------------------- + // Failed settlement and recovery + // ----------------------------------------------------------------------- + + /// Record a settlement attempt that did not land. + /// + /// The contract cannot observe why a pull failed — a reverted `transfer` + /// takes the whole transaction with it, so there is no in-band way to + /// classify and continue. The payee reports it instead, signing for their + /// own escrow, and the classification is what the backend alerts on. + /// + /// Nothing is pushed and nothing is retried here. The claim is preserved + /// exactly as it was; this only records that the destination is not + /// working, which is what eventually opens the dead-letter path. + pub fn report_failure(env: Env, id: u64, reason: FailureReason) -> u32 { + let escrow = Self::load(&env, id); + escrow.payee.require_auth_for_args((id,).into_val(&env)); + + let mut state = Self::settlement(&env, id); + // Checked before the claim amount: after a sweep the claim is legitimately + // zero, and reporting NothingToClaim there would name a symptom of the + // dead-lettering rather than the dead-lettering itself. + if state.dead_lettered { + panic_with_error!(&env, Error::AlreadyDeadLettered); + } + if escrow.claimable() <= 0 { + panic_with_error!(&env, Error::NothingToClaim); + } + + state.failures += 1; + state.last_reason = reason; + + let exhausted = !state.failed && state.failures >= MAX_SETTLEMENT_FAILURES; + if exhausted { + state.failed = true; + state.failed_at = env.ledger().sequence(); + } + Self::save_settlement(&env, &state); + + SettlementFailed { + id, + payee: escrow.payee, + reason, + failures: state.failures, + exhausted, + } + .publish(&env); + + state.failures + } + + /// Clear the failure record after the payee has fixed their account. + /// + /// The recovery path is the ordinary `claim` — the payee pulls once their + /// trustline is back. This exists so a recovered escrow stops looking + /// broken to an operator, and so a later failure counts from zero rather + /// than from a stale total. + pub fn clear_failure(env: Env, id: u64) { + let escrow = Self::load(&env, id); + escrow.payee.require_auth_for_args((id,).into_val(&env)); + + let state = Self::settlement(&env, id); + if !state.failed { + panic_with_error!(&env, Error::SettlementNotFailed); + } + if state.dead_lettered { + panic_with_error!(&env, Error::AlreadyDeadLettered); + } + + let amount = escrow.claimable(); + Self::save_settlement( + &env, + &Settlement { + id, + failures: 0, + last_reason: FailureReason::Other, + failed_at: 0, + failed: false, + dead_lettered: false, + }, + ); + + SettlementRecovered { + id, + payee: escrow.payee, + amount, + } + .publish(&env); + } + + /// Dead-letter a settlement that has stayed broken. + /// + /// **Policy:** the value returns to the payer. The payee has demonstrably + /// been unable to receive it across `MAX_SETTLEMENT_FAILURES` attempts and + /// a `DEAD_LETTER_DELAY` window, and returning it to the party that put it + /// in is the only destination that needs no new trust assumption. It is + /// not burned and it does not accrue to the contract. + /// + /// **Who may trigger it:** the arbiter. Not the payer, who would otherwise + /// have an incentive to grief a payee into the failed state and sweep; + /// not the payee, for whom it does nothing. + /// + /// The escrow's claimed total is advanced to its released total, so the + /// swept amount stops being claimable. This is the same accounting a + /// successful pull performs, which keeps the balance invariant intact. + pub fn dead_letter(env: Env, id: u64) -> i128 { + Self::arbiter(&env).require_auth(); + + let mut escrow = Self::load(&env, id); + let mut state = Self::settlement(&env, id); + + if !state.failed { + panic_with_error!(&env, Error::SettlementNotFailed); + } + if state.dead_lettered { + panic_with_error!(&env, Error::AlreadyDeadLettered); + } + if env.ledger().sequence() < state.failed_at + DEAD_LETTER_DELAY { + panic_with_error!(&env, Error::DeadLetterNotReady); + } + + let amount = escrow.claimable(); + if amount <= 0 { + panic_with_error!(&env, Error::NothingToClaim); + } + + // State first, token second. Advancing `claimed` here is what stops + // the payee pulling the same value after it has been swept. + escrow.claimed += amount; + state.dead_lettered = true; + Self::save(&env, &escrow); + Self::save_settlement(&env, &state); + + token::Client::new(&env, &escrow.token).transfer( + &env.current_contract_address(), + &escrow.payer, + &amount, + ); + + DeadLettered { + id, + payer: escrow.payer, + amount, + reason: state.last_reason, + } + .publish(&env); + + amount + } + // ----------------------------------------------------------------------- // Reads // ----------------------------------------------------------------------- @@ -396,6 +643,28 @@ impl PaymentRouter { Self::load(&env, id).releasable() } + /// Recovery state for an escrow. Healthy escrows report zeroed counters + /// rather than an error, so a caller need not special-case the common path. + pub fn settlement_status(env: Env, id: u64) -> Settlement { + Self::load(&env, id); + Self::settlement(&env, id) + } + + /// Ledger at which the dead-letter path opens, or `None` while the escrow + /// is healthy. + pub fn dead_letter_at(env: Env, id: u64) -> Option { + let state = Self::settlement(&env, id); + if state.failed && !state.dead_lettered { + Some(state.failed_at + DEAD_LETTER_DELAY) + } else { + None + } + } + + pub fn max_settlement_failures(_env: Env) -> u32 { + MAX_SETTLEMENT_FAILURES + } + pub fn max_releases(_env: Env) -> u32 { MAX_RELEASES } @@ -420,6 +689,32 @@ impl PaymentRouter { } } + /// Recovery state, defaulted for an escrow that has never failed. + fn settlement(env: &Env, id: u64) -> Settlement { + env.storage() + .persistent() + .get(&DataKey::Settlement(id)) + .unwrap_or(Settlement { + id, + failures: 0, + last_reason: FailureReason::Other, + failed_at: 0, + failed: false, + dead_lettered: false, + }) + } + + /// Same TTL policy as the escrow itself: recovery state that outlived the + /// escrow it describes would be useless, and one that expired first would + /// silently reset the failure count. + fn save_settlement(env: &Env, state: &Settlement) { + let key = DataKey::Settlement(state.id); + env.storage().persistent().set(&key, state); + env.storage() + .persistent() + .extend_ttl(&key, PERSISTENT_THRESHOLD, PERSISTENT_TTL); + } + fn save(env: &Env, escrow: &Escrow) { let key = DataKey::Escrow(escrow.id); env.storage().persistent().set(&key, escrow); diff --git a/payment-router/src/test.rs b/payment-router/src/test.rs index a66d3c1..92ffb34 100644 --- a/payment-router/src/test.rs +++ b/payment-router/src/test.rs @@ -449,3 +449,264 @@ fn test_only_the_payee_can_pull() { assert_eq!(f.token().balance(&stranger), 0); assert_eq!(f.client().claimable(&id), 1_000); } + +// --------------------------------------------------------------------------- +// Failed settlement, recovery, and the dead-letter path +// --------------------------------------------------------------------------- + +/// Advance the ledger far enough for the dead-letter window to open. +fn advance(f: &Fixture, ledgers: u32) { + use soroban_sdk::testutils::Ledger as _; + let at = f.env.ledger().sequence(); + f.env.ledger().set_sequence_number(at + ledgers); +} + +#[test] +fn test_a_healthy_escrow_reports_zeroed_recovery_state() { + let f = setup(10_000); + let id = f.open(10_000); + + let state = f.client().settlement_status(&id); + assert_eq!(state.failures, 0); + assert!(!state.failed); + assert!(!state.dead_lettered); + // Nothing to recover from, so no dead-letter deadline exists yet. + assert_eq!(f.client().dead_letter_at(&id), None); +} + +#[test] +fn test_a_reported_failure_is_classified_and_preserves_the_claim() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &3_000); + + assert_eq!( + f.client().report_failure(&id, &FailureReason::NoTrustline), + 1 + ); + + let state = f.client().settlement_status(&id); + assert_eq!(state.failures, 1); + assert_eq!(state.last_reason, FailureReason::NoTrustline); + // Below the threshold, so still healthy - and the claim is untouched. + assert!(!state.failed); + assert_eq!(f.client().claimable(&id), 3_000); + assert_eq!(f.token().balance(&f.router), 10_000); +} + +#[test] +fn test_the_escrow_enters_the_failed_state_after_the_third_attempt() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &3_000); + + for expected in 1..=f.client().max_settlement_failures() { + assert_eq!( + f.client() + .report_failure(&id, &FailureReason::NoDestination), + expected + ); + } + + let state = f.client().settlement_status(&id); + assert!(state.failed); + assert_eq!(state.failed_at, f.env.ledger().sequence()); + // The value is still the payee's - Failed preserves the claim. + assert_eq!(f.client().claimable(&id), 3_000); + assert_eq!( + f.client().dead_letter_at(&id), + Some(state.failed_at + 120_960) + ); +} + +#[test] +fn test_a_payee_who_fixes_their_account_pulls_normally_and_clears_the_record() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &3_000); + + for _ in 0..3 { + f.client().report_failure(&id, &FailureReason::NoTrustline); + } + assert!(f.client().settlement_status(&id).failed); + + // Recovery is pull-based: the ordinary claim, once the account works. + f.client().clear_failure(&id); + assert_eq!(f.client().claim(&id), 3_000); + assert_eq!(f.token().balance(&f.payee), 3_000); + + let state = f.client().settlement_status(&id); + assert!(!state.failed); + assert_eq!(state.failures, 0); + assert_eq!(f.client().dead_letter_at(&id), None); +} + +#[test] +fn test_reporting_a_failure_needs_something_actually_owed() { + let f = setup(10_000); + let id = f.open(10_000); + + // Nothing released, so no settlement could have been attempted. + assert_eq!( + f.client().try_report_failure(&id, &FailureReason::Other), + Err(Ok(raised(Error::NothingToClaim))) + ); +} + +#[test] +fn test_only_the_payee_can_report_a_failure_on_their_own_escrow() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &3_000); + + let stranger = Address::generate(&f.env); + f.env.mock_auths(&[MockAuth { + address: &stranger, + invoke: &MockAuthInvoke { + contract: &f.router, + fn_name: "report_failure", + args: (id,).into_val(&f.env), + sub_invokes: &[], + }, + }]); + + assert!(f + .client() + .try_report_failure(&id, &FailureReason::Other) + .is_err()); + assert_eq!(f.client().settlement_status(&id).failures, 0); +} + +#[test] +fn test_the_dead_letter_path_is_shut_until_the_delay_elapses() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &3_000); + for _ in 0..3 { + f.client() + .report_failure(&id, &FailureReason::AssetRestricted); + } + + // The window exists for the payee's benefit: a frozen asset is fixable. + assert_eq!( + f.client().try_dead_letter(&id), + Err(Ok(raised(Error::DeadLetterNotReady))) + ); + assert_eq!(f.client().claimable(&id), 3_000); + + advance(&f, 120_959); + assert_eq!( + f.client().try_dead_letter(&id), + Err(Ok(raised(Error::DeadLetterNotReady))) + ); +} + +#[test] +fn test_dead_lettering_returns_the_value_to_the_payer_and_ends_the_claim() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &3_000); + for _ in 0..3 { + f.client() + .report_failure(&id, &FailureReason::NoDestination); + } + advance(&f, 120_960); + + assert_eq!(f.client().dead_letter(&id), 3_000); + + // Policy: back to the party that put it in - the only destination that + // needs no new trust assumption. Not burned, not kept by the contract. + assert_eq!(f.token().balance(&f.payer), 3_000); + assert_eq!(f.token().balance(&f.payee), 0); + assert_eq!(f.token().balance(&f.router), 7_000); + + // And the swept value is no longer claimable, so it cannot be paid twice. + assert!(f.client().settlement_status(&id).dead_lettered); + assert_eq!(f.client().claimable(&id), 0); + assert_eq!( + f.client().try_claim(&id), + Err(Ok(raised(Error::NothingToClaim))) + ); +} + +#[test] +fn test_a_dead_lettered_escrow_cannot_be_swept_or_reported_again() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &3_000); + for _ in 0..3 { + f.client().report_failure(&id, &FailureReason::Other); + } + advance(&f, 120_960); + f.client().dead_letter(&id); + + let payer_after = f.token().balance(&f.payer); + assert_eq!( + f.client().try_dead_letter(&id), + Err(Ok(raised(Error::AlreadyDeadLettered))) + ); + assert_eq!( + f.client().try_report_failure(&id, &FailureReason::Other), + Err(Ok(raised(Error::AlreadyDeadLettered))) + ); + assert_eq!(f.token().balance(&f.payer), payer_after); +} + +#[test] +fn test_dead_lettering_a_healthy_escrow_is_refused() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &3_000); + + assert_eq!( + f.client().try_dead_letter(&id), + Err(Ok(raised(Error::SettlementNotFailed))) + ); + assert_eq!( + f.client().try_clear_failure(&id), + Err(Ok(raised(Error::SettlementNotFailed))) + ); +} + +#[test] +fn test_the_dead_letter_sweep_is_the_arbiters_call_not_the_payers() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &3_000); + for _ in 0..3 { + f.client().report_failure(&id, &FailureReason::NoTrustline); + } + advance(&f, 120_960); + + // The payer is the beneficiary of the sweep, so letting them trigger it + // would reward griefing a payee into the failed state. + f.env.mock_auths(&[MockAuth { + address: &f.payer, + invoke: &MockAuthInvoke { + contract: &f.router, + fn_name: "dead_letter", + args: (id,).into_val(&f.env), + sub_invokes: &[], + }, + }]); + assert!(f.client().try_dead_letter(&id).is_err()); + assert_eq!(f.client().claimable(&id), 3_000); + + f.env.mock_all_auths(); + assert_eq!(f.client().dead_letter(&id), 3_000); +} + +#[test] +fn test_failure_handling_does_not_disturb_the_arbiters_dispute_path() { + let f = setup(10_000); + let id = f.open(10_000); + f.client().release_partial(&id, &2_000); + f.client().report_failure(&id, &FailureReason::NoTrustline); + + // A failing payout on the released portion is unrelated to a dispute over + // the remainder; both must still work. + f.client().dispute(&id, &5_000); + f.client().resolve_dispute(&id, &false); + assert_eq!(f.client().releasable(&id), 8_000); + assert_eq!(f.client().settlement_status(&id).failures, 1); +}