Skip to content

feat:alerting for stuck entries and a fee-bump ceiling - #706

Merged
robertocarlous merged 1 commit into
Neurowealth:mainfrom
bibi-fay:feat/alerting-for-stuck-entries-and-a-fee-bump-ceiling
Oct 20, 2026
Merged

robertocarlous merged 1 commit into
Neurowealth:mainfrom
bibi-fay:feat/alerting-for-stuck-entries-and-a-fee-bump-ceiling

Conversation

@bibi-fay

Copy link
Copy Markdown
Contributor

Summary

Adds the transactional outbox for on-chain operations (#325). Business state and the intent to submit to Stellar are now written in the same DB transaction. A prioritized dispatcher then turns each intent into a submission, so a crash or failed submit no longer loses the operation.

Changes

  • src/outbox/service.ts: DB-facing operations: idempotent enqueue (by idempotencyKey), atomic claim (PENDING -> SUBMITTED via conditional updateMany), retry/terminal transitions, admin force-retry and cancel, queue stats, and the txHash reconciliation used by the events oracle.
  • src/outbox/dispatcher.ts: dispatchOne for synchronous callers (deposit/withdraw, referral payout), dispatchInBackground for fire-and-forget ops, and runDispatchSweep for crash recovery and retries. All paths converge on submitClaimedOp.
  • src/outbox/stateMachine.ts: pure transition rules, priority ordering (CRITICAL > NORMAL > LOW, FIFO within a tier), and full-jitter backoff.
  • src/outbox/executors.ts: exports resolveSignerPublicKey and executeOutboxPayload, which the dispatcher imports. These were declared but not exported, which broke compilation across many test suites.
  • Added explanatory comments throughout the outbox modules. No behavior change from the comments.

Behavior notes

  • Retries: transient errors return the op to PENDING with backoff. Once maxAttempts is reached, it moves to FAILED, which raises a critical alert and publishes outbox.op_failed.
  • On-chain rejection: a resolved status: 'failed' is terminal and not retried, matching the pre-Durable Outbox & Prioritized On-Chain Transaction Queue #325 deposit/withdraw behavior.
  • Fees: the multiplier compounds per attempt up to feeBumpMaxAttempts, clamped at maxAbsFee. CRITICAL ops use the aggressive base fee under congestion.
  • Congestion: LOW ops are deferred during high/severe congestion, bounded by lowMaxDeferMs.
  • Stuck ops: SUBMITTED ops past the timeout are checked against the chain first, then requeued with a fee bump or failed.
  • Concurrency: submissions are serialized per signer through signerLock.
  • Freeze guard: isUserHalted blocks dispatch for inactive or frozen users.

Testing

  • npx tsc --noEmit passes
  • tests/unit/outbox/stateMachine.test.ts
  • tests/integration/outbox/dispatcher.integration.test.ts
  • Full suite run in CI

Known follow-ups

  • findClaimableOps applies take: limit without an ORDER BY, so priority sorting only reorders the fetched batch. With a backlog larger than batchSize, a CRITICAL op can be left out of a sweep.
  • The doc comment on computeBackoffMs says base * 2^attempt, but the code uses 2^(attempt - 1).

Closes #588
Closes #703
Closes #704
Closes #702

@drips-wave

drips-wave Bot commented Oct 20, 2026

Copy link
Copy Markdown

@bibi-fay Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@robertocarlous
robertocarlous merged commit 6d7bc4f into Neurowealth:main Oct 20, 2026
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants