Skip to content

fix: cross-replica rate limits, job tick locks, treasury sweep balance and large-sweep guard - #705

Merged
robertocarlous merged 4 commits into
Neurowealth:mainfrom
AbelOsaretin:fix/stellar-wave-635-636-654-655
Oct 20, 2026
Merged

robertocarlous merged 4 commits into
Neurowealth:mainfrom
AbelOsaretin:fix/stellar-wave-635-636-654-655

Conversation

@AbelOsaretin

Copy link
Copy Markdown
Contributor

Closes #635, Closes #636, Closes #654, Closes #655

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring (no functional or behavioral changes)
  • Performance improvement
  • Documentation update
  • Build / CI configuration change
  • Dependency update
  • Other:

Summary

Four multi-replica correctness bugs in the Stellar wave batch:

  1. Treasury sweep reads a balances array that does not exist, so sweeps never plan #654 — getNativeBalance parsed a balances array that rpc.Server.getAccount never returns, so the treasury sweep always saw 0 and planned nothing.
  2. Large-amount treasury sweeps open a MultisigEnvelope that no code ever completes #655 — sweeps above requiresApprovalAbove opened a MultisigEnvelope that nothing ever advanced, stranding funds in PLANNED.
  3. Scheduled jobs run on every replica with no leader election or advisory-lock guard #636 — every replica ran every scheduled job on the same tick; money-moving jobs relied on per-job idempotency by convention.
  4. Rate limiter is per-pod in-memory: multi-replica deployments enforce N× the documented tier limits #635 — express-rate-limit used the in-process MemoryStore, so n pods enforced n x the documented tier budgets.

Motivation / Context

Closes #635, Closes #636, Closes #654, Closes #655.

Changes

Issue Change
#654 src/stellar/client.ts gains getNativeBalance, reading the on-chain account entry (getAccountEntry, stroops → XLM). treasurySweep.ts calls it.
#655 runSweep refuses amounts above requiresApprovalAbove with a thrown error + critical alert before any sweep row is written, instead of opening a dead envelope.
#636 scheduleResilientJob takes a per-tick pg_try_advisory_xact_lock keyed by job name; losing replicas skip. All 22 jobs classified in docs/job-observability.md / src/jobs/jobClassification.ts.
#635 New RedisRateLimitStore backs every limiter with the shared Redis counter. NODE_ENV=test keeps the in-process store. docs/RATE_LIMITING.md documents the model.

Testing

  • tests/unit/stellar/nativeBalance.test.ts — realistic xdr.AccountEntry fixture; stroops → XLM.
  • tests/unit/jobs/treasurySweep.test.ts — large sweeps now throw + alert; no envelope/row written.
  • tests/unit/jobs/resilientScheduler.test.ts — two scheduler instances, one lock: single execution per tick; skipped tick when lock is held.
  • tests/unit/jobs/jobClassification.test.ts — every jobName in src/jobs/ is classified; no orphans.
  • tests/unit/middleware/rateLimitStore.test.ts — two app instances, one Redis: combined count trips the limit; RateLimit-Remaining / Retry-After derive from the shared counter; prefixes isolate limiter types.
npx jest tests/unit/stellar/nativeBalance.test.ts tests/unit/jobs/treasurySweep.test.ts tests/unit/jobs/resilientScheduler.test.ts tests/unit/jobs/jobClassification.test.ts tests/unit/middleware/rateLimitStore.test.ts
# 5 suites, 25 tests passed

npx tsc --noEmit   # clean
npx eslint <touched files>   # clean

Tradeoffs

Out of scope

  • Implementing the MultisigEnvelope lifecycle (src/stellar/multisig.ts assembleTransaction is still the minimal stub).
  • Alerting when a tick is skipped on all pods (lock starvation signal) — proposed in Scheduled jobs run on every replica with no leader election or advisory-lock guard #636 but not in its acceptance criteria.
  • Pre-existing unrelated test failures in the repo (missing migration rollback.sql files, referral/mail/secrets suites, etc.) that the local pre-push hook surfaces. Not touched.

rpc.Server.getAccount returns an Account with only a sequence number, so
the old balances-array parse always yielded 0 and no sweep ever planned.
Read the on-chain account entry instead and convert stroops to XLM.

Closes Neurowealth#654
…lope

The multisig envelope lifecycle stops at PENDING — nothing signs, assembles
or enqueues it — so sweeps above requiresApprovalAbove sat in PLANNED with
funds never moved. Refuse those amounts with a thrown error and a critical
alert before any sweep row is written.

Closes Neurowealth#655
Every pod ran every scheduled job on the same tick, so money-moving and
destructive work depended on each job being idempotent by convention. Each
tick now takes a pg_try_advisory_xact_lock keyed by job name — one winner,
the rest skip — and every job is classified in docs/job-observability.md.

Closes Neurowealth#636
MemoryStore counted per pod, so n replicas enforced n x the documented
tier limits. Counters now live in Redis shared by every replica; NODE_ENV=test
keeps the in-process store so unit tests never need Redis. Retry-After and
RateLimit-* headers derive from the shared counter.

Closes Neurowealth#635
@drips-wave

drips-wave Bot commented Oct 20, 2026

Copy link
Copy Markdown

@AbelOsaretin Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment