Repository navigation
fix: cross-replica rate limits, job tick locks, treasury sweep balance and large-sweep guard - #705
Merged
robertocarlous merged 4 commits intoOct 20, 2026
Conversation
rpc.Server.getAccount returns an Account with only a sequence number, so the old balances-array parse always yielded 0 and no sweep ever planned. Read the on-chain account entry instead and convert stroops to XLM. Closes Neurowealth#654
…lope The multisig envelope lifecycle stops at PENDING — nothing signs, assembles or enqueues it — so sweeps above requiresApprovalAbove sat in PLANNED with funds never moved. Refuse those amounts with a thrown error and a critical alert before any sweep row is written. Closes Neurowealth#655
Every pod ran every scheduled job on the same tick, so money-moving and destructive work depended on each job being idempotent by convention. Each tick now takes a pg_try_advisory_xact_lock keyed by job name — one winner, the rest skip — and every job is classified in docs/job-observability.md. Closes Neurowealth#636
MemoryStore counted per pod, so n replicas enforced n x the documented tier limits. Counters now live in Redis shared by every replica; NODE_ENV=test keeps the in-process store so unit tests never need Redis. Retry-After and RateLimit-* headers derive from the shared counter. Closes Neurowealth#635
|
@AbelOsaretin Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #635, Closes #636, Closes #654, Closes #655
Type of Change
Summary
Four multi-replica correctness bugs in the Stellar wave batch:
getNativeBalanceparsed abalancesarray thatrpc.Server.getAccountnever returns, so the treasury sweep always saw 0 and planned nothing.requiresApprovalAboveopened aMultisigEnvelopethat nothing ever advanced, stranding funds inPLANNED.express-rate-limitused the in-processMemoryStore, sonpods enforcedn xthe documented tier budgets.Motivation / Context
Closes #635, Closes #636, Closes #654, Closes #655.
Changes
src/stellar/client.tsgainsgetNativeBalance, reading the on-chain account entry (getAccountEntry, stroops → XLM).treasurySweep.tscalls it.runSweeprefuses amounts aboverequiresApprovalAbovewith a thrown error + critical alert before any sweep row is written, instead of opening a dead envelope.scheduleResilientJobtakes a per-tickpg_try_advisory_xact_lockkeyed by job name; losing replicas skip. All 22 jobs classified indocs/job-observability.md/src/jobs/jobClassification.ts.RedisRateLimitStorebacks every limiter with the shared Redis counter.NODE_ENV=testkeeps the in-process store.docs/RATE_LIMITING.mddocuments the model.Testing
tests/unit/stellar/nativeBalance.test.ts— realisticxdr.AccountEntryfixture; stroops → XLM.tests/unit/jobs/treasurySweep.test.ts— large sweeps now throw + alert; no envelope/row written.tests/unit/jobs/resilientScheduler.test.ts— two scheduler instances, one lock: single execution per tick; skipped tick when lock is held.tests/unit/jobs/jobClassification.test.ts— everyjobNameinsrc/jobs/is classified; no orphans.tests/unit/middleware/rateLimitStore.test.ts— two app instances, one Redis: combined count trips the limit;RateLimit-Remaining/Retry-Afterderive from the shared counter; prefixes isolate limiter types.Tradeoffs
executeEmergencySweep) still work and never readrequiresApprovalAbove.pg_try_advisory_xact_lockinsidedb.$transactionrather than session-scopedpg_advisory_lock, because Prisma returns pooled connections between calls and a session lock would leak. The transaction is held for the run (timeout 600s).REDIS_URLis unset. Crashing the API seemed worse than degraded enforcement in single-replica dev; the warning makes the multi-replica misconfiguration visible.Out of scope
MultisigEnvelopelifecycle (src/stellar/multisig.tsassembleTransactionis still the minimal stub).rollback.sqlfiles, referral/mail/secrets suites, etc.) that the localpre-pushhook surfaces. Not touched.