Repository navigation
fix(stellar): make service-worker event pagination and cursor advancement lossless - #228
Conversation
…ment lossless Both checked-in service-worker scan paths asked getEvents for a single 1,000-event page, ignored pagination.cursor, and then advanced the stored cursor straight to the latest ledger. A sync that hit the page cap, timed out, or was killed mid-flight therefore marked the whole unread backlog as done and silently dropped announcements. Neither path checked HTTP status before reading the body or validated the JSON-RPC envelope. Every event read now goes through one shared, tested implementation: it follows pagination until the committed boundary, clamps a first scan to the retention window, maps out-of-range requests to a typed retention gap, and awaits a per-page callback so progress is persisted after each page that was actually delivered. Requests carry an abortable timeout, and HTTP, envelope, result-shape and cursor-stall failures surface as typed errors so callers can distinguish an outage (keep the cursor) from a retention gap (recover it). The obsolete stellar-notification-sw duplicate worker is removed; its consumers attach to the single app worker, leaving one registration and one cursor. Closes wraith-protocol#221
|
@gbengaeben Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
|
@gbengaeben is attempting to deploy a commit to the truthixify's projects Team on Vercel. A member of the Team first needs to authorize it. |
There was a problem hiding this comment.
The per-page checkpoint is not lossless when an RPC page boundary splits one ledger. With a non-empty page cursor, the code commits pageMaxLedger, and consumers persist committedLedger + 1. If page 1 ends partway through ledger L and page 2 fails or the scan is interrupted, the restart begins at L+1 and permanently skips the remaining events from ledger L. The tests only page across monotonically increasing ledger ranges, so they miss this case. Please persist the opaque RPC cursor, or otherwise commit only ledgers proven fully consumed, and add a rejected-second-page regression where both pages contain events from the same ledger.
The per-page checkpoint committed the highest ledger a page contained, but a getEvents page can stop partway through one ledger: the RPC cursor resumes inside it on the next page. Consumers persist committedLedger + 1, so when page 1 ended inside ledger L and page 2 then failed or the sync was interrupted, the restart began at L + 1 and permanently skipped the rest of L's events. The pagination tests only paged across monotonically increasing ledger ranges, so none of them split a ledger. Events arrive in ledger order, so with a cursor still outstanding every ledger below the page's last one is proven complete while the last one is not. Mid-scan commits now stop at pageMaxLedger - 1 and the boundary ledger is re-read on resume; consumers already dedupe matches by stealth address. This also removes a silent loss where the loop broke on first contact with throughLedger and reported success with part of the boundary ledger unread. Adds a rejected-second-page regression with both pages carrying events from the same ledger, a boundary-ledger test, and derives the worker's resume position from the emitted PAGE checkpoint instead of a hardcoded value. Refs wraith-protocol#221
truthixify
left a comment
There was a problem hiding this comment.
The new checkpoint rule preserves a partially consumed ledger by committing only through pageMaxLedger - 1 while a continuation cursor exists, and the boundary-ledger regressions cover both interruption and completion. Local RPC/worker suites (21 tests) and production build pass.
Closes #221
Problem
Both checked-in service-worker scan paths asked
getEventsfor a single1,000-event page, ignored
pagination.cursor, and then advanced the storedcursor straight to the latest ledger. Neither checked HTTP status before reading
the body, and neither validated the JSON-RPC envelope or result shape.
Consequences:
scanned and silently dropped every announcement past the first page.
still jumped the cursor to the tip, so those events were never re-read.
cursor, so the app had two independent and disagreeing notions of "scanned".
Changes
One shared, lossless RPC implementation (
src/lib/stellar/rpcEvents.ts)pagination.cursoruntil the committed boundary is reached ratherthan stopping at the first page. The 1,000-event cap is now a page size, never
a scan size.
onPageis awaited after every page that was successfully fetched andhanded to the caller, so
committedLedgeronly ever covers events the callerhas seen. An interrupted scan resumes from the first unread ledger instead of
jumping to the tip.
AbortController+ per-request timeout),HTTP status checked before the body is read, and typed errors for timeouts,
HTTP failures, malformed envelopes, result-shape violations and cursor stalls.
repeated cursor fails loudly instead of looping forever.
Announcement decoding split out (
src/lib/stellar/announcementRpc.ts)parseAnnouncementEventnow lives in one place instead of being duplicated inthe page component and the worker. Undecodable payloads are counted and
reported rather than aborting a scan.
Duplicate worker removed (
src/sw/stellar-notification-sw.ts, deleted)src/lib/swRegistration.ts,which reuses the existing registration instead of registering a second script
at the same scope. One worker, one cursor.
src/sw/app-sw.tsand the vault flow read events through the shared module,so all four call sites share the same pagination, timeout and validation rules.
Retention handling
window instead of asking for a ledger the RPC has already dropped.
the page for an explicit recovery scan; the cursor is never advanced silently.
Tests
src/lib/stellar/rpcEvents.test.ts— 12 cases: pagination past the 1,000-eventcap (2,500 events over 3 pages), resume-after-mid-scan-failure, abortable
timeout, non-2xx rejection, envelope/result validation, malformed-entry
dropping, retention gap mapping, first-scan clamping, committed-boundary stop,
truncation reporting, cursor-stall detection, typed JSON-RPC errors.
src/workers/stellar-scanner.worker.test.ts— per-pagePAGEprogress pastthe 1,000-event cap via the real
self.onmessage/postMessagecontract, andcursor retention when a later page fails so the rescan resumes at the failed
page.
e2e/scanner-retention.spec.ts— updated for the new per-page messages;verified passing on chromium against live testnet RPC.
Verification
pnpm exec tsc --noEmit— cleanpnpm exec prettier --check .— cleanpnpm exec vitest run— 344 passed; the 2 failures insrc/store/nameWatchlistStore.test.tsare pre-existing ondevelop@ae3dca5(verified against a clean tree) and are addressed separately by fix: lifecycle-manage name auction storage sync to prevent test failures #227
pnpm exec vite build— clean; the service worker bundles to 55 kB and doesnot pull the Stellar SDK into SW context
pnpm exec playwright test e2e/scanner-retention.spec.ts --project=chromium— passed