Skip to content

fix(stellar): make service-worker event pagination and cursor advancement lossless - #228

Merged
truthixify merged 2 commits into
wraith-protocol:developfrom
gbengaeben:fix/221-lossless-sw-event-pagination
Oct 18, 2026
Merged

truthixify merged 2 commits into
wraith-protocol:developfrom
gbengaeben:fix/221-lossless-sw-event-pagination

Conversation

@gbengaeben

Copy link
Copy Markdown
Contributor

Closes #221

Problem

Both checked-in service-worker scan paths asked getEvents for a single
1,000-event page, ignored pagination.cursor, and then advanced the stored
cursor straight to the latest ledger. Neither checked HTTP status before reading
the body, and neither validated the JSON-RPC envelope or result shape.

Consequences:

  • A sync that hit the 1,000-event page cap marked the whole unread backlog as
    scanned and silently dropped every announcement past the first page.
  • A sync that hit a timeout, an HTTP error, or was simply killed mid-flight
    still jumped the cursor to the tip, so those events were never re-read.
  • Two competing workers registered at the same scope, each keeping its own
    cursor, so the app had two independent and disagreeing notions of "scanned".

Changes

One shared, lossless RPC implementation (src/lib/stellar/rpcEvents.ts)

  • Follows pagination.cursor until the committed boundary is reached rather
    than stopping at the first page. The 1,000-event cap is now a page size, never
    a scan size.
  • onPage is awaited after every page that was successfully fetched and
    handed to the caller, so committedLedger only ever covers events the caller
    has seen. An interrupted scan resumes from the first unread ledger instead of
    jumping to the tip.
  • Abortable, time-bounded requests (AbortController + per-request timeout),
    HTTP status checked before the body is read, and typed errors for timeouts,
    HTTP failures, malformed envelopes, result-shape violations and cursor stalls.
  • The cursor only advances over ledgers a page actually delivered, and a
    repeated cursor fails loudly instead of looping forever.

Announcement decoding split out (src/lib/stellar/announcementRpc.ts)

  • parseAnnouncementEvent now lives in one place instead of being duplicated in
    the page component and the worker. Undecodable payloads are counted and
    reported rather than aborting a scan.

Duplicate worker removed (src/sw/stellar-notification-sw.ts, deleted)

  • Its consumers attach to the single app worker via src/lib/swRegistration.ts,
    which reuses the existing registration instead of registering a second script
    at the same scope. One worker, one cursor.
  • src/sw/app-sw.ts and the vault flow read events through the shared module,
    so all four call sites share the same pagination, timeout and validation rules.

Retention handling

  • A first scan with no stored cursor is clamped to the edge of the retention
    window instead of asking for a ledger the RPC has already dropped.
  • A stored cursor that has fallen out of the window surfaces a retention gap to
    the page for an explicit recovery scan; the cursor is never advanced silently.

Tests

  • src/lib/stellar/rpcEvents.test.ts — 12 cases: pagination past the 1,000-event
    cap (2,500 events over 3 pages), resume-after-mid-scan-failure, abortable
    timeout, non-2xx rejection, envelope/result validation, malformed-entry
    dropping, retention gap mapping, first-scan clamping, committed-boundary stop,
    truncation reporting, cursor-stall detection, typed JSON-RPC errors.
  • src/workers/stellar-scanner.worker.test.ts — per-page PAGE progress past
    the 1,000-event cap via the real self.onmessage/postMessage contract, and
    cursor retention when a later page fails so the rescan resumes at the failed
    page.
  • e2e/scanner-retention.spec.ts — updated for the new per-page messages;
    verified passing on chromium against live testnet RPC.

Verification

  • pnpm exec tsc --noEmit — clean
  • pnpm exec prettier --check . — clean
  • pnpm exec vitest run — 344 passed; the 2 failures in
    src/store/nameWatchlistStore.test.ts are pre-existing on develop @ ae3dca5
    (verified against a clean tree) and are addressed separately by fix: lifecycle-manage name auction storage sync to prevent test failures #227
  • pnpm exec vite build — clean; the service worker bundles to 55 kB and does
    not pull the Stellar SDK into SW context
  • pnpm exec playwright test e2e/scanner-retention.spec.ts --project=chromium — passed

…ment lossless

Both checked-in service-worker scan paths asked getEvents for a single
1,000-event page, ignored pagination.cursor, and then advanced the stored
cursor straight to the latest ledger. A sync that hit the page cap, timed out,
or was killed mid-flight therefore marked the whole unread backlog as done and
silently dropped announcements. Neither path checked HTTP status before reading
the body or validated the JSON-RPC envelope.

Every event read now goes through one shared, tested implementation: it follows
pagination until the committed boundary, clamps a first scan to the retention
window, maps out-of-range requests to a typed retention gap, and awaits a
per-page callback so progress is persisted after each page that was actually
delivered. Requests carry an abortable timeout, and HTTP, envelope, result-shape
and cursor-stall failures surface as typed errors so callers can distinguish an
outage (keep the cursor) from a retention gap (recover it).

The obsolete stellar-notification-sw duplicate worker is removed; its consumers
attach to the single app worker, leaving one registration and one cursor.

Closes wraith-protocol#221
@drips-wave

drips-wave Bot commented Oct 16, 2026

Copy link
Copy Markdown

@gbengaeben Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@vercel

vercel Bot commented Oct 16, 2026

Copy link
Copy Markdown

@gbengaeben is attempting to deploy a commit to the truthixify's projects Team on Vercel.

A member of the Team first needs to authorize it.

@truthixify truthixify left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The per-page checkpoint is not lossless when an RPC page boundary splits one ledger. With a non-empty page cursor, the code commits pageMaxLedger, and consumers persist committedLedger + 1. If page 1 ends partway through ledger L and page 2 fails or the scan is interrupted, the restart begins at L+1 and permanently skips the remaining events from ledger L. The tests only page across monotonically increasing ledger ranges, so they miss this case. Please persist the opaque RPC cursor, or otherwise commit only ledgers proven fully consumed, and add a rejected-second-page regression where both pages contain events from the same ledger.

The per-page checkpoint committed the highest ledger a page contained,
but a getEvents page can stop partway through one ledger: the RPC cursor
resumes inside it on the next page. Consumers persist committedLedger + 1,
so when page 1 ended inside ledger L and page 2 then failed or the sync
was interrupted, the restart began at L + 1 and permanently skipped the
rest of L's events. The pagination tests only paged across monotonically
increasing ledger ranges, so none of them split a ledger.

Events arrive in ledger order, so with a cursor still outstanding every
ledger below the page's last one is proven complete while the last one is
not. Mid-scan commits now stop at pageMaxLedger - 1 and the boundary
ledger is re-read on resume; consumers already dedupe matches by stealth
address. This also removes a silent loss where the loop broke on first
contact with throughLedger and reported success with part of the boundary
ledger unread.

Adds a rejected-second-page regression with both pages carrying events
from the same ledger, a boundary-ledger test, and derives the worker's
resume position from the emitted PAGE checkpoint instead of a hardcoded
value.

Refs wraith-protocol#221

@truthixify truthixify left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new checkpoint rule preserves a partially consumed ledger by committing only through pageMaxLedger - 1 while a continuation cursor exists, and the boundary-ledger regressions cover both interruption and completion. Local RPC/worker suites (21 tests) and production build pass.

@truthixify
truthixify merged commit 8e74226 into wraith-protocol:develop Oct 18, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Wave 10] Make service-worker event pagination and cursor advancement lossless

2 participants