Skip to content

fix: persist reorg-safe cursor in Stellar metrics indexer #253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Bethelamos wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

fix: persist reorg-safe cursor in Stellar metrics indexer #253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e77d989
fix: [Wave 10] Persist a reorg-safe cursor in the Stellar metrics (#235)
Bethelamos Oct 17, 2026
a9c1c4f
fix(review): address maintainer feedback on #253
Bethelamos Oct 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 39 additions & 11 deletions stellar/scripts/metrics-indexer/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Set environment variables to configure the indexer:
export RPC_URL="https://futurenet.sorobanrpc.com"

# Network passphrase
export NETWORK_PASSPHRASE="Test SDF Future Network ; October 2022"
export NETWORK_PASSTHRASE="Test SDF Future Network ; October 2022"

# HTTP server port for Prometheus metrics
export PORT=9090
Expand All @@ -35,25 +35,30 @@ export RPC_TIMEOUT_MS=10000 # abort a single request after this long
export RPC_MAX_ATTEMPTS=4 # total attempts, including the first
export RPC_RETRY_BASE_MS=250 # jittered backoff base
export RPC_RETRY_MAX_MS=5000 # jittered backoff ceiling
export RPC_MAX_RESPONSE_BYTES=1048576 # hard cap on response bytes read
export REC_MAX_RESPONSE_BYTES=1048576 # hard cap on response bytes read

# Health thresholds
export HEALTH_STALE_AFTER_MS=15000 # default: 3 * POLL_INTERVAL_MS
export HEALTH_FAILURE_THRESHOLD=3 # consecutive total failures -> unhealthy

# Cursor persistence and reorg safety
export CURSOR_STATE_PATH="./.metrics-indexer-cursor.json" # last confirmed cursor
export CURSOR_REWIND_LEDGERS=1 # ledgers to rewind on startup for reorg safety
export CURSOR_MAX_EVENT_IDS=10000 # bounded dedupe window of replayed event ids

# Contract addresses to monitor
export STEALTH_REGISTRY_ADDRESS="CC..."
export STEALTH_SENDER_ADDRESS="CC..."
export STEALTH_BATCH_SENDER_ADDRESS="CC..."
export WRAITH_NAMES_ADDRESS="CC..."
export WRATH_NAMES_ADDRESS="CC..."
export STEALTH_SPLITTER_ADDRESS="CC..."
export STEALTH_VAULT_ADDRESS="CC..."
export GOVERNANCE_ADDRESS="CC..."
```

Any address left unset is skipped, so the indexer can be pointed at a subset of
the deployment. Leaving **every** address unset is a configuration error: the
poll fails and `/health` reports `degraded`/`unhealthy` instead of a healthy
poll fails and `/health` reports `degraded/`unhealthy` instead of a healthy
no-op.

## Usage
Expand Down Expand Up @@ -90,7 +95,7 @@ The health endpoint reports `healthy`, `degraded`, or `unhealthy`:
with no failures.
- `degraded` — a recent poll was stale or some contracts failed; the process is
still running. Responds `200`.
- `unhealthy` — `HEALTH_FAILURE_THRESHOLD` consecutive total failures (or no
- `unhealthy` — HEALTH_FAILURE_THRESHOLD consecutive total failures (or no
success yet after the stale window). Responds `503`.

```json
Expand All @@ -99,17 +104,35 @@ The health endpoint reports `healthy`, `degraded`, or `unhealthy`:
"stale": false,
"uptimeMs": 42000,
"pollIntervalMs": 5000,
"rpcHost": "https://futurenet.sorobanrpc.com",
"rpc@host": "https://futurenet.sorobanrpc.com",
"lastSuccessAt": 1699999999999,
"lastError": { "message": "RpcTimeoutError: RPC getEvents timed out after 10000ms", "at": 1700000000000 },
"lastError": {"message": "RpcTIMeoutError: RPC getEvents timed out after 10000ms", "at": 1700000000000},
"consecutiveFailures": 1,
"polls": { "total": 12, "failed": 3, "skipped": 0, "lastDurationMs": 82 }
"polls": {"total": 12, "failed": 3, "skipped": 0, "lastDurationMs": 82}
}
```

The response only ever carries the redacted RPC host (`scheme://host`) and a
truncated error summary — never the full URL, credentials, or a provider body.

## Cursor Persistence and Reorg Safety

The indexer persists the last confirmed cursor to `CURSOR_STATE_PATH` after
every successful poll, so a restart resumes from where it left off instead of
replaying from the default cursor.

- **Restart.** On startup the persisted cursor is loaded and used as the
`fetchContractEvents` cursor. If the file is missing, the default cursor
is used.
- **Reorg rewind.** Before fetching, the cursor ledger is rewound by
`CURSOR_REWIND_LEDGERS` ledgers so events from a short reorg are re-fetched.
- **Deduplication.** Replayed events are deduplicated by event id within a
bounded window of `CURSOR_MAX_EVENT_IDS` ids, so a rewind or replay never
double-counts a metric.
- **Corruption.** A missing, unreadable, or malformed state file is treated as
absent: the indexer logs a warning, falls back to the default cursor, and
rewrites a fresh state file on the next successful poll.

## Resilience

- **Abortable timeouts.** Each RPC request runs under an `AbortController` with
Expand Down Expand Up @@ -222,12 +245,11 @@ cap), scheduling and counters in `runtime.js` (`createPoller`,
The indexer expects events in the WraithMetricEvent format:

**Topic:** `(metric, contract_identifier, metric_name)`

**Data:** `(value, dimensions)`

Soroban `Symbol`s are capped at 9 characters, so contracts emit abbreviated
topics (`wr_names`, `res_miss`, ...). `index.js` maps them back to the canonical
names via `CONTRACT_SYMBOLS`, `METRIC_SYMBOLS`, and `DIMENSION_SYMBOLS`; keep
nomes via `CONTRACT_SYMBOLS`, `METRIC_SYMBOLS`, and `DIMENSION_SYMBOLS`; keep
those tables in sync with `stellar/wraith-metrics/src/lib.rs`.

See `stellar/METRICS.md` for the full specification, including the complete
Expand All @@ -243,6 +265,12 @@ npm test # node --test; covers timeout/abort, retry+jitter, response cap,
# prevention, health states, and redaction
```

The suite also covers cursor persistence across restarts, deduplication of
replayed events, and recovery from a corrupted cursor state file. The
integration test drives the real `getEvents` polling loop through a restart and
a reorg rewind, asserting the store is loaded, committed, rewound, and used to
deduplicate replayed events.

### Verify the dashboard against a synthetic event stream

`synthetic-stream.js` replays a hand-written WraithMetricEvent stream — one
Expand Down Expand Up @@ -274,7 +302,7 @@ Then invoke contract functions and observe metrics being updated at `/metrics`.
For production use:

1. Use a persistent storage backend (e.g., Redis, PostgreSQL) instead of in-memory storage
2. Implement event cursor persistence to avoid reprocessing events
2. Point `CURSOR_STATE_PATH` at durable storage (the cursor is persisted to a file by default)
3. Add authentication/authorization to the metrics endpoint
4. Configure appropriate retention policies in Prometheus
5. Set up alerts based on metric thresholds
Expand Down
Loading
Loading