feat(ingest): bound every ingester cycle with a deadline, abandoned-cycle metric, and overlap-proof abandonment (#351)

[felixumeh]

Closes #351

Summary of Changes

Bounds every ingester cycle with a deadline so one slow upstream can no longer stall an ingester while every health signal still reports the service as healthy. Issues below are fixed by this PR:

• Closes Ingester loops have no upper bound on a single cycle #351: Each round of an ingester loop (SDEX, AMM, Soroswap, Aquarius venue adapters, plus the 1-minute snapshot appender) now runs under runCycleWithDeadline() in the new src/ingest/cycleDeadline.ts. A cycle that exceeds its deadline is aborted at its next unit checkpoint, counted in the new ingester_abandoned_cycles_total metric labelled venue + network, logged, and abandoned — the loop resumes after teardown of work already in flight instead of awaiting the abandoned cycle's remaining work.

All five acceptance criteria from the issue:

• A cycle that exceeds its deadline is abandoned, not awaited indefinitely — the cycle races a timer; on expiry its AbortSignal fires and every checkpoint stops further pairs/pools from starting, so the abandoned cycle stops at its in-flight unit rather than grinding through the rest of the round.
• The deadline is configurable per ingester — INGEST_CYCLE_DEADLINE_MS with per-venue (INGEST_CYCLE_DEADLINE_MS_SOROSWAP) and per-venue-per-network (INGEST_CYCLE_DEADLINE_MS_SOROSWAP_MAINNET) overrides, precedence documented in getCycleDeadlineMs() and .env.example; default 60s, 0 explicitly disables the deadline. Follows the paired-var convention used everywhere else in src/config.ts.
• Abandoned cycles are counted in a metric — ingester_abandoned_cycles_total{venue,network} (prom-client Counter in src/metrics.ts); cardinality is venues x networks, bounded by the adapter registry, documented in docs/ingest-metrics.md alongside the existing metrics, with a promql query example.
• No partial write survives an abandoned cycle — and, per the issue's "check how each ingester writes before changing the control flow", the checkpoints were placed to match the existing write units: upsertPricePoints is one createMany(skipDuplicates); the SDEX/AMM page upsert and its cursor advance are checked before the sequence as a whole, never between the two statements; snapshotPool, Soroswap's per-pool write and the snapshot appender are each either skipped whole or run whole. Skipping costs only a refetch from the same cursor, and skipDuplicates makes the replay idempotent.
• A test with a slow upstream proves the loop keeps making progress — src/__tests__/ingestCycleDeadline.test.ts drives the real startVenueAdapter loop with a 250ms upstream under a 50ms deadline and asserts ≥3 cycles start, maxActive === 1 (no two cycles ever overlap), and ≥2 abandonments land on the counter with venue/network labels.

Guarantees in detail

• Overlap is impossible, structurally: runCycleWithDeadline() never resolves while the previous cycle's promise is unsettled — abandonment awaits unwind, not work. The unwind is prompt because the signal is passed to fetchWithTimeout({ signal }) (the house pattern from src/fetch.ts/src/alerts/delivery.ts) and checked at every unit boundary. If a single in-flight call ignores even its own timeout, the loop still refuses to start an overlapping cycle and escalates with a loud [cycle-deadline] … still has not unwound error after a 10s grace (CYCLE_UNWIND_GRACE_MS), rather than silently double-running a venue.
• Failures keep their backoff unchanged: cycle errors are rethrown untouched after clearTimeout, so startVenueAdapter's exponential backoff and restartIngester fault isolation behave exactly as before; only deadline expiry counts as an abandonment (metric, no error backoff), so a slow venue is not also punished with retry delay.
• Adapter contract stays backward compatible: ingestPair(pair, network, signal?) adds an optional third parameter; all four adapters, both writeVenuePricePoints and the snapshot loop thread it through; existing custom adapters and tests compile and run unmodified.
• Testability without touching production semantics: startVenueAdapter(adapter, network, { stopSignal }) lets a loop stop at a cycle boundary — used by the new test; unset in production where loops legitimately run until process exit.

Files changed

File	Change
src/ingest/cycleDeadline.ts	New — runCycleWithDeadline(), getCycleDeadlineMs() precedence chain, CycleDeadlineExceeded, grace-window logging
src/metrics.ts	New ingester_abandoned_cycles_total counter (venue, network)
src/ingest/venues/runner.ts	Cycle wrapped in the deadline; per-pair abandonment checkpoint; writeVenuePricePoints pre-write checkpoint; VenueAdapterLoopOptions
src/ingest/venues/types.ts	ingestPair gains optional signal
src/ingest/venues/{sdex,amm,soroswap,aquarius}.ts	Signal threaded from ingestPair → outbound fetches (fetchWithTimeout({ signal })) and checkpoints before each write unit
src/ingesters/snapshot.ts	Minute-append wrapped in the deadline (sleep excluded; single atomic INSERT … ON CONFLICT)
.env.example	INGEST_CYCLE_DEADLINE_MS[_<NETWORK>] documented (drift guard)
docs/ingest-metrics.md	Metric table, label table, cardinality bounds, query example
.changeset/ingest-cycle-deadline.md	Minor bumpsheet entry
src/__tests__/ingestCycleDeadline.test.ts	New — 8 tests: slow-upstream progress, no-partial-write, precedence/invalid/0-disables, error propagation

The branch also carries one pre-existing test-hygiene commit (test: reset upsertPricePoints mock between soroswap ingester cases) — clearAllMocks retains implementations, so a rejection stub could leak into the next test under a shuffled run.

Verification (all run locally, mirroring .github/workflows/ci.yml)

• npm ci → up to date, exit 0
• prisma generate → exit 0
• npx tsc --noEmit → exit 0, zero errors
• npm run build → exit 0 (emits dist/)
• npm test → exit 0 — 67 files passed, 673 tests passed, 1 skipped (includes the 8 new Ingester loops have no upper bound on a single cycle #351 tests; .env.example drift guard and OpenAPI sync green)
• Targeted run of ingestCycleDeadline.test.ts → 8/8 pass with live [cycle-deadline] … abandoned after exceeding its 50ms deadline logs proving the behavior end-to-end

Notes for reviewers

• Overlap with feat(ingest): add schema verification, cycle deadlines, lag telemetry, and /status health breakdown #463: that open PR also proposes cycle deadlines (among schema verification and lag telemetry). This PR implements Ingester loops have no upper bound on a single cycle #351 standalone against current main; happy to reconcile whichever direction you prefer — the runCycleWithDeadline() seam is small and would slot under whatever scheduling feat(ingest): add schema verification, cycle deadlines, lag telemetry, and /status health breakdown #463 lands.
• Known-unfixed (deliberately, out of Ingester loops have no upper bound on a single cycle #351's scope): the stellar-sdk Horizon call itself cannot be interrupted mid-round-trip — the timeout there belongs to fetchWithTimeout-style per-call deadlines (fix: enforce deadline timeouts on all outbound fetch calls #379); the cycle deadline ensures we refuse to begin work for an abandoned cycle and to overlap cycles regardless.
• The k6 prices load test is already failing on main (per the issue text) and is not part of PR CI; unaffected here.
• npm test includes one intentionally skipped integration e2e test (tests/integration/e2e.test.ts), pre-existing and unrelated.

[drips-wave]

@felixumeh Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits