fix: cache failover staleness, idempotency leases, outbox fencing, and heavy-route prefetch

[kingksjo]

fix: cache failover staleness, idempotency leases, outbox fencing, and heavy-route prefetch

Summary

This PR implements four related reliability/performance issues in a single branch:

• [backend] Redis fail-open can serve stale vault summary indefinitely #1553 — Redis fail-open can serve stale vault summary indefinitely
• [backend] Idempotency processing key can block retries (no pending-slot timeout) #1556 — Idempotency processing key can block retries (no pending-slot timeout)
• [backend] EventOutbox.lockedAt/lockedBy without lease expiry risks stuck relay #1557 — EventOutbox.lockedAt/lockedBy without lease expiry risks stuck relay
• [frontend] Extend route prefetch beyond 4 dashboard routes #1569 — Extend route prefetch beyond the four dashboard routes

It also repairs committed merge artifacts in backend/src/idempotency.ts and
backend/src/eventOutbox.ts (duplicate class/interface declarations and
undefined prisma references) so both modules load and compile again, and
aligns IdempotencyStore.pruneStaleKeys() with the dryRun result contract
already asserted by idempotencyRetention.test.ts.

Changes

#1553 — Cache fail-open staleness + bypass

backend/src/redisCache.ts

• Added getFromCacheDetailed() returning { entry, backend, stale, fromFallback, ageMs }.
  getFromCache() remains backward compatible.
• Fallback LRU hits while Redis is configured-but-down are reported as stale,
  with entry age derived from the stored TTL/expiry.
• New Prometheus counters:
  • redis_cache_fallback_total{route,reason}
  • redis_cache_stale_serve_total{route,backend}
  • redis_cache_bypass_total{route}

backend/src/middleware/cache.ts

• Emits X-Cache-Stale, X-Cache-Backend, X-Cache-Age, and
  Warning: 110 - "Response is stale" on fallback hits.
• Fallback responses are sent with Cache-Control: no-store so a CDN/browser
  cannot pin a stale vault summary indefinitely.
• Added a force-refresh path: X-Cache-Bypass: true (or
  Cache-Control: no-cache / no-store) skips the lookup and repopulates the
  entry; the response carries X-Cache-Bypass: true.
• The same headers/bypass semantics apply to the in-memory-only path.

Test: backend/src/__tests__/cacheFailover.test.ts covers stale metadata,
headers, metrics, bypass detection, and cache repopulation.

#1556 — Idempotency processing lease + safe reclaim

backend/src/idempotency.ts

• Added a processing lease (IDEMPOTENCY_PROCESSING_LEASE_MS, default
  2 minutes) independent of the 24h replay-retention TTL. The TTL semantics are
  documented in the module header: retention governs how long completed/failed
  responses can be replayed; the lease governs how long a pending record may
  block retries.
• isProcessingLeaseExpired() and processingLeaseRemainingSeconds() helpers.
• reclaimExpiredIdempotencyRecord() performs a guarded delete
  (status: 'pending' + createdAt <= cutoff) so a concurrent completion is
  never clobbered; enforceIdempotency() reclaims an expired slot and allows
  the retry.
• reclaimExpiredIdempotencyRecords() bulk reaper and
  startIdempotencyLeaseReaper() for periodic sweeps.
• Pending 409 responses now return the actual lease-derived retryAfter.

Test: backend/src/__tests__/idempotencyPendingLease.test.ts covers expired
lease recovery, live-lease blocking, guarded-delete races, and completed replay.

#1557 — Outbox lease expiry, fencing, and stuck-relay reaper

backend/src/eventOutbox.ts

• lockedAt/lockedBy are explicitly documented and used as a lease
  (OUTBOX_LOCK_TIMEOUT_MS, default 60s).
• Added a fencing token (buildOutboxLeaseToken, surfaced as
  EventOutboxRecord.leaseToken) built from the owner and claim timestamp.
  Relay/failure/dead-letter writes now use a conditional updateMany on the
  exact (lockedBy, lockedAt) pair, so a worker whose lease was re-claimed
  cannot overwrite the newer attempt.
• Added reapStuckRelays(), which releases expired leases. It runs at the start
  of every relay cycle and on its own timer (OUTBOX_REAPER_INTERVAL_MS,
  default half the lease).
• processOutbox() claims rows with a status + expiry guard and stopped writing
  completion state through unfenced update() calls.

Test: backend/src/__tests__/eventOutboxLease.test.ts covers lease configuration,
fencing-token derivation, expired-lease reaping, crash recovery on the next
relay cycle, and mid-delivery re-claim fencing.

#1569 — Route prefetch coverage + LCP measurement

frontend/src/lib/routePrefetch.ts

• Added /vault-health to routeImports and LazyVaultHealth.
• Added HEAVY_ROUTES (/compare, /vault-health, /settings) to the idle
  prefetch via prefetchDashboardRoutes(); added prefetchHeavyRoutes().
• getRoutePrefetchHandlers() now warms on onMouseEnter, onPointerEnter,
  onFocus, and onTouchStart, and failed prefetches are retryable.
• Added LCP observation (observeLcp()) and navigation sampling
  (recordRouteNavigation() / getRoutePrefetchMetrics()) so warm vs cold
  navigations can be compared in the field.

frontend/src/App.tsx

• Prefetches the heavy routes in the existing idle callback and records each
  navigation's prefetch/LCP status.
• Uses LazyVaultHealth and removes duplicate route declarations.

frontend/src/components/Navbar.tsx + i18n locales

• Wired prefetch handlers to every route link (including /compare, /admin,
  /vault-health, /settings) in desktop, mobile, and dropdown menus.
• Added nav.vaultHealth / nav.settings translations (en/es).

Test: frontend/src/lib/routePrefetch.test.ts covers heavy-route coverage,
handler exposure, and navigation metric recording.

Test/verification notes

No build was run per the task instructions. New tests were added for every
acceptance criterion and the existing suites were reviewed for API
compatibility:

• getFromCache() keeps its original signature/return shape.
• IdempotencyStore and idempotencyStore remain importable from
  idempotency.ts via re-export.
• pruneStaleKeys() now returns the dryRun flag expected by
  idempotencyRetention.test.ts.
• EventOutboxRecord keeps all prior fields (and additionally exposes
  vaultId/leaseToken); getMetrics() keeps its exact six-field shape.
• Existing outbox model-guard, ordering, abort, and metrics tests remain
  compatible.

Closes #1553
Closes #1556
Closes #1557
Closes #1569

[drips-wave]

@kingksjo Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits