[chideraisiguzor]

Closes #289
Closes #290
Closes #298
Closes #299

Issues

#289 Add Alert Deduplication, Grouping, and On-Call Escalation Routing for Oracle Incidents

Description:
Add an incident routing layer in front of oracle alerting that fingerprints, deduplicates, groups, escalates, and auto-resolves alerts, so a flapping feed produces one actionable incident rather than hundreds of pages.

Problem Statement & Context:
Staleness, divergence, and quorum metrics with alerting are covered, but raw threshold alerts without routing are self-defeating: a single provider outage trips dozens of rules across many feeds, on-call mutes the channel, and the real divergence signal is lost in the noise. The practical failure mode is not the absence of an alert — it is alert fatigue trained into the humans who are the last line of defense for payout decisions.

Scope & Acceptance Criteria:

• Alert fingerprinting (feed ID + alert class + source, excluding volatile fields), a dedup window, and grouping into a single incident per feed outage.
• Severity model with routing: page for payout-triggering divergence, ticket for degraded non-payout feeds.
• Escalation policy with acknowledgement tracking, re-notification, and escalation to a secondary on-call after a configured delay.
• Automatic resolution when the underlying condition clears, with an incident timeline written to the audit log.
• Out of scope: vendor selection for paging (delivery must be pluggable via a notifier interface) and the metrics themselves.

Implementation Guidelines:

1. Key Files/Modules: a new alerting/routing module under src/services/ or src/observability/, wired to the metrics and anomaly detection emitters, config in .env.example, and notifier adapters.
2. Design/Architecture: an emit → dedupe/group → route → escalate pipeline with an explicit state machine per incident, persisted so a restart does not re-page or lose an open incident.
3. Edge Cases/Constraints: alert storms from a single provider, incidents spanning a restart, duplicate emits from multiple worker processes, and the requirement that routing failure never blocks the ingestion or settlement path.
4. Testing: dedup and grouping tests with a fake clock, escalation timing tests, auto-resolve tests, and a storm test asserting bounded notification volume for N failing feeds.

Definition of "Done":

• Code written, tested, and documented.
• All acceptance criteria met.
• PR submitted with a clear description, passing CI, relevant screenshots/test output.
• Code reviewed and approved.

Complexity: Medium (130 points)

#290 Add Dead-Man's-Switch Meta-Monitoring for the Oracle Pipeline and Alerting Stack

Description:
Emit a periodic heartbeat from the oracle pipeline to an external monitor and add a local watchdog that verifies metric and alert freshness, so a total pipeline failure — dead process, stalled scheduler, unreachable database, or broken alerting — produces an alert from outside the failing system.

Problem Statement & Context:
Every existing alert is generated by the same process whose failure is the most likely disaster. If src/index.ts dies, the scheduler stops, the metrics endpoint stops being scraped, or the alerting integration breaks, the system goes silent — and silence looks exactly like a healthy quiet feed. For a service that authorizes payouts, "no news" must be proven, not assumed.

Scope & Acceptance Criteria:

• A heartbeat emitted to an external dead-man's-switch endpoint on a documented cadence, sent only after a successful poll cycle and successful DB write, not merely because the event loop is alive.
• Heartbeat payload includes feed coverage counts and last successful publish timestamp so a stalled-but-alive pipeline is detectable remotely.
• A local watchdog that checks metrics-scrape freshness and alert-delivery success and raises a high-severity alert when they lapse.
• Documented expected cadence, grace period, and runbook entry for "heartbeat missing".
• Out of scope: vendor selection for the external monitor; delivery must be configurable via env and disableable for local dev.

Implementation Guidelines:

1. Key Files/Modules: src/index.ts service loop, src/services/oracleMonitor.ts cycle completion hook, a new heartbeat module, .env.example, and docs/ runbook updates.
2. Design/Architecture: heartbeat is driven by the completed-work signal rather than a standalone timer, so a wedged loop that never finishes a cycle stops heartbeating by construction.
3. Edge Cases/Constraints: transient network failure to the heartbeat receiver must not page (require N consecutive misses), heartbeat endpoints must not receive secrets, and the feature must not create load proportional to feed count.
4. Testing: simulate a stalled cycle and assert no heartbeat is sent; simulate an unreachable receiver and assert grace-period behavior; unit test the watchdog's staleness thresholds.

Definition of "Done":

• Code written, tested, and documented.
• All acceptance criteria met.
• PR submitted with a clear description, passing CI, relevant screenshots/test output.
• Code reviewed and approved.

Complexity: Low (70 points)

#298 Support Versioned Composite and Derived Feeds Built from Upstream Feeds

Description:
Add support for composite feeds whose value is computed deterministically from one or more upstream consensus feeds via a declared, versioned formula (for example a TVL ratio or a cross-market spread used as a payout condition). Composite feeds participate in the same scheduling, evidence, anomaly, and trust machinery as first-class feeds.

Problem Statement & Context:
Several coverage conditions are naturally expressed over more than one primitive — a ratio between two TVL series, a spread between two price feeds, or a threshold on an aggregate. Today that logic would have to live in ad-hoc code in the claim processor, where it is untested, undocumented, and invisible to the evidence and trust layers. Making derived values first-class keeps every payout-triggering number auditable, reproducible, and subject to the same corroboration and circuit-breaker rules as raw readings.

Scope & Acceptance Criteria:

• Define a composite feed definition schema: input feed ids, a deterministic expression/formula, unit and rounding policy, output feed id, and an explicit version.
• Evaluate composite feeds only over upstream consensus outputs (never over raw single-source readings), with declared freshness requirements on each input.
• Emit a consensus-quality decision and evidence record for every composite evaluation that records the formula version, input decision ids, and computed value.
• Recompute deterministically: identical inputs and formula version must always produce the identical output, byte-for-byte.
• Support dependency-cycle detection and evaluation ordering (topological) across composites of composites.
• Out of scope: a user-authored expression language UI, and re-deriving historical composite values for periods before the feature ships (historical bootstrap is handled by its own issue).

Implementation Guidelines:

1. Key Files/Modules: the feed/coverage catalog in src/common/coverage-catalog.ts, the oracle encoding helpers in src/oracle/oracle-encoding.ts, the feed definition and consensus modules behind the adapter registry, and src/db/schema.sql plus a migration for composite feed definitions and their decision/evidence linkage.
2. Design/Architecture: represent formulas as a small, versioned, non-Turing-complete AST evaluated by a pure function with an explicit numeric type (fixed decimal) so results are reproducible across nodes and replays; treat composite evaluation as just another producer in the pipeline so scheduling, backpressure, and evidence capture are reused; resolve evaluation order from a dependency graph built at config load.
3. Edge Cases/Constraints: reject cycles, self-reference, and undefined units at config load; define behavior when an input is stale, quarantined, or in degraded trust mode (composite inherits the weakest required input state); guard against divide-by-zero, overflow, and NaN propagation; ensure the formula version is stored with every decision so replay uses the version in force at the time.
4. Testing: property-based tests that evaluation is deterministic and version-stable; table-driven tests for unit conversions, rounding, division by zero, and overflow; graph tests for cycles and ordering; consensus tests asserting a composite goes degraded/suspended when a required input degrades; replay test reproducing composite outputs from recorded input decisions.

Definition of "Done":

• Code written, tested, and documented.
• All acceptance criteria met.
• PR submitted with a clear description, passing CI, relevant screenshots/test output.
• Code reviewed and approved.

Resources:
None — no existing design docs or external references to link.

Complexity: High (120 points)

#299 Preview Payout Impact of a Proposed Manual Oracle Override Before Applying It

Description:
Add a dry-run impact preview for manual oracle overrides: before an operator forces or suppresses a payout-triggering value, the system reports which active policies would trigger, the total and per-policy payout exposure, and which decisions would become inconsistent with existing evidence. The override is applied only after the operator confirms against that preview.

Problem Statement & Context:
Manual override is a necessary safety valve when a feed is wrong or a source is compromised, but today an operator can only set a value and observe the consequences afterwards. A single mistyped threshold or a value that flips a large batch of policies can settle payouts that were never justified, and unwinding them requires the compensation workflow plus a dispute. Showing blast radius before the action turns override from a blind action into a reviewed decision and produces the artifact needed for later audit.

Scope & Acceptance Criteria:

• Given a proposed override (feed id, value or suppressed state, effective window), compute a dry-run preview listing affected active policies, whether each would trigger under current and proposed conditions, and aggregate payout exposure.
• The preview must be non-mutating: it writes no decisions, pays nothing, and does not alter feed state.
• Applying an override must require the preview token/id from a preview generated within a short validity window, so the applied action matches what was reviewed.
• Every preview and its eventual application (or abandonment) is recorded in the evidence ledger with actor, rationale, parameters, and preview id.
• Enforce authorization for who may preview versus apply, and reject overrides exceeding a configurable exposure ceiling without an additional approval.
• Out of scope: the multi-signature approval workflow for policy changes, and the compensation/remediation path for already-settled claims.

Implementation Guidelines:

1. Key Files/Modules: the operator CLI under scripts/, the operator/admin route module under src/routes/, src/services/claimProcessor.ts (policy evaluation must be refactored into a pure, callable trigger-evaluation function reused by the preview), the evidence-ledger repository, and src/db/schema.sql plus a migration for override previews and their application linkage.
2. Design/Architecture: extract the trigger evaluation into a side-effect-free function parameterized by feed state so the preview runs the real logic rather than a parallel approximation; compute the preview against a consistent snapshot (single read transaction) and persist it with a hash of its inputs; store the applied override keyed to the preview id so mismatches are detectable.
3. Edge Cases/Constraints: previews must be cheap enough to run interactively — bound the policy scan and report truncation clearly; handle policies created or modified between preview and apply (invalidate the preview rather than silently applying); ensure the preview path cannot be repurposed to settle claims; the exposure ceiling check must be enforced server-side, not in the CLI only.
4. Testing: tests asserting the preview performs zero writes; equivalence test showing preview-triggered set matches actual post-apply triggered set for the same inputs; tests for expired preview tokens, mutated policy sets between preview and apply, and exposure-ceiling rejection; authorization tests for preview vs apply roles; evidence test confirming preview and application records are both present and linked.

Definition of "Done":

• Code written, tested, and documented.
• All acceptance criteria met.
• PR submitted with a clear description, passing CI, relevant screenshots/test output.
• Code reviewed and approved.

Resources:
None — no existing design docs or external references to link.

Complexity: Medium (65 points)

Summary

Each issue is independent and can land as its own PR. #289 and #290 both touch .env.example, configuration.ts, and env.schema.ts; coordinate edits to avoid merge conflicts. #298 and #299 both add tables to schema.sql and the single existing migration file; if the project prefers one migration per feature, split into new migration files instead of appending to 001. The alert routing layer must never block ingestion or settlement (fire-and-forget with bounded queues), and the heartbeat must be driven by the completed-work signal rather than a timer. Composite formulas must use the existing 1e7 fixed-point BigInt helpers, never floats, and store the formula version with every decision. The override preview must reuse the extracted pure trigger-evaluation function and run inside a single read transaction; apply must revalidate the policy set and enforce the exposure ceiling server-side.

Changes

• Implement the alert routing layer for issue Add Alert Deduplication, Grouping, and On-Call Escalation Routing for Oracle Incidents #289. Extend AlertingService (or add a new routing service in this module) with: alert fingerprinting (feedId + alertClass + source, excluding volatile fields like timestamps/values), a configurable dedup window, grouping of alerts into a single incident per feed outage, a severity model that routes payout-triggering divergence to 'page' and degraded non-payout feeds to 'ticket', a persisted incident state machine (open/acknowledged/escalated/resolved) with acknowledgement tracking, re-notification, and escalation to a secondary on-call after a configured delay, automatic resolution when the underlying condition clears, and an incident timeline written to the audit log. Define a Notifier interface with pluggable adapters (log/in-memory default) so delivery is swappable. Persist incidents and transitions in Postgres so a restart does not re-page or lose open incidents. Wire the module to the metrics/anomaly emitters and register it in AppModule. Add tests with a fake clock for dedup, grouping, escalation timing, auto-resolve, and a storm test asserting bounded notification volume for N failing feeds. (src/alerting/alerting.service.ts)
• Register the new alert routing providers (routing service, notifier adapters, incident repository) in AlertingModule and export them so oracle/metrics emitters can inject them. (src/alerting/alerting.module.ts)
• (1) Add the persisted incident state machine tables for issue Add Alert Deduplication, Grouping, and On-Call Escalation Routing for Oracle Incidents #289 (incidents, incident_transitions, incident_acknowledgements, incident_timeline) with indexes on fingerprint+status and created_at, so dedup/grouping and escalation survive restarts and duplicate emits from multiple workers are handled via a unique constraint on (fingerprint, status='open').; (2) Add the composite feed definition and decision/evidence linkage tables (composite_feed_definitions with version, expression AST JSON, unit, rounding policy, output feed id; composite_decisions linking formula version, input decision ids, computed value) with indexes for replay lookup by output feed id and version.; (3) Add the override previews and override applications tables (preview id, input hash, actor, rationale, parameters, exposure totals, validity window, applied/abandoned status, linked preview id) so previews and their eventual application are auditable and mismatches are detectable. (src/db/schema.sql)
• (1) Add a migration creating the alert incident tables (incidents, incident_transitions, incident_acknowledgements, incident_timeline) referenced by the routing layer, including the partial unique index enforcing one open incident per fingerprint and the audit-timeline table.; (2) Add a migration creating composite_feed_definitions and composite_decisions tables with the formula-version and input-decision-id linkage required for deterministic replay of composite evaluations.; (3) Add a migration creating override_previews and override_applications tables with the preview-id linkage, input hash, validity window, and evidence-ledger references required by the preview/apply flow. (src/db/migrations/001_idempotency_and_oracle_partitioning.sql)
• (1) Add environment variables for alert routing: ALERT_DEDUP_WINDOW_MS, ALERT_GROUP_WINDOW_MS, ALERT_ESCALATION_DELAY_MS, ALERT_RENOTIFY_INTERVAL_MS, ALERT_NOTIFIER (log|memory|webhook), and optional webhook URL. Document them with comments in the example env file.; (2) Add heartbeat and watchdog environment variables to the example env file: HEARTBEAT_ENABLED, HEARTBEAT_URL, HEARTBEAT_INTERVAL_MS, HEARTBEAT_GRACE_PERIOD_MS, HEARTBEAT_MAX_CONSECUTIVE_MISSES, WATCHDOG_METRICS_STALENESS_MS, WATCHDOG_ALERT_DELIVERY_STALENESS_MS, with comments documenting cadence, grace period, and that the feature is disableable for local dev.; (3) Add environment variables for override previews: OVERRIDE_PREVIEW_VALIDITY_MS, OVERRIDE_EXPOSURE_CEILING, and role configuration distinguishing preview vs apply authorization, with comments documenting the server-side enforcement. (.env.example)
• (1) Add the alert routing configuration block (dedup window, grouping window, escalation delay, re-notify interval, notifier selection, webhook URL) to the registered app config so services can read it via ConfigService.; (2) Add the heartbeat/watchdog configuration block (enabled flag, URL, interval, grace period, max consecutive misses, watchdog staleness thresholds) to the registered app config.; (3) Add the override preview configuration block (preview validity window, exposure ceiling, preview/apply role settings) to the registered app config so the server enforces them rather than the CLI. (src/config/configuration.ts)
• (1) Add zod validation for the new alert routing env vars (ALERT_DEDUP_WINDOW_MS, ALERT_GROUP_WINDOW_MS, ALERT_ESCALATION_DELAY_MS, ALERT_RENOTIFY_INTERVAL_MS, ALERT_NOTIFIER, ALERT_WEBHOOK_URL) so misconfiguration fails fast at startup.; (2) Add zod validation for the heartbeat and watchdog env vars (HEARTBEAT_ENABLED, HEARTBEAT_URL, HEARTBEAT_INTERVAL_MS, HEARTBEAT_GRACE_PERIOD_MS, HEARTBEAT_MAX_CONSECUTIVE_MISSES, WATCHDOG_METRICS_STALENESS_MS, WATCHDOG_ALERT_DELIVERY_STALENESS_MS), ensuring URL format and integer bounds.; (3) Add zod validation for OVERRIDE_PREVIEW_VALIDITY_MS and OVERRIDE_EXPOSURE_CEILING (integer bounds, non-negative ceiling) so invalid override configuration fails fast at startup. (src/config/env.schema.ts)
• Implement the dead-man's-switch heartbeat module for issue Add Dead-Man's-Switch Meta-Monitoring for the Oracle Pipeline and Alerting Stack #290: a HeartbeatService that is invoked only from the oracleMonitor cycle-completion hook after a successful poll cycle and successful DB write (not a standalone timer), POSTs a payload containing feed coverage counts and last successful publish timestamp to a configurable external endpoint, requires N consecutive send failures before raising an alert (grace period), never includes secrets, and is disableable via env for local dev. Also add a local watchdog that checks metrics-scrape freshness and alert-delivery success and raises a high-severity alert when they lapse. Add unit tests for stalled cycle (no heartbeat), unreachable receiver grace behavior, and watchdog staleness thresholds. (src/services/oracleMonitor.ts)
• Wire the heartbeat into the service loop: after a successful poll cycle and DB write completes, call the heartbeat module's completed-work signal. Ensure the heartbeat is driven by the completed-work signal rather than a standalone timer so a wedged loop stops heartbeating by construction, and that heartbeat failures never block ingestion or settlement. (src/index.ts)
• Add a runbook section documenting the dead-man's-switch: expected heartbeat cadence, grace period, what a missing heartbeat means, and the operator runbook entry for 'heartbeat missing' (issue Add Dead-Man's-Switch Meta-Monitoring for the Oracle Pipeline and Alerting Stack #290). (docs/threat-model.md)
• Implement versioned composite/derived feed support for issue Support Versioned Composite and Derived Feeds Built from Upstream Feeds #298: extend the coverage catalog with a composite feed definition schema (input feed ids, deterministic non-Turing-complete AST expression, unit and rounding policy, output feed id, explicit version). Add a pure fixed-decimal AST evaluator (using src/common/fixed-point.ts) that evaluates only over upstream consensus outputs with declared freshness, rejects cycles/self-reference/undefined units at config load, builds a dependency graph for topological ordering across composites of composites, inherits the weakest required input state (stale/quarantined/degraded), and guards divide-by-zero, overflow, and NaN. Emit a consensus-quality decision and evidence record per evaluation recording formula version, input decision ids, and computed value. Add property-based determinism/version-stability tests, table-driven unit/rounding/div-zero/overflow tests, graph cycle/ordering tests, consensus degradation tests, and a replay test. (src/common/coverage-catalog.ts)
• Extend the oracle encoding helpers to encode composite feed values deterministically (fixed-decimal to i128) and to include the formula version in the encoded decision payload, so composite outputs are byte-for-byte reproducible across nodes and replays. (src/oracle/oracle-encoding.ts)
• Refactor claim processing for issue Preview Payout Impact of a Proposed Manual Oracle Override Before Applying It #299: extract the policy trigger evaluation into a pure, side-effect-free function parameterized by feed state (current vs proposed override) so the preview runs the real logic rather than a parallel approximation. Ensure the preview path cannot settle claims. Add tests asserting the preview performs zero writes and that the preview-triggered set matches the actual post-apply triggered set for the same inputs. (src/services/claimProcessor.ts)
• Add the override preview service and apply flow for issue Preview Payout Impact of a Proposed Manual Oracle Override Before Applying It #299: given a proposed override (feed id, value or suppressed state, effective window), compute a non-mutating dry-run preview listing affected active policies, whether each would trigger under current and proposed conditions, and aggregate payout exposure, computed against a single read-transaction snapshot with a bounded policy scan and explicit truncation reporting. Persist the preview with a hash of its inputs and a short validity window, return a preview token/id, and require that token on apply (revalidating the policy set and invalidating on change). Enforce a configurable server-side exposure ceiling requiring additional approval, and record every preview and its application/abandonment in the evidence ledger with actor, rationale, parameters, and preview id. Add tests for expired tokens, mutated policy sets, ceiling rejection, authorization, and evidence linkage. (src/claim/claim.service.ts)
• Add ops endpoints for override preview and apply (POST preview, POST apply with preview token) under the ops claim controller, enforcing distinct authorization for preview vs apply roles and rejecting overrides exceeding the exposure ceiling without additional approval. Ensure the preview endpoint is non-mutating and cannot be repurposed to settle claims. (src/claim/ops-claim.controller.ts)