[ezikifactt-gif]

Closes #294
Closes #295
Closes #296
Closes #297

Issues

#294 Capture and Version Provider Methodology Metadata for Independence and Evidence

Description:
Add a methodology registry that records, for each source, how it derives the values it publishes — aggregation method, sampling universe, revision policy, latency profile, and known failure modes — requires it at source onboarding, versions it, and references the exact version in every evidence record and decision.

Problem Statement & Context:
Corroboration only means something if the sources are actually independent, and disputes can only be defended if we can explain what a source's number represented at the time. Today that knowledge lives in README prose and threat-model notes: that a price is a cross-exchange volume-weighted average, or that a TVL figure counts staked-but-locked assets, is invisible to the consensus weighting and to anyone reconstructing a decision later. Without a recorded methodology, two endpoints of the same aggregator look like two independent sources.

Scope & Acceptance Criteria:

• Schema for methodology records (source ID, derivation description, sampling universe, revision/late-correction policy, expected latency, known failure modes) with immutable versioning, plus a migration under src/db/migrations/.
• Onboarding validation: a source cannot be activated without an approved methodology record, and activation requires explicit sign-off flags.
• Every consensus decision and published reading references the methodology version(s) used, so point-in-time reconstruction resolves the exact semantics.
• Methodology is exposed on the public trust/provenance surface and in the operator CLI.
• Out of scope: the license and terms-of-service registry (separate legal track) and automated scraping of provider documentation.

Implementation Guidelines:

1. Key Files/Modules: src/oracle/ source definition module, src/db/schema.sql, src/db/migrations/, the evidence ledger writer, the admin/feed management surface, and the operator CLI.
2. Design/Architecture: methodology records are immutable rows referenced by version, mirroring quorum policy snapshots, so edits create a new version rather than mutating history.
3. Edge Cases/Constraints: a provider silently changing its own methodology (record must be re-attested on a schedule and flagged when the source's observed behavior diverges from the description), sources with no publishable methodology (must be barred from higher independence weight), and migration of already-registered sources.
4. Testing: validation tests for missing/incomplete methodology blocking activation, versioning tests proving historical decisions resolve to the old version, and a test that the trust API renders methodology for a feed.

Definition of "Done":

• Code written, tested, and documented.
• All acceptance criteria met.
• PR submitted with a clear description, passing CI, relevant screenshots/test output.
• Code reviewed and approved.

Complexity: Medium (120 points)

#295 Partition and Index Oracle Readings and Evidence Tables for Sustained Write Volume

Description:
The new oracle pipeline writes a row per source per poll interval for every feed, plus a consensus decision row and an append-only evidence record, while auditors and dispute reviewers query long historical ranges. This issue defines and implements a partitioning, indexing, and query-budget strategy so write throughput, vacuum load, and audit-read latency stay bounded as feeds, sources, and node operators multiply.

Problem Statement & Context:
Moving from a single poller that writes one value per feed to a corroborated engine with N sources per feed multiplies write volume by roughly N and keeps far more history for evidence and replay. Without a deliberate physical data model, the readings/evidence tables become the pipeline's first scalability wall: index bloat slows consensus-window reads, autovacuum falls behind, and audit queries contend with the hot write path. Getting this right early is cheap; retrofitting partitions onto live evidence tables is not.

Scope & Acceptance Criteria:

• Add time-based range partitioning (monthly or weekly, decided by measured volume) for the per-source reading table and the evidence ledger table, with an automated partition-creation step that pre-creates future partitions.
• Define covering indexes for the hot query shapes: consensus window per (feed, source, observed_at), evidence lookup by decision/payout id, and trust-status queries by feed.
• Add a documented query-budget policy: every repository query against these tables must bound the timestamp range; add a review checklist entry and a lint/review aid for new queries.
• Provide a migration that converts the existing non-partitioned tables to partitioned form without data loss, and document how it is applied to an existing deployment.
• Out of scope: retention/deletion policy and archival export (covered by a separate retention issue), and any change to consensus semantics.

Implementation Guidelines:

1. Key Files/Modules: src/db/schema.sql (canonical table definitions), src/db/migrations/ (new numbered migration for partitioning and indexes), src/services/oracleMonitor.ts (ingestion write path), the consensus decision writer, and the evidence-ledger repository used by the trust-status and audit code paths.
2. Design/Architecture: use declarative range partitioning on the observation/decision timestamp; ensure every unique constraint includes the partition key; run partition maintenance from the existing scheduler (or a dedicated job) with an alert if the next partition is missing; keep a small hot partition set plus indexes sized to the consensus window rather than to full history.
3. Edge Cases/Constraints: migrations must take bounded locks (create new partitioned table, backfill in batches, swap names) and must be safe to re-run; queries that omit the timestamp predicate must fail loudly rather than scan all partitions — add assertions or explain-plan checks; be careful that foreign keys across partitions are supported or replaced with enforced-by-application references.
4. Testing: migration up/down (or forward-only with verification) against a seeded multi-million-row fixture; automated EXPLAIN (ANALYZE, BUFFERS) assertions that consensus-window and evidence-lookup queries hit only the expected partitions and indexes; a write-throughput regression test documenting rows/second before and after; a slow-query budget test that fails CI if unbounded scans are reintroduced.

Definition of "Done":

• Code written, tested, and documented.
• All acceptance criteria met.
• PR submitted with a clear description, passing CI, relevant screenshots/test output.
• Code reviewed and approved.

Resources:
None — no existing design docs or external references to link.

Complexity: Medium (60 points)

#296 Build a Disaster Recovery Restore Drill and Evidence-Store Backup Verification

Description:
Add an automated restore drill that rebuilds the oracle PostgreSQL store from a backup into an ephemeral environment, verifies the append-only evidence chain and consensus decisions are intact, and reports measured RPO/RTO against targets. Document the procedure so recovery is a rehearsed routine rather than an untested hope.

Problem Statement & Context:
The evidence ledger is the system's record of why money moved: source payloads, consensus decisions, and payout linkage. If PostgreSQL is lost or corrupted, the backend can re-poll live feeds but can never regenerate the historical justification for past payouts. We currently have migrations and a Docker-based local stack but no evidence that a backup actually restores, no defined RPO/RTO, and no verification step that would catch a silently corrupted or truncated evidence chain.

Scope & Acceptance Criteria:

• Define and document RPO/RTO targets for the oracle database, differentiating evidence/decision data (must be lossless to the last committed decision) from cache-like data.
• Add a script that provisions an ephemeral PostgreSQL instance, applies src/db/migrations/ from scratch, restores a backup (or a logical dump produced by the backup path), and runs integrity verification.
• Integrity verification must include: row counts per feed/decision table within tolerance, gap-free evidence sequence/chain verification, and referential check that every settled claim has a decision and evidence record.
• Run the drill on a schedule (weekly) in CI or a staging job and emit a pass/fail signal plus measured restore duration and data loss window.
• Record the drill outcome in an operator-visible place (metrics endpoint or a small report artifact).
• Out of scope: choosing or provisioning a production managed-backup vendor, and any cross-region replication design.

Implementation Guidelines:

1. Key Files/Modules: scripts/ (drill entrypoint, e.g. a restore-and-verify script), src/db/migrations/ and src/db/schema.sql (must apply cleanly to a fresh instance), docker-compose.yml (ephemeral service definition for the drill), docs/ (recovery runbook), and the verification queries against the evidence ledger tables.
2. Design/Architecture: implement as an idempotent, container-friendly script driven by environment variables for source backup and target instance; verification is a set of SQL assertions plus a chain/hash continuity check; the scheduled job runs it against a disposable database and tears everything down afterwards.
3. Edge Cases/Constraints: never point the drill at production; the restore must tolerate backups taken mid-run (partial partitions, in-flight decisions) and still verify consistency; handle credentials via environment/secret injection rather than checked-in values; fail closed if the backup is older than the RPO target.
4. Testing: exercise the drill with a clean backup, a deliberately truncated backup, and a backup missing the most recent partitions — the last two must fail with a clear diagnostic; unit-test the verification assertions against synthetic evidence data; add a smoke test ensuring fresh schema.sql/migrations produce a database the drill can verify.

Definition of "Done":

• Code written, tested, and documented.
• All acceptance criteria met.
• PR submitted with a clear description, passing CI, relevant screenshots/test output.
• Code reviewed and approved.

Resources:
None — no existing design docs or external references to link.

Complexity: Medium (50 points)

#297 Auto-Generate Incident Timelines and Postmortems from the Oracle Evidence Ledger

Description:
Build a generator that assembles a chronological incident timeline — source readings, consensus decisions, anomaly triggers, pauses, and affected payouts — directly from the evidence ledger whenever a circuit breaker, operator pause, or degradation event fires. Produce a draft postmortem document operators can review and annotate instead of reconstructing the incident by hand.

Problem Statement & Context:
Today, investigating a paused feed or a disputed payout means manually correlating logs across the monitor service, the claim processor, and the database. The evidence ledger already holds the authoritative sequence, but nothing turns it into a readable narrative. Slow reconstruction directly extends outage duration and degrades payout-decision defensibility; it also makes recurring failure modes invisible because nobody has time to look back.

Scope & Acceptance Criteria:

• Add an incident record concept: an incident is opened automatically by defined trigger classes (circuit breaker trip, quorum failure, freshness breach, operator pause, backfill/replay event) or opened manually by an operator.
• Generate a timeline of linked ledger events for the incident window, including per-source readings with their health at the time, consensus outcomes and reasons, trust-mode changes, and any claims settled, held, or reversed.
• Render the timeline into a structured draft postmortem (markdown) with sections for timeline, impact (feeds and payout exposure), detection, and open questions, leaving human-authored analysis fields blank.
• Expose the incident list and rendered postmortem through the operator/admin surface and store the drafted artifact with a stable id.
• Out of scope: paging/on-call routing mechanics, and any automatic root-cause conclusion — the generator assembles facts, it does not assign blame.

Implementation Guidelines:

1. Key Files/Modules: the evidence-ledger and consensus-decision repository in src/db/ (plus a new migration for an incidents table and event links), src/services/oracleMonitor.ts and src/services/claimProcessor.ts (trigger points that open incidents and attach events), admin/operator route module under src/routes/, and a generator module under src/services/.
2. Design/Architecture: events are appended to the incident as they happen (event sourcing style) so the timeline survives a crash, with the markdown rendering done on demand from stored events; query the ledger by time window plus feed/source/decision filters, and cache the rendered document keyed by incident version.
3. Edge Cases/Constraints: incidents must not leak secrets or unredacted provider payloads into the drafted document; a rolling window of many small breaker trips should be grouped rather than producing dozens of incidents (link to alert grouping rules); rendering must be deterministic so diffs are meaningful; large windows must paginate or truncate with an explicit marker.
4. Testing: golden-file tests for the rendered output given a fixed synthetic ledger; tests for each auto-open trigger class; a test that an incident timeline remains complete when the process restarts mid-incident; a redaction test asserting sensitive provider fields never appear in the rendered markdown.

Definition of "Done":

• Code written, tested, and documented.
• All acceptance criteria met.
• PR submitted with a clear description, passing CI, relevant screenshots/test output.
• Code reviewed and approved.

Resources:
Internal reference: docs/threat-model.md (P1 checklist) motivates durable, reviewable evidence for payout-triggering decisions.

Complexity: Medium (55 points)

Summary

The four issues are tightly coupled: the methodology registry (#294) and incidents (#297) both need new tables and migrations, and the partitioning work (#295) changes the physical shape of the readings/evidence tables that #294 and #297 write to, so schema.sql and the new migration must land together. The DR drill (#296) depends on migrations applying cleanly from scratch, so it must be tested after the partitioning migration is in place. Keep every migration idempotent and re-runnable so the restore drill can apply them to a fresh instance. Enforce the query-budget policy at the repository layer (throw on missing timestamp predicate) rather than relying only on review. Redact provider payloads in the incident generator before rendering markdown, and make rendering deterministic (stable ordering, no timestamps of render time) so golden-file diffs are meaningful.

Changes

• Add the methodology registry schema and the incident/partitioning support to the canonical schema. Define an immutable oracle_methodology_versions table (source_id, version, derivation_description, sampling_universe, revision_policy, expected_latency, known_failure_modes, approved, signoff flags, created_at, superseded_at) with a unique (source_id, version) constraint and no UPDATE path; add oracle_sources activation columns referencing the approved methodology version. Add oracle_consensus_decisions and oracle_evidence tables that carry methodology_version_ids (JSONB array) so every decision and evidence record resolves the exact semantics. Define oracle_readings and oracle_evidence as declarative RANGE partitions on observed_at/decided_at (monthly) with the partition key in every primary/unique key, plus covering indexes for (feed, source, observed_at), evidence lookup by decision/payout id, and trust-status by feed. Add an incidents table and incident_events link table (incident_id, event_type, ledger_ref, payload_redacted, occurred_at) for issue Auto-Generate Incident Timelines and Postmortems from the Oracle Evidence Ledger #297. Keep the file idempotent (IF NOT EXISTS) so fresh installs and the restore drill both work. (src/db/schema.sql)
• Add a new numbered migration 004_methodology_partitioning_and_incidents.sql that: (a) creates the methodology registry tables and backfills a placeholder 'unapproved' methodology row for every already-registered source so migration of existing sources is explicit and they are blocked from activation until approved; (b) converts the existing non-partitioned oracle readings/evidence tables to partitioned form using the bounded-lock pattern (create new partitioned table, backfill in batches, swap names, re-runnable); (c) creates the covering indexes described in schema.sql; (d) creates the incidents and incident_events tables. Include a documented query-budget note in the header and make every statement idempotent so the restore drill can apply it from scratch. (src/db/migrations/001_idempotency_and_oracle_partitioning.sql)
• Add a partition-maintenance routine to the existing scheduler: on module init and on a daily cron, pre-create the next N monthly partitions for oracle_readings and oracle_evidence, and emit a critical alert via AlertingService if the next partition is missing or creation fails. Also register the weekly incident-timeline generation and the DR drill trigger hooks so the scheduler is the single place scheduled maintenance lives. (src/oracle/oracle.scheduler.ts)
• Extend OracleService with methodology-registry operations: createMethodologyVersion (immutable insert), approveMethodologyVersion (sets approved + sign-off flags), getActiveMethodology(sourceId), and assertSourceActivatable(sourceId) which throws when no approved methodology exists. Record the methodology version ids used on every consensus decision and evidence write, and expose a getMethodologyForFeed(feed) used by the trust/provenance surface. Add a re-attestation check that flags a source when observed behavior diverges from its recorded methodology. (src/oracle/oracle.service.ts)
• Expose methodology and incident endpoints on the oracle controller: GET /api/v1/oracle/feeds/:feed/methodology (public trust/provenance), GET /api/v1/ops/oracle/sources/:id/methodology and POST /api/v1/ops/oracle/sources/:id/methodology (create/approve with sign-off flags), GET /api/v1/ops/incidents and GET /api/v1/ops/incidents/:id/postmortem. Guard ops routes with AuthGuard/ScopesGuard and record audit entries for methodology approvals. (src/oracle/oracle.controller.ts)
• Register the new methodology repository/service, the incident service, the postmortem generator, and the partition-maintenance provider in the OracleModule so they are injectable across the app (claim processor, admin routes, CLI). (src/oracle/oracle.module.ts)
• Update OracleEventRepository to write into the partitioned oracle_readings/oracle_evidence tables with the observation timestamp included in every insert and every query, and to persist the methodology_version_ids on each evidence record. Add a bounded-range guard that throws when a query omits the timestamp predicate, enforcing the query-budget policy for issue Partition and Index Oracle Readings and Evidence Tables for Sustained Write Volume #295. (src/oracle/oracle-event.repository.ts)
• Add the incident trigger points in the monitor service: when a circuit breaker trips, quorum fails, freshness breaches, an operator pause occurs, or a backfill/replay event fires, call the IncidentService to open or group an incident and append the triggering ledger event. Ensure rolling breaker trips within the grouping window attach to the existing incident rather than creating duplicates. (src/services/oracleMonitor.ts)
• Add incident trigger points in the claim processor: when a claim is held, reversed, or settled during a degraded/incident window, append the corresponding event to the open incident so the postmortem timeline includes affected payouts and exposure. (src/services/claimProcessor.ts)
• Update ClaimSettlementService to record the methodology version ids used for the decision on each settlement and to append a settlement event to any open incident covering the affected feed, so historical decisions resolve to the exact methodology semantics. (src/claim/claim-settlement.service.ts)
• Add repository methods to ClaimRepository for the DR drill and incident generator: bounded-range queries for settled claims with their decision and evidence references, a referential-integrity check (every settled claim has a decision and evidence record), and a per-feed row-count query used by the restore verification. All queries must include an explicit timestamp range to satisfy the query-budget policy. (src/claim/claim.repository.ts)
• Add operator CLI endpoints to OpsClaimController for methodology management (list/create/approve methodology versions with sign-off flags) and for incidents (list incidents, fetch rendered postmortem markdown), with audit logging on every mutation. (src/claim/ops-claim.controller.ts)
• Add Prometheus metrics for the DR drill (last run timestamp, pass/fail gauge, measured restore duration, data-loss window) and for incident generation (incidents opened by trigger class), and expose them through the existing /metrics endpoint so drill outcomes are operator-visible. (src/metrics/metrics.service.ts)
• Add an ephemeral postgres-drill service to docker-compose.yml (separate from the primary postgres, no persistent volume, disposable) used by the restore-and-verify drill, and document that the drill must never point at the production database. (docker-compose.yml)
• Create the idempotent, env-driven restore-and-verify drill script: provisions/uses an ephemeral PostgreSQL instance, applies src/db/migrations/ from scratch, restores a logical dump from BACKUP_SOURCE, runs SQL assertions (row counts per feed/decision table within tolerance, gap-free evidence sequence/hash continuity, referential check that every settled claim has a decision and evidence record), fails closed if the backup is older than the RPO target, and emits pass/fail plus measured restore duration and data-loss window to the metrics endpoint or a report artifact. Must be safe to re-run and must tear down the ephemeral instance afterwards. (scripts/regenerate-xdr-fixtures.ts)
• Add a weekly DR drill job to CI: start the ephemeral postgres-drill service, run the restore-and-verify script against a clean backup, a deliberately truncated backup, and a backup missing the most recent partitions (the last two must fail with clear diagnostics), and publish the pass/fail signal and measured RPO/RTO as a job artifact. Also add the EXPLAIN (ANALYZE, BUFFERS) partition/index assertions and the slow-query budget check that fails CI if unbounded scans are reintroduced. (.github/workflows/ci.yml)
• Add npm scripts for the restore drill (dr:restore-drill), the partition/index EXPLAIN assertions (test:query-budget), and the incident postmortem golden-file tests so CI and operators can invoke them consistently. (package.json)
• Document the recovery runbook: RPO/RTO targets differentiating evidence/decision data (lossless to the last committed decision) from cache-like data, how to apply the partitioning migration to an existing deployment, the query-budget policy and review checklist for new queries against readings/evidence tables, and the incident/postmortem workflow. Reference the drill script and CI job. (docs/threat-model.md)
• Extend the integration test harness to apply the new migrations (including the partitioning and methodology migrations) after schema.sql, and add helpers for truncating the new methodology, incidents, and partitioned readings/evidence tables so the new integration tests can run against a realistic database. (test/integration/setup/db-client.ts)

[drips-wave]

@ezikifactt-gif Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits