[misrasamuelisiguzor-oss]

Closes #287

Issues

#287 Build a Versioned Reference Data Registry for Feed Entity Validation

Description:
Add a versioned reference data store covering the entities that payout-triggering feeds describe — airports, routes and carriers for FlightDelay, protocols, markets and chains for LiquidationShield, asset identifiers for price/TVL feeds — and validate every incoming reading against it before the reading becomes eligible for consensus.

Problem Statement & Context:
The mocked triggers currently accept whatever the fixture returns. Live NEXUS and aviation data will arrive with provider-specific entity IDs that must be resolved, validated, and canonicalized: an unknown airport code, a delisted protocol, or two different assets sharing a ticker (USDC, USDT on multiple chains) would otherwise produce a confident, wrong payout. Reference data also gives evidence records a stable canonical entity reference so a decision can be re-examined years later.

Scope & Acceptance Criteria:

• Schema for reference entities and datasets with explicit dataset version and effective-from/effective-to timestamps, plus a new migration under src/db/migrations/.
• Import tooling (script under scripts/) to load and version a dataset, with a dry-run diff mode.
• Ingestion-time validation: readings referencing unknown or retired entities are rejected into the dead-letter path with a reason, never counted toward quorum.
• Entity resolution table mapping provider-native IDs to canonical IDs, including many-to-one cases and provider-specific aliases.
• Out of scope: licensing or purchasing the authoritative upstream datasets (tracked separately); this issue delivers the registry, tooling, and enforcement.

Implementation Guidelines:

1. Key Files/Modules: src/oracle/ (new reference data module), src/db/schema.sql, src/db/migrations/, scripts/ for the importer, and validation hooks in src/services/oracleMonitor.ts.
2. Design/Architecture: reference data is immutable per version; feeds pin a dataset version in config and updates are promoted deliberately, so a dataset refresh cannot retroactively change how a past reading was interpreted.
3. Edge Cases/Constraints: retired airports that still appear in delayed historical flights, entity renames, ambiguous ticker collisions resolved only by chain+contract, and unknown-entity storms that must not flood alerts (rate-limited aggregation).
4. Testing: unit tests for unknown entity, retired entity, alias resolution, ambiguous match rejection, and version pinning; an integration test proving a rejected entity never reaches consensus.

Definition of "Done":

• Code written, tested, and documented.
• All acceptance criteria met.
• PR submitted with a clear description, passing CI, relevant screenshots/test output.
• Code reviewed and approved.

Complexity: Medium (150 points)

Summary

The reference-data module should be fail-closed: if a feed has no pinned dataset version or the pinned version is not effective, readings are rejected to the dead-letter path. Ambiguous alias mappings must be treated as rejections, not resolved arbitrarily. The import script must never mutate published versions; changes create a new version with effective dates. Rate limiting for unknown-entity alerts should be implemented in-process (e.g. sliding window) and configurable via env. The integration test must prove rejected readings never reach quorum, so the quorum/aggregation logic in OracleService may need a small seam for testability.

Changes

• Create a new SQL migration file (e.g. src/db/migrations/004_reference_data_registry.sql) that defines the versioned reference data registry tables: reference_datasets (id, name, version, effective_from, effective_to, status, created_at, immutable once published), reference_entities (id, dataset_id, canonical_id, entity_type, attributes JSONB, retired_at), and reference_entity_aliases (id, dataset_id, provider, provider_native_id, canonical_entity_id, is_ambiguous, created_at) with unique constraints on (dataset_id, provider, provider_native_id) and indexes for resolution lookups. Also add a dead_letter_readings table (or extend existing dead-letter storage) with reason and raw payload columns, and a feed_reference_pins table mapping feed name to dataset version. Ensure the migration is idempotent (IF NOT EXISTS) and includes comments explaining immutability and effective date semantics. (src/db/migrations/001_idempotency_and_oracle_partitioning.sql)
• Update src/db/schema.sql to include the reference data registry tables (reference_datasets, reference_entities, reference_entity_aliases), feed_reference_pins, and dead_letter_readings definitions so the canonical schema reflects the new registry. Keep the existing claims table and indexes intact; add the new tables with the same DDL as the migration and note that the migration is the authoritative source for existing databases. (src/db/schema.sql)
• Create a new reference-data module under src/oracle/reference-data/ containing: reference-data.types.ts (dataset, entity, alias, resolution result types), reference-data.repository.ts (Postgres-backed queries for resolving provider-native IDs to canonical IDs within a pinned dataset version, checking effective_from/effective_to and retired status, detecting ambiguous mappings), reference-data.service.ts (validation entry point that takes an OracleReading with entity references and returns a validation result with reason codes unknown_entity, retired_entity, ambiguous_entity, or ok), and reference-data.module.ts exporting the service. Include rate-limited aggregation of unknown-entity alerts (e.g. a token-bucket or sliding-window counter keyed by provider/entity type) to avoid alert storms. (src/oracle/oracle.module.ts)
• Extend the OracleReading interface in src/oracle/oracle-reading.ts to carry optional entity references (e.g. entities: Array<{ type: string; provider: string; providerNativeId: string }>) and a referenceDatasetVersion field so ingestion can validate them. Also add optional canonicalEntityIds populated after resolution, and a validationStatus field (pending, accepted, rejected) with rejectionReason for dead-letter routing. (src/oracle/oracle-reading.ts)
• Wire reference-data validation into the oracle ingestion path in src/oracle/oracle.service.ts: before a reading is accepted for quorum, call ReferenceDataService.validateReading using the feed's pinned dataset version; if the result is rejected (unknown, retired, or ambiguous entity), route the reading to the dead-letter path via OracleEventRepository (or a new dead-letter repository method) with the reason and do not include it in quorum/aggregation. Ensure the service loads the pinned dataset version from feed configuration and fails closed if the pin is missing or the dataset version is not effective. (src/oracle/oracle.service.ts)
• Add dead-letter persistence and reference-data-aware recording to src/oracle/oracle-event.repository.ts: add a recordDeadLetter(reading, reason) method that inserts into dead_letter_readings with the raw payload and reason, and ensure record/recordBatch skip or annotate readings that were rejected by reference-data validation so they never count toward quorum. Also add a method to fetch the pinned dataset version for a feed if not already provided by configuration. (src/oracle/oracle-event.repository.ts)
• Update src/services/oracleMonitor.ts to hook reference-data validation into the monitor path: when readings are observed, validate entity references against the pinned dataset version and emit rate-limited alerts for unknown-entity storms (using the aggregation from ReferenceDataService). Ensure rejected readings are forwarded to the dead-letter path and excluded from quorum calculations. (src/services/oracleMonitor.ts)
• Add feed reference-data version pinning to src/config/configuration.ts: introduce a configuration namespace (e.g. referenceData) that reads per-feed dataset version pins from environment variables (e.g. REFERENCE_DATA_PIN_) and exposes a helper to resolve the pinned version for a feed. Default to the latest active dataset version only when explicitly allowed; otherwise fail closed. (src/config/configuration.ts)
• Extend src/config/env.schema.ts with optional environment variables for reference data: REFERENCE_DATA_PIN_ entries, REFERENCE_DATA_IMPORT_DRY_RUN, and rate-limit settings for unknown-entity alerts (e.g. REFERENCE_DATA_UNKNOWN_ALERT_WINDOW_MS, REFERENCE_DATA_UNKNOWN_ALERT_MAX). Validate them with the existing zod helpers and document that pins are required in production. (src/config/env.schema.ts)
• Create a new import script at scripts/import-reference-data.ts that loads reference datasets from a JSON/CSV source, validates schema, assigns a new immutable version with effective_from/effective_to, and supports a --dry-run flag that prints a diff of added, changed, retired, and ambiguous alias mappings without writing. The script should use the reference-data repository and fail if it attempts to mutate an already-published version. (scripts/regenerate-xdr-fixtures.ts)
• Add unit tests for the reference-data module covering unknown entities, retired entities, alias resolution (many-to-one), ambiguous matches, and version pinning. Place them alongside the new module (e.g. src/oracle/reference-data/reference-data.service.spec.ts) and ensure they run under the default Jest config. (src/oracle/oracle.service.spec.ts)
• Add an integration test (e.g. test/integration/reference-data.ispec.ts) that seeds a versioned dataset with aliases and a retired entity, ingests oracle readings with unknown/retired/ambiguous entities, and asserts they are written to the dead-letter path and never counted toward quorum or consensus. Also assert that a valid aliased entity resolves to the canonical ID and is accepted. Ensure the test uses the existing testcontainers setup and truncates the new tables between tests. (test/integration/claims.ispec.ts)
• Update test/integration/jest.integration.config.ts if needed to include the new integration test file pattern and ensure the global setup applies the new reference-data migration before tests run. No functional change if the existing glob already matches; add a comment documenting that the reference-data migration is applied by the setup. (test/integration/jest.integration.config.ts)

[drips-wave]

@misrasamuelisiguzor-oss Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits