LedgerReconciliationService never fetches on-chain data: reconciliation flags every stored entry missing #427

Problem

The "ledger reconciliation" service compares on-chain data against the database but never fetches any on-chain data. LedgerReconciliationService.fetchOnChainData is a placeholder that returns an empty list, so the comparison is structurally vacuous:

// app/backend/src/onchain/ledger-reconciliation.service.ts
private fetchOnChainData(_startLedger: number, _endLedger: number): OnChainLedgerEntry[] {
  // Placeholder for actual Horizon API call
  // In production, this would query the Stellar Horizon API
  return [];
}

In processReconciliation, the forward loop over onChainData therefore never runs, and the reverse loop flags every stored BalanceLedger row as a discrepancy with ledger: -1:

for (const storedEntry of storedEntries) {
  const onChainEntry = onChainData.find(e => e.id === storedEntry.id); // always undefined
  if (!onChainEntry) {
    discrepancies.push({ ledger: -1, type: 'missing', expected: null, observed: storedEntry, severity: 'medium' });
  }
}

Consequence: triggerReconciliation reports a completed job whose discrepancies are a false-positive "missing" row for every DB ledger entry and whose on-chain side was never examined. The feature advertises a trust boundary — "compare on-chain vs stored" — but can neither detect a genuinely missing on-chain entry, an amount mismatch, nor an event-type mismatch. Operators acting on its actionable flag act on noise. Additionally, the amount comparison that would run uses storedEntry.amount.toNumber(), a lossy float conversion for the i128-scale token amounts the contract uses.

Root cause

The service was scaffolded with a simulated data source and shipped before the Horizon/tx-source integration was implemented; the reverse-loop guard makes the missing source produce misleading output instead of an explicit "unimplemented" failure.

Why this is architecturally hard

1. The on-chain source is the whole point. A real implementation must page Stellar/Soroban ledger data (Horizon or the RPC's transaction/ledger history) and correlate it with BalanceLedger rows, including deciding what constitutes a ledger "entry" for a smart-contract platform whose events (package_created, etc.) are not Horizon balance rows.
2. The comparison contract is ill-defined. OnChainLedgerEntry has id, ledger, amount, eventType, but BalanceLedger stores different fields; the fix must define the join key and the exact equivalence for amount (raw integer units, not toNumber() floats).
3. Failure must be loud, not "completed". If the source is unavailable, the job must surface failed/degraded with a clear error rather than a completed report; otherwise the same silent-confidence bug recurs.
4. It must reuse existing infra. The onchain BullMQ queue and OnchainProcessor already exist, and SorobanAdapter.getTransactionStatus/the event stream are the likely data source — the design must integrate with them rather than add a parallel Horizon client.

Proposed design

Replace fetchOnChainData with a real, pageable source (RPC/Horizon) returning the events/entries within [startLedger, endLedger], define the join key, and treat a source failure as a failed job. Use integer/BigInt comparison for amounts. Keep the discrepancy classification (missing/amount_mismatch/count_mismatch) but only emit ledger: -1 when genuinely unknown, and never fabricate a completed summary when the source returned nothing.

Acceptance criteria

Service

• processReconciliation reads real on-chain data for the requested ledger range; a source error fails the job rather than returning a completed report.
• With on-chain data present, a stored row with no on-chain counterpart produces a real missing discrepancy with the correct ledger, and an amount difference above thresholdPercent produces an amount_mismatch using exact integer units.
• With zero on-chain entries available, the job does not flag every stored row as missing with ledger: -1.

Tests

• Unit tests inject a fake on-chain source and assert each discrepancy type and the actionable computation.

Out of scope

Replacing ioredis-mock with testcontainers and the BalanceLedger Float→Decimal work are separate, already-tracked concerns.

Getting started

Files: app/backend/src/onchain/ledger-reconciliation.service.ts, app/backend/src/onchain/onchain.processor.ts, app/backend/src/onchain/ledger-backfill.service.ts.

cd app/backend
npm test

Good first files to read: ledger-reconciliation.service.ts (the current, vacuous flow) and onchain.processor.ts (the queue job that would invoke it).