bug(agent): finalization recovery capacity rejection leaves acknowledged KAs unreadable

[Bojan131]

Summary

Base mainnet DKG 10.0.12 can acknowledge a publish while the receiving cores discard the graph-scoped finalization envelope because the durable recovery inbox is full. The affected KAs then remain unreadable on every StorageACK candidate: Jenkins reports a successful publish, but the follow-up remote-sync check finds no triples on any acknowledged core.

This is distinct from:

• Investigate recurring SWM store scheduler queue wait timeouts on mainnet #1990: store-scheduler admission/deadline pressure;
• Base publish ACKCollector retries on /dkg/10.0.2/storage-ack protocol negotiation failures #1987: /dkg/10.0.2/storage-ack negotiation failures.

Those two families are still live and likely contribute to the backlog, but this issue tracks the separate correctness failure at the finalization recovery capacity boundary.

Reproduction evidence

Jenkins

• Run: V10 Base Mainnet #514
• Window: approximately 2026-08-05T08:40Z–09:21Z
• Runtime: Base mainnet, DKG 10.0.12
• Workload: two publishers, 33 KAs each

Results:

Publisher	Publish	Query Remote (sync)
DMaaST	27/33 (81.82%)	30/33 (90.91%)
SBB	31/33 (93.94%)	25/33 (75.76%)
Total	58/66	55/66

All 11 remote-sync failures have the same terminal evidence:

Query Remote (sync) found no readable copy of <ual> across 3 storage candidate(s) — each StorageACK core returned no triples.

Affected UAL suffixes:

• SBB: 3454, 3455, 3456, 3481, 3482, 3484, 3485, 3486
• DMaaST: 3395, 3409, 3410

Loki correlation

For every one of those 11 UALs, the receiving cores logged Finalization recovery inbox rejected <ual>: capacity at the corresponding publish time. A bounded UAL query produced 22 capacity-rejection records for the 11 current-run assets (two receiving cores per asset); UAL 3409 also showed a background replay fetch failed.

Across the full last-12-hour Base window:

sum by (service_instance_id) (
  count_over_time(
    {service_instance_id=~"SBB|DMaaST|umanitek|EG - Luigi"}
      |= "Finalization recovery inbox rejected"
    [12h]
  )
)

Core	Rejections
EG - Luigi	167
umanitek	152
Total	319

No DKG startup, shutdown, restart, SIGTERM, auto-update, or restart markers were present for these Base cores in the same 12-hour window, so this is not a restart artifact.

The recovery backlog was also unhealthy: Finalization recovery background replay failed ...: fetch failed appeared 987 times across the four Base cores in the same window. That explains why capacity remained occupied, but it does not make silently consuming new finalized envelopes safe.

Source-level cause

At tag v10.0.12:

1. packages/agent/src/finalization-recovery-sqlite-policy.ts:6-10 bounds the live inbox to 128 entries globally, 32 per peer, and 64 per context graph.
2. packages/agent/src/finalization-recovery-sqlite-store.ts:139-140 returns { status: 'capacity' } when that bound is reached.
3. packages/agent/src/finalization-recovery.ts:300-304 turns a failed receive() into return true: the envelope is considered consumed, and the comment explicitly says Oxigraph is left untouched.

The anti-DoS bound and fail-closed behavior are correct security properties. The bug is that a valid finalized envelope rejected only because of temporary capacity has no durable/resumable handoff, while the transport/caller is told it was consumed.

Expected behavior

A capacity-full recovery inbox must remain bounded and fail closed, but it must not silently lose a valid finalization. Capacity exhaustion should produce an explicit retryable outcome and a bounded re-delivery/resume path once capacity becomes available, or retain a minimal durable pending marker that is sufficient to recover the envelope without admitting unbounded payload state.

Acceptance criteria

• A valid graph-scoped finalization arriving at inbox capacity is not reported as successfully consumed unless it is durably recoverable.
• The sender/caller receives a retryable capacity outcome, or the receiver durably records enough bounded state to resume.
• When backlog drains, the rejected KA is materialized without daemon restart or manual re-subscribe.
• Finalization anti-DoS limits remain enforced; no unbounded in-memory or SQLite queue is introduced.
• A regression fills the 128-entry live inbox, sends another finalized KA, drains capacity, and proves that KA becomes readable by UAL.
• A publish acceptance test proves every StorageACK core that acknowledged the KA eventually exposes the verified triples, or returns an explicit non-ack/retryable outcome.
• Metrics distinguish live-entry capacity, capacity rejections, replay progress, and permanently unrecoverable entries.

Related

• Investigate recurring SWM store scheduler queue wait timeouts on mainnet #1990 — store scheduler / Blazegraph pressure
• Base publish ACKCollector retries on /dkg/10.0.2/storage-ack protocol negotiation failures #1987 — ACK protocol negotiation and quorum fragility
• finalization recovery #1924 post-review architecture and test-hardening follow-ups #1957 — finalization recovery architecture and quota hardening follow-ups

[branarakic]

Kept open by a pre-close audit (25 Aug 2026). This came up in a sweep of issues whose fixes shipped via testnet-canary and were therefore never auto-closed. Most were closed; this one was held back because the shipped work does not cover all of its acceptance criteria.

AC1-AC5 and AC7 are genuinely met, but AC6 is not delivered anywhere in the tree: there is no publish acceptance test proving that every StorageACK core which acknowledged a KA eventually exposes the verified triples (or returns an explicit non-ack/retryable outcome). That criterion is not a coverage nicety here - it restates the issue's entire reproduction evidence (Jenkins reporting 58/66 publishes successful while 11 UALs returned no triples on any acknowledged core), so nothing on main demonstrates the reported symptom is gone. Secondary, and the reason the remaining gap matters in production: promotePending only promotes when hasFinalizationRecoveryCapacity is true (finalization-recovery-sqlite-store.ts:327), and stuck RECEIVED live entries are only pruned after rawTtlMs = 7 days (finalization-recovery-sqlite-policy.ts:15,116-118). In the incident's actual condition - 987 'background replay failed: fetch failed' events pinning the 128 live slots - the new 1024-entry deferred spool buys bounded headroom rather than the drain AC3 assumes. Close only after an end-to-end publish acceptance run, or re-scope AC6 into its own issue by agreement with the reporter.