fix(outbox): reclaim expired Processing messages in Entity Framework Core and Cosmos DB

[samtrion]

Closes #813

Summary

The Entity Framework Core and Cosmos DB outbox repositories now reclaim messages that stay in Processing longer than the processing lease, like the SQL Server, PostgreSQL, MySQL, SQLite and MongoDB providers already do. Without this, a graceful shutdown mid-batch, a cancellation between send and completion, a failure after the claim, or a crashed host left messages in Processing forever.

Changes

• Entity Framework Core: EntityFrameworkOutboxRepository takes IOptions<OutboxOptions> (registered by AddOutbox) and validates ProcessingLeaseTimeout > TimeSpan.Zero. The GetPendingAsync claim filter is widened to (Pending AND NextRetryAt due) OR (Processing AND UpdatedAt <= now - ProcessingLeaseTimeout). The cutoff is computed outside the expression tree. Because the claim filter is also the predicate of the bulk ExecuteUpdate (see decisions/2026-09-28-entityframework-outbox-claim-concurrency.md), a competing poller re-checks it against the committed row and fails, since the reclaim writes a fresh UpdatedAt. The tracking executors reject the same race through the Status/UpdatedAt concurrency tokens. No executor changes were needed, and the claim-concurrency guards from fix(entityframework): prevent double claims in bulk FetchAndMarkAsync under READ COMMITTED #883 are unchanged.
• Cosmos DB:
  • New CosmosDbOutboxOptions.ProcessingLeaseTimeout (default 5 minutes). This mirrors MongoDbOutboxOptions.ProcessingLeaseTimeout. CosmosDbOutboxOptionsValidator rejects values <= TimeSpan.Zero at host startup (ValidateOnStart), next to the PartitionKeyPath check; the repository constructor keeps its guard for direct construction.
  • The pending query adds OR (c.status = 1 AND c.updatedAt <= @leaseExpiredBefore). The IfMatchEtag claim keeps a reclaim exclusive. The query keeps its single-property ORDER BY c._ts, which needs no composite index under the default indexing policy. CosmosDbOutboxTests pass against the emulator with the new query.
  • ClaimMessagesAsync now returns the documents already claimed when a later patch fails with a CosmosException (for example 429). With nothing claimed yet, and on cancellation, the exception still propagates. Any document stranded that way is reclaimed once the lease expires.
• Docs:
  • ProcessIndividuallyAsync remarks: remaining messages are reclaimed after the lease expires.
  • OutboxOptions.ProcessingLeaseTimeout remarks: now name the providers that honor it.
  • Class remarks of both repositories.
  • Cosmos DB README options table: new ProcessingLeaseTimeout row (validated at startup).
  • Entity Framework Core README: new "Processing Lease Reclaim" section on configuring and sizing OutboxOptions.ProcessingLeaseTimeout.
  • OutboxEventTypeResolver.DeadLetterUnresolvableAsync remarks: no longer suggest that some providers do not reclaim expired leases.
  • ADR 2026-09-24-outbox-unresolvable-event-types.md: the line-58 consequence no longer describes the gap.
• New ADR (state: proposed, for the maintainer to accept in review): decisions/2026-09-28-entityframework-and-cosmosdb-outbox-lease-reclaim.md. It records the use of UpdatedAt as the lease timestamp (no schema change), where each provider reads the lease from, and the partial-claim behavior.

Bug hunt

Suspicion	Confirmed?	Test
EF GetPendingAsync never returns a message stuck in Processing past the lease	Yes (red on EF InMemory and EF SQLite before the fix)	OutboxTestsBase.Should_GetPendingAsync_Reclaim_Expired_Processing (all providers)
Cosmos DB GetPendingAsync never returns a message stuck in Processing past the lease	Yes (red on the Cosmos DB emulator before the fix)	same
A Processing message is reclaimed before its lease expires	No (green before and after, on all providers)	OutboxTestsBase.Should_GetPendingAsync_Keep_Processing_Within_Lease
Two overlapping EF pollers both reclaim the same expired message	No duplicate after the fix. Before the fix nothing was reclaimed at all (red: first.Count == 0)	EntityFrameworkOutboxClaimRaceTestsBase.GetPendingAsync_WhenReclaimsOverlap_ReturnsDisjointBatches (SQLite, PostgreSQL, SQL Server RCSI, MySQL/tracking executor)
Cosmos DB claim throws away already-claimed documents when a later patch is throttled (429)	Yes (red before the fix)	CosmosDbOutboxRepositoryClaimTests.GetPendingAsync_WhenLaterPatchIsThrottled_ReturnsMessagesClaimedSoFar
Cosmos DB claim swallows a failure when nothing was claimed	No (kept as a guard)	CosmosDbOutboxRepositoryClaimTests.GetPendingAsync_WhenFirstPatchIsThrottled_ThrowsCosmosException
Two Cosmos DB pollers both reclaim one expired document	No: the reclaim patch of a stale Processing document carries IfMatchEtag and is skipped on 412	CosmosDbOutboxRepositoryClaimTests.GetPendingAsync_WhenReclaimPatchThrowsPreconditionFailed_SkipsExpiredProcessingCandidate
EF repository ignores the configured OutboxOptions.ProcessingLeaseTimeout (hardcoded lease would pass the shared tests)	No (green: 30 s and 1 h leases both honored, reclaimed row gets UpdatedAt = now)	EntityFrameworkOutboxRepositoryInvariantTests.GetPendingAsync_Uses_configured_ProcessingLeaseTimeout
Cosmos DB accepts a non-positive ProcessingLeaseTimeout at host startup (only the scoped repository constructor threw, on first poll)	Yes (red before the fix)	CosmosDbOutboxOptionsValidatorTests.Validate_When_ProcessingLeaseTimeout_is_not_positive_fails

Acceptance criterion "after a graceful shutdown mid-batch, the remaining messages are dispatched once the lease expires": Should_GetPendingAsync_Reclaim_Expired_Processing covers it. The test claims 3 messages, completes 1, advances past the lease, and checks that exactly the other 2 are returned once and are not returned again.

Impact

• CosmosDbOutboxOptions gains a public ProcessingLeaseTimeout property (additive). OutboxOptions.ProcessingLeaseTimeout now also applies to the Entity Framework Core provider. There are no interface changes, so external IOutboxRepository implementers are not affected. They SHOULD reclaim expired leases the same way.
• Messages already stranded in Processing in existing EF/Cosmos DB installations are reclaimed on the first poll after the upgrade. They may be delivered again, which matches at-least-once semantics.
• A dispatch that takes longer than the lease can be delivered twice, the same as with the other providers. Choose the lease comfortably above the longest dispatch.
• No schema change. GetPendingCountAsync still counts only Pending.

Test evidence

• dotnet build Pulse.slnx -c Release: 0 errors, 0 warnings.
• Unit (NetEvolve.Pulse.Tests.Unit, net8.0/net9.0/net10.0): 7203 passed, 0 failed (after merging origin/main).
• Integration, local (net10.0) (earlier run, unchanged provider code since):
  • All *EntityFramework* outbox classes (InMemory, SQLite, and Docker-backed SQL Server, PostgreSQL, MySQL; outbox and claim-race tests): 271 tests. The first run had 7 failures at 0 ms that came from fixture startup on a heavily loaded machine. Those 7 tests passed on a targeted rerun (5/5 and 48/48, including all new reclaim, lease and race tests).
  • CosmosDbOutboxTests on the emulator: 52/52 passed, including the new reclaim tests.
  • SQLiteAdoNetOutboxTests: 51/51 passed.
• Red before the fix: EF InMemory and EF SQLite (Should_GetPendingAsync_Reclaim_Expired_Processing, GetPendingAsync_WhenReclaimsOverlap_ReturnsDisjointBatches), Cosmos DB emulator (Should_GetPendingAsync_Reclaim_Expired_Processing), the Cosmos DB partial-claim unit test, and the Cosmos DB lease validator test.
• CI (head 1e29379): all checks green, including Run Tests (unit + integration with Docker-backed providers), Code Formatting, Commitlint, CodeQL, NativeAOT, codecov/patch and codecov/project.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.13%. Comparing base (ed5c2dd) to head (1e29379).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #903      +/-   ##
==========================================
+ Coverage   96.07%   96.13%   +0.06%     
==========================================
  Files         274      274              
  Lines       11514    11527      +13     
  Branches     1072     1070       -2     
==========================================
+ Hits        11062    11082      +20     
+ Misses        232      229       -3     
+ Partials      220      216       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.