Add resumable scheduled piece deletion

[LexLuthr]

Summary

Fixes #283.

Moves scheduled piece deletion out of nextProvingPeriod() and into explicit, resumable storage-provider transactions.

Changes

• Add processPieceDeletions(uint256 setId, uint256 removalCount).
• Process removalCount entries from the tail of the verifier’s scheduled-removal queue.
• Retain the unprocessed queue prefix for later transactions.
• Support both legacy bitmap-backed datasets and compact PieceV2 removal markers.
• Clear only the processed legacy bitmap bits during partial deletion.
• Invalidate the active challenge before notifying the listener, preventing proofs against the mutated tree.
• Notify listeners through piecesRemoved(uint256 dataSetId, uint256 removedPieceCount) after PDP state is internally consistent.
• Emit the exact processed piece IDs through PiecesRemoved, split into chunks of at most 100 IDs.
• Make nextProvingPeriod() revert with the pending deletion count until the queue is empty.
• Allow an active dataset whose final piece was removed to close its proving lifecycle with zero leaves.
• Enforce the proof/deadline boundary in SimplePDPService.
• Append REMOVE_PROCESSED to PDPRecordKeeper.OperationType.
• Update public interfaces and all listener implementations and mocks.

This adds no per-call protocol batch cap or new storage fields. MAX_ENQUEUED_REMOVALS remains unchanged.

Tests

Added coverage for:

• partial count-based draining from the queue tail;
• zero and oversized removal counts;
• storage-provider permissions and deleted datasets;
• legacy bitmap preservation during partial processing;
• compact removal-marker preservation;
• legacy and compact storage compatibility;
• exact processed-ID events and event chunking;
• processed-count listener callbacks;
• listener-revert rollback of pieces, queue, leaf count, markers, and challenge state;
• active-challenge invalidation and stale-proof rejection;
• rollover blocking while deletions remain pending;
• fresh challenge creation after the queue is drained;
• additions entering challengeRange only during rollover;
• final-piece deletion and zero-leaf lifecycle closure;
• SimplePDPService proof/deadline timing rules.

Follow-up

FilecoinWarmStorageService must implement piecesRemoved(uint256,uint256) and move its bounded metadata cleanup out of nextProvingPeriod() in the corresponding filecoin-services change.

Curio must call processPieceDeletions() with a suitable removal count and retry with smaller counts when necessary. Those changes are outside this PR.

[wjmelements]

What happens if the removed pieces are part of the current challenge?

[Kubuxu]

What happens if the removed pieces are part of the current challenge?

You raise a very good point that I haven't considered yet because I focused only on the first skim.
processPieceDeletions needs to be callable only after proof is submitted but before nextProvingPeriod is called. We already ensure the latter, but we need to ensure the former.

[ZenGround0]

You raise a very good point that I haven't considered yet because I focused only on the first skim.
processPieceDeletions needs to be callable only after proof is submitted but before nextProvingPeriod is called. We already ensure the latter, but we need to ensure the former.

This callback appeared out of nowhere. FWSS does not implement it.

Unfortunately These two things cancel out. Because of the way things are structured I think the only way to determine if proof has already been submitted is by inspecting listener state. So we'll need to callback to check this for us.

[Kubuxu]

determine if proof has already been submitted

AFAIK, nextProvingPeriod creates a window of time during which the proof is valid to submit. As long as we are past that window, it should be safe to action removals. We don't need to know if the proof was submitted. Only whether we are past the valid proof submission window.

[ZenGround0]

AFAIK, nextProvingPeriod creates a window of time during which the proof is valid to submit. As long as we are past that window, it should be safe to action removals. We don't need to know if the proof was submitted. Only whether we are past the valid proof submission window.

Unfortunately from verifier perspective nextProvingPeriod only bumps up the challenge sampling epoch by the finality number. This is decoupled from the challenge window and proving period parameters which are listener recorded and enforced. Verifier does track the epoch of last proving but without knowing proving period and challenge window params that is not sufficient for figuring out whether we have proven "this period". Verifier doesn't understand that concept. It will be simplest and in line with existing solutions to similar problems to use a listener callback to decided if scheduling is ok.

The closest to a verifier-only solution we have is to inspect nextChallengeEpoch in Verifier state and assert it is in the future, i.e. allow removals after nextProvingPeriod. If we modify proving set before challenge sampling then we still prevent grinding. But grinding becomes more effective the closer we get to the challenge sampling epoch because of L1 forking so any solution would necessarily degrade grinding security and I wouldn't want to do that.

For what it's worth as time has gone on this design choice was more and more obviously wrong. We should have had proving period and challenge window params set per dataset in create message.

[Kubuxu]

You are right, I thought we were bounding the window.

Alternatively, what if we invalidate the challenge sampling epoch when removal happens?

[Kubuxu]

The latest commits introduce the mechanism I mentioned here: #297 (comment)
While preserving the callback, we need to align on the exact direction we are taking.

[Kubuxu]

SGTM I'm going to do one more pass tomorrow on fresh mind