Allow capping data to N blocks with API retention NET-896#77
Allow capping data to N blocks with API retention NET-896#77kalabukdima wants to merge 3 commits into
Conversation
Add an optional `max_blocks` to the Api config that trims the tail to keep at most `max_blocks` from the head.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { |
There was a problem hiding this comment.
All tests could be probably shrink-ed probasbly just to two. What would be more valuable to test is an actual capping behaviour.
| data_sources: Vec<ReqwestDataClient>, | ||
| // Static safety cap on retained blocks for Api-controlled datasets: even | ||
| // if the portal stops advancing the floor, the tail is trimmed to keep at | ||
| // most this many blocks behind the tip. `None` means grow indefinitely. |
There was a problem hiding this comment.
I think it would be valuable to mention that it's not a hard limit, but rather soft limit, as it may happen that the first chunk has accumulated a lot of blocks, which would go above the max_blocks limit. And example: max_blocks = 100K, first chunk has [0..150K], when the trigger fires at 250K, we get window 0..250K which is 2.5x times larger than the expected max_blocks limit.
| Head(u64), | ||
| // Retention is set dynamically from the portal | ||
| Api, | ||
| // Retention is set dynamically from the portal. `max_blocks`, if set, caps |
There was a problem hiding this comment.
let's ask Claude to trim those verboose comments.
define-null
left a comment
define-null
left a comment
There was a problem hiding this comment.
Approving this PR, as max_blocks seems to be a best_effort thing, but in my view it will be hard to reason about with this soft limit approach.
2 participants
Goal
If the archival data got stuck, the storage grows until running out of disk space leading to unpleasant consequences.
We can't reliably prevent it from happening. But we can sacrifice older blocks to keep serving the real-time data. Much less users will be affected by a gap than by a stale head.
Config changes