CLDSRV-936 Lifecycle noncurrent expiration stalls when listing truncates on a bare master#6205
Open
nicolas2bert wants to merge 3 commits into
Open
Conversation
Contributor
Hello nicolas2bert,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Branches have divergedThis pull request's source branch To avoid any integration risks, please re-synchronize them using one of the
Note: If you choose to rebase, you may have to ask me to rebuild |
nicolas2bert
force-pushed
the
bugfix/CLDSRV-936/listLifecycleNonCurrents-baremaster
branch
from
fc1d50e to
1602dc3
Compare
Contributor
Incorrect fix versionThe
Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:
Please check the |
nicolas2bert
force-pushed
the
bugfix/CLDSRV-936/listLifecycleNonCurrents-baremaster
branch
2 times, most recently
from
8b35ba5 to
02aaafa
Compare
❌ 29 Tests Failed:
View the top 3 failed test(s) by shortest run time
View the full list of 7 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
| // a malformed (non-null) marker that decode rejects by returning an Error | ||
| const result = decodeVersionIdMarker('@@@bad@@@'); | ||
| assert(result instanceof Error); | ||
| assert.strictEqual(result.is.InvalidArgument, true); |
Contributor
There was a problem hiding this comment.
It could be better to compare value instead of boolean to see in failure message the actual vs expected value instread of false !== true
Suggested change
| assert.strictEqual(result.is.InvalidArgument, true); | |
| assert.strictEqual(result.message, 'InvalidArgument'); |
BourgoisMickael
approved these changes
Jun 24, 2026
jonathan-gramain
approved these changes
Jun 24, 2026
…tes on a bare master ++
Apply Prettier formatting to the four files touched by this branch so the prettier:diff CI check passes. Pure formatting, no logic changes.
nicolas2bert
force-pushed
the
bugfix/CLDSRV-936/listLifecycleNonCurrents-baremaster
branch
from
02aaafa to
046ca1a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug in one sentence
The lifecycle noncurrent listing returns the marker version-id-marker=null when a scan page stops on a "bare master" (an object with no internal versionId). CloudServer can't decode 'null', so the worker crashes. The bucket hasn't changed in months, so it crashes every cycle and the noncurrent versions are never expired.
How a bare master (null master) gets created
bucket created, not versioned yet.
PUT foo -> stored as a plain master key "foo", the value has NO versionId field (x-amz-version-id: "null"). <- this is the "bare master"
versioning ENABLED later on the bucket.
foo is NEVER re-PUT -> the bare master stays as-is, forever.
So a bare master = an object written before versioning, never re-PUT afterward. If it had been overwritten, the system would have converted it into a normal null version (id 99999...,isNull2:true), which is harmless. To trigger the bug the bare master never has to be overwritten (which is the case).
How the noncurrent listing works in metadata (bucketd)
The bucket-processor asks CloudServer for a DelimiterNonCurrent listing, which hits bucketd.
bucketd scans the keys in order, newest->oldest per key.
For each key:
the first version seen = the current one -> not returned, but its last-modified is kept in memory.
the following versions = noncurrent -> returned, with a staleDate (= the in memory last-modified).
The staleDate is the last-modified of the version that replaced it, ie the moment that version became noncurrent. It's this staleDate that gets compared to NoncurrentDays. A noncurrent is only returned if staleDate < beforeDate, ie old enough.
Two limits bound each page:
max-keys (default 1000) -> number of noncurrent versions returned.
maxScannedLifecycleListingEntries (default 10000) -> number of entries scanned, so that a bucket full of current objects but with few noncurrents still responds quickly.
When a limit is hit -> IsTruncated=true + a continuation marker (NextKeyMarker + NextVersionIdMarker), and backbeat re-requests the next page with those markers, until IsTruncated=false.
The NextVersionIdMarker is computed as getVersionId() || 'null'. And that's exactly where it goes wrong: on a bare master, getVersionId() is undefined -> the marker becomes the string 'null'.
Example of the issue:
scan# key kind marker becomes
1 bar master (regular) 982... (real id)
2 bar \0 982... current version 982...
3 bar \0 981... non-current -> expirable 981...
... ... ~10k entries, all real ids ...
10000 foo BARE master (no versionId) getVersionId() || 'null' = "null"
>>> STOP (scanned 10000): IsTruncated=true,
NextKeyMarker="foo", NextVersionIdMarker="null" <-- the poison marker
10001 baz ... (next page - never reached)
The noncurrent versions to delete are on the next page, behind that marker.
Condition to trigger the issue: the 10000th scanned entry is a bare master.
If it were a normal version/master, the marker would be a valid id and nothing would break.
NOTE: Only scan-limit truncation can produce 'null', max-keys truncation always stops on a real noncurrent -> real id.
Why it's a bug
The marker round-trip is asymmetric. 'null' is guarded on the way out, not on the way in:
Page 1 version-id-marker= -> returns NextVersionIdMarker:"null" (encode handle 'null' properly-> OK)
Page 2 version-id-marker=null-> decode("null") = Error (decode does NOT handle 'null' properly) -> encode(Error) -> TypeError -> CloudServer worker crash.
bucketd answers page 2 without any problem (HTTP 200). The crash is 100% on the CloudServer side, while re-encoding the version-id-marker=null from the request itself.
Why it stays stuck
The processor gives up after its retries and moves on to something else. But the versions to expire are on page 2, which crashes every time. The bucket doesn't change -> every cycle page 1 re-emits 'null', page 2 re-crashes.
Net progress = zero, indefinitely -> "noncurrent versions never removed".
the fix : treat version-id-marker=null as "no marker" instead of blindly decoding it.