fix(ci): sds-elastic e2e setup tolerates already-consumed block devices#2684
Open
nevermarine wants to merge 1 commit into
Open
fix(ci): sds-elastic e2e setup tolerates already-consumed block devices#2684nevermarine wants to merge 1 commit into
nevermarine wants to merge 1 commit into
Conversation
On a pipeline rerun the raw OSD disks are already consumed into sds-elastic
OSD LVMVolumeGroups, so they are no longer .status.consumable == true and the
block-device readiness wait looped until timeout ("Consumable blockdevices did
not reach 6 in time").
Count a BlockDevice as an OSD candidate when it is either still consumable
(fresh run) or already backing an sds-elastic OSD LVG (rerun, detected via
.status.lvmVolumeGroupName). Apply the same predicate when labelling devices
with app=elastic-osd so the ElasticCluster blockDeviceSelector keeps matching
them, including after a partial run or a BlockDevice rediscovery that dropped
the label.
Signed-off-by: Maksim Fedotov <maksim.fedotov@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The sds-elastic e2e setup (
configure-sds-elastic.sh+wait-sds-elastic.sh) waited for>= NBlockDevices with.status.consumable == truebefore creating theElasticCluster. On a pipeline rerun the raw OSD disks are already consumed into sds-elastic OSDLVMVolumeGroups (sds-elastic-<ec>-osd-*), so they are no longerconsumableand the wait looped until timeout:This PR makes the setup idempotent across restarts:
consumable(fresh run) or already backing an sds-elastic OSD LVG (rerun), detected via.status.lvmVolumeGroupNamematching^sds-elastic-.*-osd-(override withELASTIC_OSD_LVG_REGEX). Real provisioning state is used rather than the user label, so detection survives a BlockDevice rediscovery.app=elastic-osdlabelling step (theElasticCluster.spec.storage.blockDeviceSelector) uses the same predicate, so it re-labels already-consumed devices too — self-healing if a label was lost during rediscovery.ELASTIC_OSD_BD_SELECT) and regex live inwait-sds-elastic.shand are reused by both scripts; readiness log/error messages renamedConsumable → OSD-candidate.Why do we need it, and what problem does it solve?
Reruns of the nightly sds-elastic e2e pipeline (or a rerun after a partial provisioning) always failed at the block-device wait, because consumed disks never return to
consumable. This unblocks pipeline restarts.What is the expected result?
The block-device readiness check passes on:
Verified the jq predicate on fixtures (consumable + elastic-consumed counted, unrelated LVGs and plain non-consumable disks excluded) and
bash -non both scripts. A live-cluster check was not possible — the referenced e2e cluster had already rotated.Checklist
Changelog entries