Skip to content

fix(ci): sds-elastic e2e setup tolerates already-consumed block devices#2684

Open
nevermarine wants to merge 1 commit into
mainfrom
fix/ci/sds-elastic-rerun-consumed-blockdevices
Open

fix(ci): sds-elastic e2e setup tolerates already-consumed block devices#2684
nevermarine wants to merge 1 commit into
mainfrom
fix/ci/sds-elastic-rerun-consumed-blockdevices

Conversation

@nevermarine

Copy link
Copy Markdown
Collaborator

Description

The sds-elastic e2e setup (configure-sds-elastic.sh + wait-sds-elastic.sh) waited for >= N BlockDevices with .status.consumable == true before creating the ElasticCluster. On a pipeline rerun the raw OSD disks are already consumed into sds-elastic OSD LVMVolumeGroups (sds-elastic-<ec>-osd-*), so they are no longer consumable and the wait looped until timeout:

Error:  Consumable blockdevices did not reach 6 in time

This PR makes the setup idempotent across restarts:

  • A BlockDevice now counts as an OSD candidate when it is either still consumable (fresh run) or already backing an sds-elastic OSD LVG (rerun), detected via .status.lvmVolumeGroupName matching ^sds-elastic-.*-osd- (override with ELASTIC_OSD_LVG_REGEX). Real provisioning state is used rather than the user label, so detection survives a BlockDevice rediscovery.
  • The app=elastic-osd labelling step (the ElasticCluster.spec.storage.blockDeviceSelector) uses the same predicate, so it re-labels already-consumed devices too — self-healing if a label was lost during rediscovery.
  • Shared predicate (ELASTIC_OSD_BD_SELECT) and regex live in wait-sds-elastic.sh and are reused by both scripts; readiness log/error messages renamed Consumable → OSD-candidate.

Why do we need it, and what problem does it solve?

Reruns of the nightly sds-elastic e2e pipeline (or a rerun after a partial provisioning) always failed at the block-device wait, because consumed disks never return to consumable. This unblocks pipeline restarts.

What is the expected result?

The block-device readiness check passes on:

  • fresh run — all disks consumable;
  • full rerun — all disks already consumed into OSD LVGs;
  • partial rerun — a mix of consumable and already-consumed disks.

Verified the jq predicate on fixtures (consumable + elastic-consumed counted, unrelated LVGs and plain non-consumable disks excluded) and bash -n on both scripts. A live-cluster check was not possible — the referenced e2e cluster had already rotated.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested (jq predicate + shell syntax); not user-facing.

Changelog entries

section: ci
type: fix
summary: sds-elastic e2e setup tolerates already-consumed block devices on pipeline rerun.
impact_level: low

On a pipeline rerun the raw OSD disks are already consumed into sds-elastic
OSD LVMVolumeGroups, so they are no longer .status.consumable == true and the
block-device readiness wait looped until timeout ("Consumable blockdevices did
not reach 6 in time").

Count a BlockDevice as an OSD candidate when it is either still consumable
(fresh run) or already backing an sds-elastic OSD LVG (rerun, detected via
.status.lvmVolumeGroupName). Apply the same predicate when labelling devices
with app=elastic-osd so the ElasticCluster blockDeviceSelector keeps matching
them, including after a partial run or a BlockDevice rediscovery that dropped
the label.

Signed-off-by: Maksim Fedotov <maksim.fedotov@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant