Skip to content

Enforce durability placement during CSI staging - #646

Open
Hrishikesh Kashyap (Hrishikeshkashyap) wants to merge 4 commits into
Azure:mainfrom
Hrishikeshkashyap:fix/csi-durability-enforcement
Open

Hrishikesh Kashyap (Hrishikeshkashyap) wants to merge 4 commits into
Azure:mainfrom
Hrishikeshkashyap:fix/csi-durability-enforcement

Conversation

@Hrishikeshkashyap

@Hrishikeshkashyap Hrishikesh Kashyap (Hrishikeshkashyap) commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Durability placement remains enforceable after binding and during node staging, rather than relying only on Pod admission.

Changes

  • Persist the effective failover mode and initial owner during CreateVolume.
  • Retain AccessibleTopology for durability volumes while preserving availability-mode recovery.
  • Resolve PVs by indexed CSI driver and volume handle, including when the PV name differs from the logical volume name.
  • Resolve durability ownership from retained PV topology when owner annotations are unavailable, and reject wrong-owner or missing-owner staging.
  • Reuse existing durability LVs without allocating replacements; preserve availability-mode create-if-missing recovery.
  • Use PV storage capacity when private capacity metadata is missing, including for static availability PVs.
  • Permit staging of existing LVs during a pending resize, while preserving controller provisioning size validation and upper-limit checks.
  • Keep orphan cleanup consistent with staging when selected-node is empty, falling back to the initial owner.
  • Persist the actual expanded LV capacity on the PV when available, and preserve expansion for volumes without a PV.

Tests

Regression coverage includes CSI identity lookup, topology and ownership fallback, static PV capacity, pending resize, orphan cleanup, expansion annotation persistence, and PV-less expansion.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Both ownership fallbacks ignore the driver's actual CSI topology key, breaking resolution from retained PV topology.

2 open findings
What changed in this PR

Enforces durability ownership beyond Pod admission by retaining PV topology and requiring existing storage during CSI staging.

Changes:

  • Persists failover policy and initial ownership during provisioning.
  • Adds staging ownership checks and an existing-only volume recovery path.
  • Extends tests and documents durability behavior.
File Description
internal/​webhook/​hyperconverged/​handler.go Adds retained-topology placement fallback.
internal/​webhook/​hyperconverged/​handler_unit_test.go Tests topology fallback.
internal/​csi/​node/​node.go Enforces ownership and existing-only staging.
internal/​csi/​node/​node_test.go Tests durability staging and failures.
internal/​csi/​core/​lvm/​node.go Exposes existing-only volume preparation.
internal/​csi/​core/​lvm/​lvm.go Prevents allocation when existing storage is required.
internal/​csi/​core/​lvm/​lvm_test.go Tests reuse and missing-volume behavior.
internal/​csi/​core/​interfaces.go Adds shared policies and existing-only interface.
internal/​csi/​core/​fake.go Tracks recovery-path calls for tests.
internal/​csi/​controller/​controller.go Persists policy and retains durability topology.
internal/​csi/​controller/​controller_test.go Tests durability topology retention.
docs/​user-guide.md Explains durability guarantees.
docs/​design/​webhooks.md Documents scheduling and staging enforcement.
docs/​design/​pv-recovery.md Distinguishes recovery policies.
docs/​architecture.md Summarizes durability enforcement.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal/csi/node/node.go Outdated
Comment thread internal/webhook/hyperconverged/handler.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Topology fallback can authorize excluded nodes, and metadata lookup can reject valid pre-provisioned durability volumes.

2 open findings
2 resolved since last review

🧠 Review effort: Balanced

Comment thread internal/csi/node/node.go Outdated
Comment thread internal/csi/node/node.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved identity resolution, static-volume capacity, and topology-label issues can select unsafe recovery behavior or block valid workloads.

3 open findings
2 resolved since last review

🧠 Review effort: Balanced

Comment thread internal/csi/node/node.go Outdated
Comment thread internal/csi/node/node.go
Comment thread internal/webhook/hyperconverged/handler.go Outdated
@Hrishikeshkashyap
Hrishikesh Kashyap (Hrishikeshkashyap) force-pushed the fix/csi-durability-enforcement branch 2 times, most recently from e5f9092 to bb7d786 Compare October 16, 2026 17:41
@Hrishikeshkashyap

Copy link
Copy Markdown
Contributor Author

Replaced the full cached-PV scan during staging with a controller-runtime field index keyed by CSI driver and volume handle. The index is registered during driver manager setup, and staging retrieves only matching PVs while preserving duplicate-identity rejection. Added tests for index registration, identity filtering, updates, and existing ambiguity checks.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Data-preservation behavior needs human validation, with unresolved staging and integration-test regressions.

2 open findings
3 resolved since last review

🧠 Review effort: Balanced

Comment thread internal/webhook/hyperconverged/handler.go
Comment thread internal/csi/node/node.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cleanup can delete a valid durability volume, and availability-mode static imports can regress during staging.

2 open issues
2 resolved since last review

Review effort: Balanced

Comment thread internal/csi/node/node.go
Comment thread internal/csi/node/node.go Outdated
@Hrishikeshkashyap
Hrishikesh Kashyap (Hrishikeshkashyap) force-pushed the fix/csi-durability-enforcement branch 2 times, most recently from 2d37563 to a74153a Compare October 19, 2026 06:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved ownership tracking and topology matching defects can expose stale data or prevent valid durability staging.

2 open issues
4 resolved since last review

Review effort: Balanced

Comment thread internal/csi/node/node.go
Comment thread internal/csi/node/node.go
@Hrishikeshkashyap Hrishikesh Kashyap (Hrishikeshkashyap) changed the title Enforce durability ownership during CSI staging Address review comments for durability staging Oct 19, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Storage ownership and recovery changes need human validation backed by verified regression tests and CSI conformance results.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Data-preservation changes span provisioning, staging, and cleanup and require human review backed by executable regression results.

No open issues 🎉

Review effort: Balanced

@Hrishikeshkashyap Hrishikesh Kashyap (Hrishikeshkashyap) changed the title Address review comments for durability staging Enforce durability placement during CSI staging Oct 22, 2026
@jmclong
jmclong (jmclong) requested a balanced review from Copilot October 22, 2026 14:15

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Admission and cache-scaling issues remain unresolved, and mixed-version rollout and rollback require human validation.

2 open issues

Review effort: Balanced

Comment thread internal/csi/node/node.go
Comment thread internal/webhook/hyperconverged/handler.go Outdated
@jmclong

Copy link
Copy Markdown
Contributor

Please add durability-mode E2E coverage for the new staging guarantees, using the existing E2E fixtures/infrastructure.

The shared E2E StorageClass does not set localdisk.csi.acstor.io/failover-mode: durability; the hyperconvergence test asserts preferred affinity, and the failover/expansion test skips hard-pinned PVs. This PR adds fake-backed durability cases but no changes under test/, so the green E2E jobs do not exercise the new policy end to end.

The important scenarios are:

  1. With an explicitly durability-mode StorageClass, write data and record a checksum, then recreate/restage the workload on its owner and verify the original data survives.
  2. Make the owner unavailable for scheduling and verify a replacement stays Pending rather than recovering on another node with an empty volume.
  3. In the disposable E2E storage fixture, make the backing LV absent and force staging on the correct owner; verify staging fails and the driver does not allocate an empty replacement LV.

These should exercise the actual provisioner -> persisted PV metadata/topology -> scheduler -> node plugin -> LVM path. The fake-backed tests are useful for branch/call-count assertions, but cannot establish those real scheduling and storage postconditions.

Retain hard topology and owner metadata for durability volumes. Reject wrong-node staging and require an existing LV instead of allocating an empty replacement, while preserving availability recovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Verify retained topology, owner validation, existing-only LVM staging, and missing-volume error handling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document retained topology, fixed ownership, and existing-only durability staging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the driver topology key for ownerless durability staging and webhook placement while retaining hostname compatibility. Cover driver topology, wrong-node rejection, and hostname fallback in regression tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Hrishikeshkashyap

Copy link
Copy Markdown
Contributor Author

Please add durability-mode E2E coverage for the new staging guarantees, using the existing E2E fixtures/infrastructure.

The shared E2E StorageClass does not set localdisk.csi.acstor.io/failover-mode: durability; the hyperconvergence test asserts preferred affinity, and the failover/expansion test skips hard-pinned PVs. This PR adds fake-backed durability cases but no changes under test/, so the green E2E jobs do not exercise the new policy end to end.

The important scenarios are:

  1. With an explicitly durability-mode StorageClass, write data and record a checksum, then recreate/restage the workload on its owner and verify the original data survives.
  2. Make the owner unavailable for scheduling and verify a replacement stays Pending rather than recovering on another node with an empty volume.
  3. In the disposable E2E storage fixture, make the backing LV absent and force staging on the correct owner; verify staging fails and the driver does not allocate an empty replacement LV.

These should exercise the actual provisioner -> persisted PV metadata/topology -> scheduler -> node plugin -> LVM path. The fake-backed tests are useful for branch/call-count assertions, but cannot establish those real scheduling and storage postconditions.

Added explicit durability tests for checksum-preserving restaging, Pending replacement when the owner is cordoned, and missing-LV rejection without recreation. Cross-compilation passes; cluster execution is still pending. Destructive coverage requires explicitly disposable Kind.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Durability recovery safety remains unverified because v0.4.1 can recreate missing durability LVs, while mixed-version rollout/rollback testing, disposable-cluster staging/resize validation, and exact runtime LVM2 provenance are unavailable.

1 open issue
1 resolved since last review

Review effort: Balanced

Copy link
Copy Markdown
Contributor

Follow-up idea: in the durability ownership fallback, consider fetching the Node as PartialObjectMetadata rather than a full Node, since this check only needs its name and labels. Also consider coalescing concurrent Node reads (for example, with singleflight keyed by node name) so simultaneous staging requests on the DaemonSet pod do not issue duplicate GETs. These can be handled in a follow-up PR, separately from this PR’s correctness changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants