Skip to content

Fix durability volume placement handling - #641

Closed
Hrishikesh Kashyap (Hrishikeshkashyap) wants to merge 6 commits into
Azure:mainfrom
Hrishikeshkashyap:fix/raven-durability-placement
Closed

Hrishikesh Kashyap (Hrishikeshkashyap) wants to merge 6 commits into
Azure:mainfrom
Hrishikeshkashyap:fix/raven-durability-placement

Conversation

@Hrishikeshkashyap

@Hrishikeshkashyap Hrishikesh Kashyap (Hrishikeshkashyap) commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Several durability placement gaps could let a Pod bypass storage restrictions, combine durability volumes that do not share a node, or recreate missing durability storage on another node. This PR makes durability placement mandatory from admission through staging while preserving availability-mode recovery.

  • ANDs storage placement with every existing non-empty required selector term instead of adding a bypassable OR term.
  • Gives durability constraints precedence regardless of mixed-volume order.
  • Intersects durability node sets and denies volumes with no common storage node.
  • Rejects missing ordinary PVCs while preserving generic-ephemeral and unbound WaitForFirstConsumer workflows.
  • Retains durability PV topology and owner metadata as a binding and scheduling backstop.
  • Rejects wrong-owner durability staging and prevents missing durability LVs from being recreated empty.
  • Preserves the existing availability-mode ownership transfer and empty-volume recovery behavior.

Type of Change

  • Bug fix
  • New feature
  • Code quality improvement
  • Other

Description

Admission and scheduling

  • Resolve each claim independently so one missing claim cannot discard another volume's placement.
  • Use <pod-name>-<volume-name> for generated generic-ephemeral claim names.
  • Keep empty required selectors and empty selector terms unsatisfiable.
  • Require all durability volumes to share a storage node.
  • Fall back to retained hard PV topology when owner annotations are unavailable.

CSI defense in depth

  • Persist the effective failover mode and initial durability owner during CreateVolume.
  • Keep AccessibleTopology for durability volumes while availability volumes continue using webhook-managed placement.
  • Validate the recorded owner before NodeStageVolume changes metadata or touches LVM.
  • Use an existing-only LVM ensure path for durability volumes and return NotFound when the LV is missing.

Behavior Change

A Pod that references a missing ordinary PVC is now denied. The caller must create the PVC and retry Pod creation so admission can resolve and enforce all storage placement constraints. Missing generated generic-ephemeral claims and initially unbound WaitForFirstConsumer claims remain supported.

How Has This Been Tested?

  • Focused controller, LVM, node, and webhook test binaries passed on Linux.
  • Added regression coverage for missing claims in both orders, generic-ephemeral naming, unbound claims, selector edge cases, mixed modes, same-node and conflicting durability volumes, topology retention, owner validation, and existing-only staging.
  • Verified with temporary driver and manager images on an AKS test cluster:
    • Missing ordinary PVCs were denied in both volume orders and no Pod was stored.
    • Missing generic-ephemeral claims preserved resolved durability placement.
    • Dynamically provisioned durability PVs retained hard node topology and initial-owner metadata.
    • Missing durability LVs returned CSI NotFound and were not recreated.
    • Mixed-mode, conflicting durability, same-node durability, and caller-affinity regressions passed.
  • Restored the test cluster to its original images and removed all temporary resources and image tags.

CSI Conformance

  • Reviewed CreateVolume and NodeStageVolume behavior against CSI specification v1.12.0.
  • Wrong-node and missing-owner staging use FailedPrecondition; missing durability metadata or LV uses NotFound.
  • No advertised capabilities or sanity-test skips changed.
  • The CSI sanity suite was not run; focused unit and live-cluster verification were used.

Checklist

  • The purpose and behavior change are documented.
  • Tests are included and updated for the code changes.
  • Architecture, webhook, recovery, and user documentation are updated.
  • Availability-mode recovery behavior is preserved.
  • No CSI capabilities or sanity-test skips were changed.

@Hrishikeshkashyap

Copy link
Copy Markdown
Contributor Author

Hrishikesh Kashyap (Hrishikesh Kashyap (@Hrishikeshkashyap)) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Microsoft"

Add the storage-node requirement to every non-empty required selector term instead of appending a bypassable OR term.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Give durability volumes precedence over availability volumes and intersect their resolved node sets. Deny admission when durability placement is missing or has no common node.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reject Pods that reference missing ordinary PVCs so placement is recalculated on a later CREATE. Preserve generic ephemeral and unbound WFFC claims while continuing to enforce resolved durability volumes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retain hard topology and owner metadata for durability volumes. Reject wrong-node staging and require an existing LV instead of allocating an empty replacement, while preserving availability recovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document affinity composition, multi-volume aggregation, unresolved PVC handling, CSI durability hardening, and live cluster verification results.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cover missing, unbound, and generic-ephemeral claims; affinity composition and volume aggregation; retained topology, owner validation, and existing-only LVM staging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jmclong
jmclong (jmclong) requested a balanced review from Copilot October 13, 2026 12:44
@Hrishikeshkashyap
Hrishikesh Kashyap (Hrishikeshkashyap) deleted the fix/raven-durability-placement branch October 13, 2026 12:44
@Hrishikeshkashyap Hrishikesh Kashyap (Hrishikeshkashyap) changed the title Fix durability placement handling (RAVEN-002-005) Superseded: fix durability volume placement handling Oct 13, 2026
@Hrishikeshkashyap
Hrishikesh Kashyap (Hrishikeshkashyap) restored the fix/raven-durability-placement branch October 13, 2026 12:47
@Hrishikeshkashyap Hrishikesh Kashyap (Hrishikeshkashyap) changed the title Superseded: fix durability volume placement handling Fix durability volume placement handling Oct 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

CSI topology-key handling and CreateVolume idempotency remain incorrect.

3 open findings
What changed in this PR

Hardens durability placement across admission, provisioning, and staging.

Changes:

  • Enforces intersected durability affinity and rejects unresolved PVC placement.
  • Preserves durability topology and prevents wrong-node or empty-volume recovery.
  • Expands regression tests and documentation.
File Description
internal/​webhook/​hyperconverged/​handler.go Enforces durability placement.
internal/​webhook/​hyperconverged/​handler_unit_test.go Adds affinity regressions.
internal/​webhook/​hyperconverged/​handler_test.go Extends webhook integration coverage.
internal/​csi/​node/​node.go Validates ownership during staging.
internal/​csi/​node/​node_test.go Tests durability staging.
internal/​csi/​core/​lvm/​node.go Adds existing-only ensure operation.
internal/​csi/​core/​lvm/​lvm.go Prevents replacement LV allocation.
internal/​csi/​core/​lvm/​lvm_test.go Tests existing-only behavior.
internal/​csi/​core/​interfaces.go Extends the node interface.
internal/​csi/​core/​fake.go Supports staging-path tests.
internal/​csi/​controller/​controller.go Persists failover metadata and topology.
internal/​csi/​controller/​controller_test.go Tests durability provisioning.
docs/​user-guide.md Documents durability behavior.
docs/​design/​webhooks.md Updates placement design.
docs/​design/​pv-recovery.md Describes mode-specific recovery.
docs/​architecture.md Updates architectural guarantees.

🧠 Review effort: Balanced


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +114 to +118
// Normalize missing or invalid values to the availability policy.
failoverMode := core.FailoverModeAvailability
switch req.GetParameters()[core.FailoverModeParam] {
case core.FailoverModeDurability:
failoverMode = core.FailoverModeDurability
Comment thread internal/csi/node/node.go
Comment on lines +570 to +573
if requirement.Key != corev1.LabelHostname ||
requirement.Operator != corev1.NodeSelectorOpIn {
continue
}
Comment on lines +445 to +448
if requirement.Key == KubernetesNodeHostNameLabel &&
requirement.Operator == corev1.NodeSelectorOpIn {
nodes = append(nodes, requirement.Values...)
}
@jmclong

Copy link
Copy Markdown
Contributor

Could you split this into separate PRs for affinity composition, multi-volume placement, claim resolution/admission policy, and CSI durability enforcement, each with its relevant tests and docs?

They share a durability-placement theme, but each fix or behavior change may need a different amount of discussion. For example, rejecting Pods with missing PVCs deserves separate consideration from fixing affinity composition. Splitting them would let us review and merge straightforward fixes without blocking them on changes that need more discussion.

@Hrishikeshkashyap

Hrishikesh Kashyap (Hrishikeshkashyap) commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Superseded by four focused pull requests as requested:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants