Repository navigation
operator: settle each scale-down step before decommissioning the next - #1998
Merged
Merged
Conversation
Lowering statefulset.replicas by more than one broker deleted several broker Pods at once, and only the first was decommissioned. Since 1743490, reconcilePools no longer aborts the chain while CheckScale is false, so reconcileDecommission ran mid-scale. The previous step's terminating Pod was still LastPod, and its decommissioned broker looked already removed, so each pass patched away another broker without decommissioning it. Partitions with a majority on those brokers lost quorum. Fixes K8S-955.
andrewstucki
requested review from
RafalKorepta,
chrisseto,
gene-redpanda and
hidalgopl
as code owners
October 21, 2026 15:47
There was a problem hiding this comment.
🟢 Approval recommended
The focused guards prevent overlapping scale-down steps, and no unresolved blocking issues remain.
No open issues 🎉
What changed in this PR
Prevents multi-broker scale-downs from deleting brokers before they are decommissioned, avoiding partition quorum loss.
Changes:
- Gate scale-down steps on settled pool scaling in both controllers.
- Require current-generation Stable and Healthy conditions in the scaling test.
- Remove an outdated comment and document the fix.
| File | Description |
|---|---|
| operator/internal/controller/redpanda/redpanda_controller.go | Gate successive Redpanda scale-down steps. |
| operator/internal/controller/redpanda/redpanda_controller_test.go | Strengthen scaling success checks. |
| operator/internal/controller/redpanda/multicluster_controller.go | Apply the same gate to StretchCluster and remove an outdated comment. |
| .changes/unreleased/operator-Fixed-20261021-170000.yaml | Document the scale-down fix. |
Review effort: Balanced
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
|
This PR is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
hidalgopl
approved these changes
Oct 27, 2026
This was referenced Oct 27, 2026
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation and see the Github Action logs for details |
andrewstucki
added a commit
that referenced
this pull request
Oct 27, 2026
…#1998) (#2024) Lowering statefulset.replicas by more than one broker deleted several broker Pods at once, and only the first was decommissioned. Since 1743490, reconcilePools no longer aborts the chain while CheckScale is false, so reconcileDecommission ran mid-scale. The previous step's terminating Pod was still LastPod, and its decommissioned broker looked already removed, so each pass patched away another broker without decommissioning it. Partitions with a majority on those brokers lost quorum. Fixes K8S-955. (cherry picked from commit fde0a00) Co-authored-by: Andrew Stucki <andrew.stucki@redpanda.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lowering statefulset.replicas by more than one broker deleted several broker Pods at once, and only the first was decommissioned. Since 1743490, reconcilePools no longer aborts the chain while CheckScale is false, so reconcileDecommission ran mid-scale. The previous step's terminating Pod was still LastPod, and its decommissioned broker looked already removed, so each pass patched away another broker without decommissioning it. Partitions with a majority on those brokers lost quorum.
Fixes K8S-955.