Conversation
xtrusia
marked this pull request as draft
August 18, 2026 01:29
xtrusia
force-pushed
the
fix/lp2163212-hypervisor
branch
from
August 18, 2026 06:45
6c38e2d to
040eb07
Compare
xtrusia
marked this pull request as ready for review
August 19, 2026 05:10
xtrusia
force-pushed
the
fix/lp2163212-hypervisor
branch
4 times, most recently
from
August 25, 2026 10:02
ba121b9 to
d202e4f
Compare
xtrusia
force-pushed
the
fix/lp2163212-hypervisor
branch
from
September 2, 2026 06:49
d202e4f to
47e59a7
Compare
Preserve the FQDN returned by MAAS so downstream removal steps can distinguish it from the short cluster node name. Partial-Bug: LP: #2163212 Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
Remove Nova services and Neutron agents by the MAAS FQDN after the target MicroOVN unit is gone. Verify that the control-plane records no longer exist so retrying node removal cannot silently leave stale state. Partial-Bug: LP: #2163212 Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
xtrusia
force-pushed
the
fix/lp2163212-hypervisor
branch
from
September 2, 2026 06:54
47e59a7 to
1f83e6e
Compare
hemanthnakkina
requested review from
ahmad-can,
gboutry,
hemanthnakkina and
hmlanigan
and removed request for
gboutry and
hemanthnakkina
September 4, 2026 05:34
xtrusia
force-pushed
the
fix/lp2163212-hypervisor
branch
from
September 4, 2026 08:19
1f83e6e to
56ee225
Compare
Raven-182
self-requested a review
September 4, 2026 22:42
hemanthnakkina
requested changes
Sep 7, 2026
xtrusia
force-pushed
the
fix/lp2163212-hypervisor
branch
from
September 8, 2026 10:03
e4e80ef to
f00aad0
Compare
hemanthnakkina
previously approved these changes
Sep 16, 2026
hmlanigan
previously approved these changes
Sep 16, 2026
hmlanigan
left a comment
There was a problem hiding this comment.
Please add QA steps to the PR description.
Please update the last two commit messages to include why the change is needed.
xtrusia
dismissed stale reviews from hmlanigan and hemanthnakkina
via
September 19, 2026 02:55
b39f45b
Removing a cinder-volume Juju unit leaves its service records in Cinder, where they still point to a node no longer in the cluster. Disable matching services before unit removal, then delete their records through a healthy Cinder control-plane unit and verify that none remain. Partial-Bug: LP: #2163212 Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
Removing a MicroCeph Juju unit can leave its DB-backed OSDs in Ceph after the node is gone. Remove those OSDs before deleting the unit. Stop when CRUSH contains target OSDs absent from MicroCeph, because that orphaned state needs separate recovery. Closes-Bug: LP: #2163212 Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
Forced node removal still aborts on Cinder client errors because the Cinder cleanup steps do not receive the force flag. Pass it to both steps so their client errors follow Nova and Neutron's force behavior, while confirmed remaining service records still fail cleanup. Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
MicroCeph disk removal can leave an empty CRUSH host bucket, so a removed node still appears in Ceph's placement tree. Remove matching short-name and FQDN buckets after their OSDs are gone, and verify the buckets disappear even when retrying with no disks left. Partial-Bug: LP: #2163212 Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
Reapplying the MicroOVN plan after node removal reuses saved machine IDs and can try to place units on a removed Juju machine. Recompute MicroOVN and token distributor placement from the remaining nodes. Clear empty architecture lists so legacy Terraform variables cannot restore removed machines. Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
xtrusia
force-pushed
the
fix/lp2163212-hypervisor
branch
from
September 19, 2026 03:16
b39f45b to
2d43874
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MAAS node removal used the short hostname to clean up services that may be registered with the machine FQDN.
It could also remove Juju units without cleaning up their Cinder service records or MicroCeph OSDs.
This left stale state after the node was removed.
Preserve the FQDN returned by MAAS and use it to remove Nova services and Neutron agents after the target MicroOVN unit has stopped. Verify that no exact-host records remain.
Disable matching Cinder volume services before removing the
cinder-volumeunit. Remove their service records through a healthy Cinder control-plane unit afterward, matching both the short hostname and FQDN.Remove DB-backed MicroCeph OSDs before removing the MicroCeph unit.
Compare the MicroCeph database with the CRUSH tree before making changes and stop removal when the two sources disagree.
Force removal passes the supported MicroCeph safety flags without hiding cleanup failures.
Closes-Bug: LP: #2163212
QA steps
Both commands stopped at the expected two-node replica and three-OSD safety checks, respectively:
Only in this disposable two-node lab, I removed the target OSD with MicroCeph's safety-check override, then retried:
juju ssh -m openstack-machines microceph/3 -- sudo microceph disk remove osd.3 --timeout 1800 --bypass-safety-checks --confirm-failure-domain-downgrade sunbeam cluster remove --force --no-hints "$TARGET"The final removal exited 0.
pr902-node04was absent from Sunbeam, Juju, Nova, Neutron, Cinder, MicroCeph disks, and the CRUSH tree.The empty CRUSH host was removed.
MicroOVN reapply kept only machine
0and token distributor0.The surviving MicroOVN unit and
osd.1stayed active.For CRUSH map removal, redeployed them. with above, delete osd not allowed because of min osd count.
The cluster had four OSDs: three on pr902-node03 and one on pr902-node04.
sunbeam cluster remove --force --no-hints "$TARGET"
The command exited 0 without manual OSD removal or --bypass-safety-checks.
The target OSD and CRUSH host were removed.