Skip to content

fix(maas): clean up node state before removal - #902

Open
xtrusia wants to merge 7 commits into
canonical:mainfrom
xtrusia:fix/lp2163212-hypervisor
Open

xtrusia wants to merge 7 commits into
canonical:mainfrom
xtrusia:fix/lp2163212-hypervisor

Conversation

@xtrusia

@xtrusia xtrusia commented Aug 18, 2026

Copy link
Copy Markdown

MAAS node removal used the short hostname to clean up services that may be registered with the machine FQDN.
It could also remove Juju units without cleaning up their Cinder service records or MicroCeph OSDs.
This left stale state after the node was removed.

Preserve the FQDN returned by MAAS and use it to remove Nova services and Neutron agents after the target MicroOVN unit has stopped. Verify that no exact-host records remain.

Disable matching Cinder volume services before removing the cinder-volume unit. Remove their service records through a healthy Cinder control-plane unit afterward, matching both the short hostname and FQDN.

Remove DB-backed MicroCeph OSDs before removing the MicroCeph unit.
Compare the MicroCeph database with the CRUSH tree before making changes and stop removal when the two sources disagree.
Force removal passes the supported MicroCeph safety flags without hiding cleanup failures.

Closes-Bug: LP: #2163212

QA steps

source admin-openrc
TARGET=pr902-node04
SURVIVING_CEPH_UNIT=microceph/2

sunbeam cluster list --format yaml
juju status -m openstack-machines
openstack compute service list --service nova-compute -f value -c Host -c State -c Status
openstack network agent list -f value -c Host -c 'Agent Type' -c Alive -c State
openstack volume service list -f value -c Binary -c Host -c Status -c State
juju ssh -m openstack-machines "$SURVIVING_CEPH_UNIT" -- sudo microceph disk list
juju ssh -m openstack-machines "$SURVIVING_CEPH_UNIT" -- sudo microceph.ceph osd tree

Both commands stopped at the expected two-node replica and three-OSD safety checks, respectively:

sunbeam cluster remove --no-hints "$TARGET"
sunbeam cluster remove --force --no-hints "$TARGET"

Only in this disposable two-node lab, I removed the target OSD with MicroCeph's safety-check override, then retried:

juju ssh -m openstack-machines microceph/3 -- sudo microceph disk remove osd.3 --timeout 1800 --bypass-safety-checks --confirm-failure-domain-downgrade
sunbeam cluster remove --force --no-hints "$TARGET"
sunbeam cluster list --format yaml
juju status -m openstack-machines
openstack compute service list --service nova-compute -f value -c Host -c State -c Status
openstack network agent list -f value -c Host -c 'Agent Type' -c Alive -c State
openstack volume service list -f value -c Binary -c Host -c Status -c State
juju ssh -m openstack-machines "$SURVIVING_CEPH_UNIT" -- sudo microceph disk list
juju ssh -m openstack-machines "$SURVIVING_CEPH_UNIT" -- sudo microceph.ceph osd tree
juju ssh -m openstack-machines "$SURVIVING_CEPH_UNIT" -- sudo microceph.ceph health detail

The final removal exited 0. pr902-node04 was absent from Sunbeam, Juju, Nova, Neutron, Cinder, MicroCeph disks, and the CRUSH tree.
The empty CRUSH host was removed.
MicroOVN reapply kept only machine 0 and token distributor 0.
The surviving MicroOVN unit and osd.1 stayed active.

For CRUSH map removal, redeployed them. with above, delete osd not allowed because of min osd count.

The cluster had four OSDs: three on pr902-node03 and one on pr902-node04.

sunbeam cluster remove --force --no-hints "$TARGET"

The command exited 0 without manual OSD removal or --bypass-safety-checks.
The target OSD and CRUSH host were removed.

@xtrusia
xtrusia marked this pull request as draft August 18, 2026 01:29
@xtrusia xtrusia changed the title fix(maas): remove stale OpenStack service records fix(maas): clean up node state before removal Aug 18, 2026
@xtrusia
xtrusia force-pushed the fix/lp2163212-hypervisor branch from 6c38e2d to 040eb07 Compare August 18, 2026 06:45
@xtrusia
xtrusia marked this pull request as ready for review August 19, 2026 05:10
@xtrusia
xtrusia force-pushed the fix/lp2163212-hypervisor branch 4 times, most recently from ba121b9 to d202e4f Compare August 25, 2026 10:02
@xtrusia
xtrusia force-pushed the fix/lp2163212-hypervisor branch from d202e4f to 47e59a7 Compare September 2, 2026 06:49
Preserve the FQDN returned by MAAS so downstream removal steps can
distinguish it from the short cluster node name.

Partial-Bug: LP: #2163212
Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
Remove Nova services and Neutron agents by the MAAS FQDN after the
target MicroOVN unit is gone. Verify that the control-plane records no
longer exist so retrying node removal cannot silently leave stale state.

Partial-Bug: LP: #2163212
Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
@xtrusia
xtrusia force-pushed the fix/lp2163212-hypervisor branch from 47e59a7 to 1f83e6e Compare September 2, 2026 06:54
@hemanthnakkina
hemanthnakkina requested review from ahmad-can, gboutry, hemanthnakkina and hmlanigan and removed request for gboutry and hemanthnakkina September 4, 2026 05:34
@xtrusia
xtrusia force-pushed the fix/lp2163212-hypervisor branch from 1f83e6e to 56ee225 Compare September 4, 2026 08:19
@Raven-182
Raven-182 self-requested a review September 4, 2026 22:42
Comment thread sunbeam-python/sunbeam/provider/maas/commands.py
@xtrusia
xtrusia force-pushed the fix/lp2163212-hypervisor branch from e4e80ef to f00aad0 Compare September 8, 2026 10:03
hemanthnakkina
hemanthnakkina previously approved these changes Sep 16, 2026
hmlanigan
hmlanigan previously approved these changes Sep 16, 2026

@hmlanigan hmlanigan left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add QA steps to the PR description.

Please update the last two commit messages to include why the change is needed.

@xtrusia
xtrusia dismissed stale reviews from hmlanigan and hemanthnakkina via b39f45b September 19, 2026 02:55
Removing a cinder-volume Juju unit leaves its service records in
Cinder, where they still point to a node no longer in the cluster.
Disable matching services before unit removal, then delete their
records through a healthy Cinder control-plane unit and verify that
none remain.

Partial-Bug: LP: #2163212
Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
Removing a MicroCeph Juju unit can leave its DB-backed OSDs in Ceph
after the node is gone. Remove those OSDs before deleting the unit.
Stop when CRUSH contains target OSDs absent from MicroCeph, because
that orphaned state needs separate recovery.

Closes-Bug: LP: #2163212
Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
Forced node removal still aborts on Cinder client errors because the
Cinder cleanup steps do not receive the force flag. Pass it to both
steps so their client errors follow Nova and Neutron's force behavior,
while confirmed remaining service records still fail cleanup.

Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
MicroCeph disk removal can leave an empty CRUSH host bucket, so a
removed node still appears in Ceph's placement tree. Remove matching
short-name and FQDN buckets after their OSDs are gone, and verify
the buckets disappear even when retrying with no disks left.

Partial-Bug: LP: #2163212
Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
Reapplying the MicroOVN plan after node removal reuses saved machine
IDs and can try to place units on a removed Juju machine. Recompute
MicroOVN and token distributor placement from the remaining nodes.
Clear empty architecture lists so legacy Terraform variables cannot
restore removed machines.

Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
@xtrusia
xtrusia force-pushed the fix/lp2163212-hypervisor branch from b39f45b to 2d43874 Compare September 19, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants