feat: backup and restore (vault + mysql) by Deezzir · Pull Request #876 · canonical/snap-openstack

Deezzir · 2026-07-06T05:28:28Z

Overview

Adds sunbeam backup, sunbeam list-backups, and sunbeam restore commands for stateful workloads in the OpenStack model.
Introduces a component-driven backup/restore framework (BackupComponent) so command flow is generic while component specifics (target resolution, parse/list/restore plans) stay in one registry.
Supports MySQL and Vault, including backup inventory listing, restore execution, and manifest output.

Automation

In addition to the new commands, I am working on an experimental feature to automate the deployment and configuration of the s3-integrator charm for each MySQL and Vault component detected in the cluster. The branch is currently based on the current changes.

See: #890

The feature, at a minimum, deploys the s3-integrator for each target application, but the operator can choose to interactively set up the S3 location (single for all integrators).

Prerequisites

The current changes are blocked by the sunbeam-charms feature introduced in the PR, adding the pause/resume actions for the control plane services, like keystone, cinder, etc. The feature allows stopping the control-plane service's communication with the corresponding databases, so the restore can complete without concurrent writes.

QA

The new commands were validated end-to-end on a new Sunbeam Cluster (MAAS) on the personal SE-Cloud.

Demo

Backup

List Backups

Restore

Deezzir · 2026-07-20T06:39:21Z

+            params[self.component.backup_id_param] = latest
+
+        try:
+            tenacity.Retrying(


I found a race condition during a MySQL component restore. After the corresponding control-plane service is paused, there is a brief window during which MySQL may have dangling client connections, so the restore action will fail to close them. Having a retry here solves the issue, but I would like to come up with something more robust and reliable. Open to suggestions

issue: I find this aggressive retry policy a bit concerning.

Was this tested with applications unpaused?

gboutry

I think this PR needs another pass with a more holistic view of the backup and restore workflows.

There is a lot of indirection, which makes the operational sequence and its safety properties difficult to reason about. In particular, BackupComponent does not appear to simplify the implementation: component-specific behavior still leaks into generic steps, especially around action parameters, point-in-time restore, pause/resume coordination, and failure recovery.

The force handling illustrates the problem. It is propagated through several layers without a clear definition of what it is supposed to bypass. In some paths it has no effect, while in others it becomes an action parameter even when the target charm does not support that parameter. This makes --force both unreliable and capable of breaking otherwise valid operations.

Before adding more fixes, I think we should simplify the design around explicit MySQL and Vault workflows, define the guarantees for backup, restore, rollback, and force behavior, and then test those workflows against the actual charm action contracts and supported database topologies. The current mock-heavy tests pass while still encoding assumptions that are invalid with Juju.

gboutry · 2026-07-20T14:23:40Z

+            params[self.component.backup_id_param] = latest
+
+        try:
+            tenacity.Retrying(


issue: I find this aggressive retry policy a bit concerning.

Was this tested with applications unpaused?

gboutry · 2026-07-20T14:26:52Z

+    return sorted(successful, key=lambda b: b.backup_id)[-1].backup_id
+
+
+def _api_app_for_mysql(app_name: str) -> str:


The default MySQL topology is single MySQL app, and if we follow this Pull request, then, to which application is mysql tied to?

Addressed in 80c3324

Deezzir · 2026-07-22T05:37:58Z

@gboutry I've refactored the logic:

Explicit VaultBackupComponent and MySQLBackupComponent classes that encapsulate the per-component backup/restore logic.
Cleaned up stale step inputs (parameters)
Removed tenacity retry in the Restore step; to fight the race condition, the MySQL restore plan now scales all mysql-router apps to clean up the stale client connections and proceed to restore.
Cleaned up generic exceptions.
Added logic to handle multi/single MySQL cluster topology

The 80c3324 commit should address all comments and change requests above

Deezzir · 2026-08-19T15:37:29Z

@hemanthnakkina I don't usually resolve the comments by myself, and they are from @gboutry. Should I resolve?

hemanthnakkina · 2026-08-21T03:38:02Z

ly resolve the comments by myself, and th

Atleast leave a comment against each conversation thread so that we know the comments are handled

Deezzir · 2026-08-21T17:39:20Z

@hemanthnakkina I've tried to handle requests in the latest commit, #876 (comment)

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

hemanthnakkina

LGTM

gboutry

Two restore defects found at this revision; details inline. The 92 targeted backup/restore unit tests passed using the project's native uv invocation. Separate mocked reproductions confirmed the uncaught exception paths and the stale-unit backup lookup.

gboutry · 2026-09-07T08:51:10Z

+            params = self.component.restore_params(
+                self.jhelper,
+                self.target,


[P2] Use the refreshed leader for the backup lookup

The leader is refreshed at line 467, but restore_params() still receives the original self.target. For a latest-backup restore, latest_backup_params() runs list-backups on that original target.unit. If scale-down removed the previously selected leader, backup lookup targets a removed unit and fails before the restore action, which correctly uses the refreshed leader. Pass a target containing the refreshed leader when resolving backup parameters. A mocked leader-change reproduction confirmed list-backups targets the old unit while restore targets the refreshed leader.

Addressed in ee47529

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

hemanthnakkina

Overall LGTM, minor comments inline

gboutry

Is there a plan for acceptance suite test integration?

hmlanigan

Putting a hold on this PR until we resolve the feature freeze exception question, acceptance tests and documenation.

Deezzir · 2026-09-18T15:50:13Z

@hmlanigan @gboutry I don't completely understand what is being requested and why was it put on hold

gboutry self-requested a review July 9, 2026 11:45

hmlanigan requested a review from freyes July 10, 2026 19:06

Deezzir force-pushed the feat/backup-restore branch 21 times, most recently from 0aab721 to 7338fbe Compare July 20, 2026 05:09

Deezzir mentioned this pull request Jul 20, 2026

feat: disaster-recovery experimental feature #890

Open

Deezzir commented Jul 20, 2026

View reviewed changes

Deezzir force-pushed the feat/backup-restore branch from c1983b3 to 434862a Compare July 20, 2026 07:05

Deezzir marked this pull request as ready for review July 20, 2026 07:06

gboutry requested changes Jul 20, 2026

View reviewed changes

gboutry reviewed Jul 21, 2026

View reviewed changes

Comment thread sunbeam-python/sunbeam/steps/backup_restore.py Outdated

Deezzir force-pushed the feat/backup-restore branch from 80c3324 to 9ddff30 Compare August 29, 2026 22:08

Deezzir added 11 commits September 2, 2026 15:21

feat: backup command (vault + mysql)

343df3f

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

tests(backup): add unit tests

c668598

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

feat: restore command (vault + mysql)

033b4e8

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

tests(restore): add unit tests

b051988

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

refactor: single module for backup/restore

be3b730

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

fix: use OPENSTACK_MODEL

2f2c2d2

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

refactor: backup/restore component agnostic wrappers

cf656ba

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

refactor: centralize unresolved target handling in list inventory flow

98c3f59

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

fix: restore safety

7d311bf

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

fix: restore retry to allow resources to settle

367ad28

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

refactor: explicit BackupComponent registry class

1e32c98

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir force-pushed the feat/backup-restore branch from 9ddff30 to 1e32c98 Compare September 2, 2026 19:21

hemanthnakkina previously approved these changes Sep 4, 2026

View reviewed changes

gboutry reviewed Sep 7, 2026

View reviewed changes

fix: restore error handling & refresh action target

ee47529

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir dismissed hemanthnakkina’s stale review via ee47529 September 8, 2026 18:15

Deezzir requested review from gboutry and hemanthnakkina September 9, 2026 16:34

hemanthnakkina requested changes Sep 16, 2026

View reviewed changes

Comment thread sunbeam-python/sunbeam/steps/backup_restore.py Outdated

Comment thread sunbeam-python/sunbeam/steps/backup_restore.py Outdated

Comment thread sunbeam-python/sunbeam/steps/backup_restore.py Outdated

fix: handle backup action failures and timeouts

d50e41d

hemanthnakkina self-requested a review September 16, 2026 04:54

hemanthnakkina approved these changes Sep 16, 2026

View reviewed changes

hemanthnakkina requested a review from hmlanigan September 18, 2026 00:31

gboutry approved these changes Sep 18, 2026

View reviewed changes

hmlanigan requested changes Sep 18, 2026

View reviewed changes

		return sorted(successful, key=lambda b: b.backup_id)[-1].backup_id


		def _api_app_for_mysql(app_name: str) -> str:

Conversation

Deezzir commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Automation

Prerequisites

QA

Demo

Backup

List Backups

Restore

Uh oh!

Deezzir Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

gboutry Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

gboutry left a comment

Choose a reason for hiding this comment

Uh oh!

gboutry Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

gboutry Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Deezzir Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Deezzir commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Deezzir commented Aug 19, 2026

Uh oh!

hemanthnakkina commented Aug 21, 2026

Uh oh!

Deezzir commented Aug 21, 2026

Uh oh!

hemanthnakkina left a comment

Choose a reason for hiding this comment

Uh oh!

gboutry left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gboutry Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

Deezzir Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hemanthnakkina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gboutry left a comment

Choose a reason for hiding this comment

Uh oh!

hmlanigan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Deezzir commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Deezzir commented Jul 6, 2026 •

edited

Loading

Deezzir commented Jul 22, 2026 •

edited

Loading

gboutry left a comment •

edited

Loading

Deezzir Sep 8, 2026 •

edited

Loading

hmlanigan left a comment •

edited

Loading