feat(backup): Add remote S3-compatible backup destinations#168
Merged
Conversation
Backups can now be mirrored to remote S3-compatible object storage (AWS S3, Cloudflare R2, Backblaze B2, MinIO) on top of the always-local copy. A remote outage never fails a backup: the local copy is primary and uploads are best-effort. Object-storage credentials are held by the credential manager, written to a private file and masked in API responses, and referenced by destinations so no secret ever lands in the agent config. Destinations are configured through the existing config API and applied live. Listing, download, and restore transparently fall back to a remote copy when the local archive has been pruned, and local retention no longer deletes remote copies. A destination connectivity check is available, and what/how/where a deployment is backed up is now documented.
Introduces a MinIO app template so an S3-compatible object store can be deployed and run by FlatRun itself, with a per-deployment root credential and its data in a flat-file mount. Adds a spec defining the object store abstraction: one concept with two kinds, external (a store FlatRun only connects to) and managed (a store FlatRun runs from a template), which backup destinations fold into. Later slices cover an object browser, deployment consumption, and replication.
Code Review SummaryThis PR adds S3-compatible remote backup destinations to FlatRun. It introduces a generic credential manager for S3 secrets and an 'Object Store' abstraction that supports both external and managed (MinIO) stores. Remote uploads are best-effort, ensuring local backups succeed even if the network is down. 🚀 Key Improvements
💡 Minor Suggestions
|
![sourceant[bot]](https://avatars.githubusercontent.com/in/1161781?s=60&v=4){kind=small}
| prefix = filter.DeploymentName + "/" | ||
| } | ||
| for _, r := range m.getRemotes() { | ||
| objs, err := r.List(context.Background(), prefix) |
There was a problem hiding this comment.
Passing context.Background() ignores the request context. While listing remotes is best-effort, using the passed context (if available) or a timeout-bounded context is preferred.
Suggested change
| objs, err := r.List(context.Background(), prefix) | |
| objs, err := r.List(ctx, prefix) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FlatRun backups were local-only, so a lost host meant lost backups. This adds remote S3-compatible object storage as a backup target on top of the always-local copy, and lays the foundation for treating object stores as a first-class concept.
Closes #105.
Why it is shaped this way (things the diff will not tell you):
0600, masked in API responses) and are referenced by id, so no secret ever lands in the flat-file config.It also seeds the object-store abstraction: a store records its
kind(external, which FlatRun only connects to, or managed, which FlatRun runs from a template), and a MinIO template lets FlatRun host its own S3 endpoint. The object browser, deployment consumption, and replication are later slices, described indocs/OBJECT_STORES.md.