Skip to content

feat(controller): make reconciles restart-safe - #1447

Open
arturre wants to merge 4 commits into
mainfrom
feat/idempotent-reconcile
Open

arturre wants to merge 4 commits into
mainfrom
feat/idempotent-reconcile

Conversation

@arturre

@arturre arturre commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Description

This pull request changes the following:

  • Shutdown timing is now deliberate. There is a new --graceful-shutdown-timeout flag (default 30s), and LeaderElectionReleaseOnCancel is now enabled. The chart value controller.gracefulShutdownTimeoutSeconds sets the drain, and the pod's terminationGracePeriodSeconds is set to that drain plus 15s (45s by default). The drain is best-effort; correctness relies on reconciles resuming after a restart.
  • NetworkUpgradeExecute saves the registry selection (RegistrySelected + status.selectedSources) before patching the ConsensusCapsule. A restart between the two steps reuses the same images instead of re-selecting and rolling the pod a second time. startTime is now only set when it is empty.
  • HelmCapsule recovers releases left in pending-* by a killed operator. A pending revision older than spec.timeout is marked failed, then upgraded; a younger one is waited out. The release is now read once per reconcile. The Helm client gains Unlock and ReleaseInfo.LastDeployed.
  • NetworkGenesis no longer leaves contradictory conditions: success clears a stale Failed, and failure or spec-mutation rejection clears a stale Ready.
  • Orbit skips the status write when nothing changed, and no longer re-stamps LastTransitionTime on every reconcile.
  • Restart tests:
    • internal/crashtest wraps a client so it "dies" after a chosen write.
    • Unit tests in internal/controller/restart_safety_test.go cover Execute, Genesis, Prepare prefetch, ConsensusCapsule image pins and Orbit.
    • test/integration/restart/ kills a real manager on envtest mid-upgrade and mid-Helm-install, and asserts that a second manager finishes the operation.

Per-controller restart notes

Controller Restart behavior
Orbit Recomputes status on every reconcile; skips the write when nothing changed.
ConsensusCapsule Rebuilds children from spec. Resolved image pins are cached in status.resolvedImages and applied after status writes.
Config CRs Re-render their ConfigMap from spec; idempotent.
HAProxyCapsule, EnvoyProxy Rebuild children from spec.
HelmCapsule Unlocks a stale pending-* release, then upgrades.
NetworkGenesis Re-upserts all ConfigMaps; Succeeded is terminal.
NetworkUpgradePrepare Writes selection and phase in one status update. Prefetch Jobs have deterministic names, so retries don't duplicate them.
NetworkUpgradeFreeze Writes the revalidation result and phase in one status update.
NetworkUpgradeExecute Reuses the saved selection; timestamps are set once; the daemon owns phase changes.
NetworkUpgradeFreezeAbort Does a read-only check, then a single terminal status write.

Related Issues

Signed-off-by: Artur Reznikov <artur.reznikov@swirldslabs.com>
@arturre
arturre requested review from a team as code owners October 27, 2026 21:19
@arturre
arturre requested a review from crypto-pablo October 27, 2026 21:19
arturre and others added 2 commits October 27, 2026 14:25
Signed-off-by: Artur Reznikov <artur.reznikov@swirldslabs.com>
@leninmehedy leninmehedy changed the title feat(controller): make reconciles restart-safe. resume op feat(controller): make reconciles restart-safe Oct 28, 2026
Comment thread internal/resources/helm/release.go
Signed-off-by: Artur Reznikov <artur.reznikov@swirldslabs.com>
@arturre
arturre requested a review from leninmehedy October 28, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Operator restart-safety: idempotent reconciles + resumable operations across pod restarts

2 participants