Skip to content

Correct drift one service at a time, and resume a rollout only once every service runs - #363

Merged
eldara-cruncher merged 2 commits into
mainfrom
fix/converge-one-at-a-time-and-safe-resume
Nov 17, 2026
Merged

eldara-cruncher merged 2 commits into
mainfrom
fix/converge-one-at-a-time-and-safe-resume

Conversation

@eldara-cruncher

@eldara-cruncher eldara-cruncher commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem
Live-drift correction deployed a drifted release's whole manifest in one call, restarting every drifted peer of a clustered database together (#361); and an automated sync resumed an unfinished rollout even while a peer was down, which rolls the next peer with it down.

Fix
Corrections go through the chart engine's new Redeploy, which rolls marked services one at a time and records nothing; an automated sync resumes an unfinished rollout only once every service in the release is running and reports resumeWaiting until then; and CE now records an unfinished rollout durably, so it survives a controller restart.

Detail
  • Pins CE v2.5.0-rc4, which carries Record a rollout that stops, and redeploy one service at a time without recording swarmcli#732.
  • converge calls redeploy, which asserts the engine for Redeploy (an optional interface, like releaseLister) and falls back to the single DeployStack for an engine without it. The application's syncPolicy.timeout bounds each service.
  • The guard is decided in record, where health is known. ResumeWaiting is set for an automated application's release whose only reason to deploy is an unfinished rollout — CE's ChangeUnfinished alone, or this controller's interrupted mark on an unchanged plan — while its health is not healthy. reconcileHeld then plans those releases as unchanged for a non-forced pass. A forced sync (the button, app sync) is not held. An upgrade with anything new in it is not held either; that is the engine's ordinary rollout.
  • Why the guard is needed: CE's heldForRollout holds back only the marked services that are up, so a down peer goes out with the first deploy and the next one is rolled while it is still down. Resume a rollout the last sync stopped part-way through #360 resumed on the next pass whatever the health. The guard also bounds retries of a permanently stuck peer: nothing resumes until it runs.
  • With swarmcli#732 the plan after a stopped rollout is an upgrade with the reason "the last rollout did not finish", so the Diff tab, app diff and an automated resume all work after a restart. The in-memory mark from Resume a rollout the last sync stopped part-way through #360 stays, for the error and the pending services.
  • app get and the overview explain resumeWaiting. Docs: concepts.md (resume, durability, the guard), configuration.md (corrections are sequential), api.md (resumeWaiting).
  • Tests: the guard in both directions, for the remembered and the engine-recorded case; a manual sync not being held; an upgrade with something new not being held; a converge going through Redeploy with the application's timeout. The two Resume a rollout the last sync stopped part-way through #360 resume tests now run against a healthy service. Disabling the guard, or the Redeploy route, fails the new tests. go test ./..., go vet ./..., golangci-lint 2.15.0 (0 issues), SPDX, vitest (294 passed), typecheck, eslint, vite build.

Closes #361

Machine-generated by an AI agent (Claude Opus 5.5) via the eldara-cruncher bot, and not human-reviewed. Claims carry a file, a line or a reproducible check so they can be verified cheaply.

🤖 Generated with Claude Code

https://claude.ai/code/session_01J2mZoxDEwj25Qv5KHaKKgN

…once every service runs

A drift correction deployed a whole manifest at once, restarting every drifted
peer of a cluster together; it now goes through the engine's Redeploy. An
automated sync resumes an unfinished rollout only once every service in the
release runs. Pins CE with recorded unfinished rollouts (swarmcli#732).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J2mZoxDEwj25Qv5KHaKKgN
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J2mZoxDEwj25Qv5KHaKKgN
@eldara-cruncher
eldara-cruncher merged commit 2b80dbf into main Nov 17, 2026
20 checks passed
@eldara-cruncher
eldara-cruncher deleted the fix/converge-one-at-a-time-and-safe-resume branch November 18, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Live-drift correction restarts drifted sequential services together

1 participant