Repository navigation
Correct drift one service at a time, and resume a rollout only once every service runs - #363
Merged
eldara-cruncher merged 2 commits intoNov 17, 2026
Merged
Conversation
…once every service runs A drift correction deployed a whole manifest at once, restarting every drifted peer of a cluster together; it now goes through the engine's Redeploy. An automated sync resumes an unfinished rollout only once every service in the release runs. Pins CE with recorded unfinished rollouts (swarmcli#732). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J2mZoxDEwj25Qv5KHaKKgN
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J2mZoxDEwj25Qv5KHaKKgN
eldara-cruncher
deleted the
fix/converge-one-at-a-time-and-safe-resume
branch
November 18, 2026 10:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Live-drift correction deployed a drifted release's whole manifest in one call, restarting every drifted peer of a clustered database together (#361); and an automated sync resumed an unfinished rollout even while a peer was down, which rolls the next peer with it down.
Fix
Corrections go through the chart engine's new
Redeploy, which rolls marked services one at a time and records nothing; an automated sync resumes an unfinished rollout only once every service in the release is running and reportsresumeWaitinguntil then; and CE now records an unfinished rollout durably, so it survives a controller restart.Detail
v2.5.0-rc4, which carries Record a rollout that stops, and redeploy one service at a time without recording swarmcli#732.convergecallsredeploy, which asserts the engine forRedeploy(an optional interface, likereleaseLister) and falls back to the singleDeployStackfor an engine without it. The application'ssyncPolicy.timeoutbounds each service.record, where health is known.ResumeWaitingis set for an automated application's release whose only reason to deploy is an unfinished rollout — CE'sChangeUnfinishedalone, or this controller's interrupted mark on an unchanged plan — while its health is nothealthy.reconcileHeldthen plans those releases as unchanged for a non-forced pass. A forced sync (the button,app sync) is not held. An upgrade with anything new in it is not held either; that is the engine's ordinary rollout.heldForRolloutholds back only the marked services that are up, so a down peer goes out with the first deploy and the next one is rolled while it is still down. Resume a rollout the last sync stopped part-way through #360 resumed on the next pass whatever the health. The guard also bounds retries of a permanently stuck peer: nothing resumes until it runs.app diffand an automated resume all work after a restart. The in-memory mark from Resume a rollout the last sync stopped part-way through #360 stays, for the error and the pending services.app getand the overview explainresumeWaiting. Docs:concepts.md(resume, durability, the guard),configuration.md(corrections are sequential),api.md(resumeWaiting).Redeploywith the application's timeout. The two Resume a rollout the last sync stopped part-way through #360 resume tests now run against a healthy service. Disabling the guard, or theRedeployroute, fails the new tests.go test ./...,go vet ./..., golangci-lint 2.15.0 (0 issues), SPDX, vitest (294 passed), typecheck, eslint,vite build.Closes #361
Machine-generated by an AI agent (Claude Opus 5.5) via the
eldara-cruncherbot, and not human-reviewed. Claims carry a file, a line or a reproducible check so they can be verified cheaply.🤖 Generated with Claude Code
https://claude.ai/code/session_01J2mZoxDEwj25Qv5KHaKKgN