Repository navigation
isb 3.3.0: a host reboot comes back in waves: staggered autostart, paced controller starts, dev sandbox disk caps - #72
Merged
Conversation
…x disk caps Stack replicas autostart in waves (slot 1 at priority 2, the rest at 1, 2s apart, a VM 10s), set on running replicas in place and kept out of the revision. Plain sandboxes default to boot.autostart=false and get root disk caps (128MiB/s read, 64MiB/s write). The controller paces instance starts and creates across every stack (--start-concurrency, --start-io-pressure on /proc/pressure/io). Claude-Session: https://claude.ai/code/session_01XJek6YFHRCRmXsmJ3v1WV4
Claude-Session: https://claude.ai/code/session_01XJek6YFHRCRmXsmJ3v1WV4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On 2026-10-24, after a hard reset, titan started 154 Incus instances at once and io pressure sat at 98% for about 85 minutes. This release makes a host reboot come back in waves.
Staggered boot policy
boot.autostart.priority(slot 1 = 2, other slots = 1) andboot.autostart.delay(2 s, VM 10 s). After a reboot, Incus brings every service's first replica up before the rest, one at a time.isb servesets these keys in place on existing replicas (a config update, no restart). They are kept out ofuser.isb.rev, so nothing rolls: the revision diff is empty.isb create/isb upsandboxes get an explicitboot.autostart=false. Long-running ones (restart: always/on-failure) still autostart, spaced.boot:key (autostart,priority,delay) overrides all of this. It is documented in compose.md and stacks.md.Host-wide start pacing in the controller
--start-concurrency/ISB_START_CONCURRENCY, default 4, at least 1 s apart./proc/pressure/iofull avg10 is at or above--start-io-pressure(default 40%,0turns it off), for up to 5 minutes each.Default disk I/O caps for dev sandboxes
limits.read 128MiBandlimits.write 64MiBon the root disk, overridable throughraw_devices.root. Stack replicas are unchanged.Verified against a real Incus 7.5.1 in a VM:
instance-updatedevents.scripts/check.shis green on the merge with 3.2.0.Not included: no backfill. Existing sandboxes keep no
boot.autostartkey, so Incus' last-state restore still brings them back after a reboot. That is a follow-up for Stephan to decide.https://claude.ai/code/session_01XJek6YFHRCRmXsmJ3v1WV4