Conversation
A restore maps the golden memory lazily, so the pages the golden warmup made resident still fault into each clone on first touch. Under nested virtualization that first touch is the whole time-to-interactive: at 100 concurrent claims the first `node -v` in a fresh clone measured p50 824 ms against 92 ms once the pages were in, while the host sat half idle. Re-running the warmup in the clone before it joins the warm pool moves that cost to refill; the claim path is unchanged.
…urst A release drops and journals the claim, then removes the VM inline. In a 100-way burst the early destroys land while other claims still run their first command: with destroys held until the burst ended the same command measured p95 254-293 ms, inline 620-646 ms. With release_delay_seconds set the release returns once the claim is gone and the VM, volume, egress and snapshot cleanup run that many seconds later; teardown failures are logged by the removal queue as before.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two refill/release-side changes measured on the leaderboard dry-run host (22 vCPU nested-virtualization guest, 128-VM warm pool, 100 concurrent claims each running
node -v). Stacked on #143; the claim and exec paths are untouched.Run the pool warmup in every clone at refill. A restore maps the golden memory lazily, so the pages the golden warmup made resident still fault into each clone on first touch, and under nested virtualization that first touch is the whole time-to-interactive: the first
node -vin a fresh clone measured p50 824 ms / p95 1210 ms at 100 concurrent, against 92 / 162 ms once the pages were in, with the host half idle. Re-running the samewarmupargv in the clone before it joins the warm pool moves that cost to refill (one exec per clone, ~30–100 ms): the same burst then measures p50 84 / p95 128 ms. A clone whose warmup fails is destroyed and refilled like a failed probe.release_delay_seconds. A release drops and journals the claim, then removes the VM inline. In a 100-way burst the early destroys land while other claims still run their first command: with destroys held until the burst ended the same command measured p95 254–293 ms, inline 620–646 ms. With the knob set the release returns once the claim is gone and the VM, volume, egress and snapshot cleanup run that many seconds later; teardown failures are logged by the removal queue as before. Default 0 keeps the inline behaviour. Hot-path cost: zero on claim and exec; release returns earlier.Tests:
TestRefillWarmsEveryClone,TestRefillCloneWarmupFailureCleansUp,TestReleaseDelayDefersTeardown; the golden-warmup test now asserts the golden run precedes the snapshot save. Docs:deploy.mdrows forwarmupandrelease_delay_seconds.End-to-end on the dry-run host with the upstream harness (
tti.bench.ts --iterations 100 --concurrency 100 --no-ingest): p50 1097 ms → 262–279 ms, p95 1945 → 293–310 ms, composite 85.8 → 97.1–97.3, withrefill_concurrency: 1so refills kicked by the claims do not contend with the burst.