sandboxd: warm every clone at refill and defer release teardown - #149
Merged
Merged
Conversation
A restore maps the golden memory lazily, so the pages the golden warmup made resident still fault into each clone on first touch. Under nested virtualization that first touch is the whole time-to-interactive: at 100 concurrent claims the first `node -v` in a fresh clone measured p50 824 ms against 92 ms once the pages were in, while the host sat half idle. Re-running the warmup in the clone before it joins the warm pool moves that cost to refill; the claim path is unchanged.
…urst A release drops and journals the claim, then removes the VM inline. In a 100-way burst the early destroys land while other claims still run their first command: with destroys held until the burst ended the same command measured p95 254-293 ms, inline 620-646 ms. With release_delay_seconds set the release returns once the claim is gone and the VM, volume, egress and snapshot cleanup run that many seconds later; teardown failures are logged by the removal queue as before.
CMGS
force-pushed
the
feat/refill-warmup
branch
from
September 11, 2026 03:57
ac0eb29 to
5fc208a
Compare
release_delay_seconds armed one time.AfterFunc per release, so a burst of releases became the same burst of `cocoon vm rm` a delay later, outside the provisioning budget every other batch teardown runs on. The release now parks the VM in pendingRemovals with a not-before stamp; retryRemovals dispatches only the due entries on the reap tick through runBounded, so the teardown is rate-limited by refill_concurrency and a survivor keeps the existing retry path. The egress listener still closes at release; the tap unlock and the volume hold release follow the removal, as on a retried survivor.
One warmup runner for the golden build and the clone (the golden keeps the argv for its sidecar stamp), warmClone reads the pool key from the sandbox, the two-line rationale becomes one, and the CA-install error assigns the outer variable instead of shadowing it. releaseDelay moves next to the other node-wide lifecycle scalars, release_delay_seconds takes omitzero like every sibling int, and the non-negative table follows the struct's field order. The fake engine latches warmup-after-snapshot as one flag and the clone test copies the fake's slices before asserting, as the file's other accessors do.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two refill/release-side changes measured on the leaderboard dry-run host (22 vCPU nested-virtualization guest, 128-VM warm pool, 100 concurrent claims each running
node -v). Follows #148 (now on main); replaces #147, which targeted the retired #143 branch. The claim and exec paths are untouched.Run the pool warmup in every clone at refill. A restore maps the golden memory lazily, so the pages the golden warmup made resident still fault into each clone on first touch, and under nested virtualization that first touch is the whole time-to-interactive: the first
node -vin a fresh clone measured p50 824 ms / p95 1210 ms at 100 concurrent, against 92 / 162 ms once the pages were in, with the host half idle. Re-running the samewarmupargv in the clone before it joins the warm pool moves that cost to refill (one exec per clone, ~30–100 ms): the same burst then measures p50 84 / p95 128 ms. A clone whose warmup fails is destroyed and refilled like a failed probe.release_delay_seconds. A release drops and journals the claim, then removes the VM inline. In a 100-way burst the early destroys land while other claims still run their first command: with destroys held until the burst ended the same command measured p95 254–293 ms, inline 620–646 ms. With the knob set the release returns once the claim is gone and its egress listener is closed; the VM goes into the existing removal queue with a not-before stamp, and the reap tick dispatches the due entries throughrunBounded, so the teardown runs on therefill_concurrencybudget like every other batch teardown and a survivor keeps the existing retry path (an earlier revision armed onetime.AfterFuncper release, which only phase-shifted the storm). Default 0 keeps the inline behaviour. Hot-path cost: zero on claim and exec; release returns earlier. Until the VM is gone it holds its memory and volume reservations without counting as a claim, which is why the knob is opt-in.Tests:
TestRefillWarmsEveryClone,TestRefillCloneWarmupFailureCleansUp,TestReleaseDelayQueuesTeardown; the golden-warmup test now asserts the golden run precedes the snapshot save. Docs:deploy.mdrows forwarmupandrelease_delay_seconds.End-to-end on the dry-run host with the upstream harness (
tti.bench.ts --iterations 100 --concurrency 100 --no-ingest,refill_concurrency: 1,release_delay_seconds: 5): p50 1097 ms → 124–136 ms, p95 1945 → 157–185 ms, composite 85.8 → 98.5–98.7 over two runs; the release request itself answers in 2–21 ms p50 and the queue drained 100 VMs while the pool refilled.The last two commits are the review round: the queue mechanism above, and layout/reuse cleanups (one warmup runner for golden and clone,
omitzero, field placement, a latched fake flag). golangci-lint 10 × 0 issues, asl clean on linux and darwin,-racetests green.