Skip to content

sandboxd: warm every clone at refill and defer release teardown - #149

Merged
CMGS merged 4 commits into
mainfrom
feat/refill-warmup
Sep 11, 2026
Merged

CMGS merged 4 commits into
mainfrom
feat/refill-warmup

Conversation

@CMGS

@CMGS CMGS commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Two refill/release-side changes measured on the leaderboard dry-run host (22 vCPU nested-virtualization guest, 128-VM warm pool, 100 concurrent claims each running node -v). Follows #148 (now on main); replaces #147, which targeted the retired #143 branch. The claim and exec paths are untouched.

Run the pool warmup in every clone at refill. A restore maps the golden memory lazily, so the pages the golden warmup made resident still fault into each clone on first touch, and under nested virtualization that first touch is the whole time-to-interactive: the first node -v in a fresh clone measured p50 824 ms / p95 1210 ms at 100 concurrent, against 92 / 162 ms once the pages were in, with the host half idle. Re-running the same warmup argv in the clone before it joins the warm pool moves that cost to refill (one exec per clone, ~30–100 ms): the same burst then measures p50 84 / p95 128 ms. A clone whose warmup fails is destroyed and refilled like a failed probe.

release_delay_seconds. A release drops and journals the claim, then removes the VM inline. In a 100-way burst the early destroys land while other claims still run their first command: with destroys held until the burst ended the same command measured p95 254–293 ms, inline 620–646 ms. With the knob set the release returns once the claim is gone and its egress listener is closed; the VM goes into the existing removal queue with a not-before stamp, and the reap tick dispatches the due entries through runBounded, so the teardown runs on the refill_concurrency budget like every other batch teardown and a survivor keeps the existing retry path (an earlier revision armed one time.AfterFunc per release, which only phase-shifted the storm). Default 0 keeps the inline behaviour. Hot-path cost: zero on claim and exec; release returns earlier. Until the VM is gone it holds its memory and volume reservations without counting as a claim, which is why the knob is opt-in.

Tests: TestRefillWarmsEveryClone, TestRefillCloneWarmupFailureCleansUp, TestReleaseDelayQueuesTeardown; the golden-warmup test now asserts the golden run precedes the snapshot save. Docs: deploy.md rows for warmup and release_delay_seconds.

End-to-end on the dry-run host with the upstream harness (tti.bench.ts --iterations 100 --concurrency 100 --no-ingest, refill_concurrency: 1, release_delay_seconds: 5): p50 1097 ms → 124–136 ms, p95 1945 → 157–185 ms, composite 85.8 → 98.5–98.7 over two runs; the release request itself answers in 2–21 ms p50 and the queue drained 100 VMs while the pool refilled.

The last two commits are the review round: the queue mechanism above, and layout/reuse cleanups (one warmup runner for golden and clone, omitzero, field placement, a latched fake flag). golangci-lint 10 × 0 issues, asl clean on linux and darwin, -race tests green.

Base automatically changed from feat/warmup-exec to main September 11, 2026 03:52
A restore maps the golden memory lazily, so the pages the golden warmup
made resident still fault into each clone on first touch. Under nested
virtualization that first touch is the whole time-to-interactive: at 100
concurrent claims the first `node -v` in a fresh clone measured p50 824 ms
against 92 ms once the pages were in, while the host sat half idle.
Re-running the warmup in the clone before it joins the warm pool moves that
cost to refill; the claim path is unchanged.
…urst

A release drops and journals the claim, then removes the VM inline. In a
100-way burst the early destroys land while other claims still run their
first command: with destroys held until the burst ended the same command
measured p95 254-293 ms, inline 620-646 ms. With release_delay_seconds set
the release returns once the claim is gone and the VM, volume, egress and
snapshot cleanup run that many seconds later; teardown failures are logged
by the removal queue as before.
@CMGS
CMGS force-pushed the feat/refill-warmup branch from ac0eb29 to 5fc208a Compare September 11, 2026 03:57
release_delay_seconds armed one time.AfterFunc per release, so a burst of
releases became the same burst of `cocoon vm rm` a delay later, outside
the provisioning budget every other batch teardown runs on. The release
now parks the VM in pendingRemovals with a not-before stamp; retryRemovals
dispatches only the due entries on the reap tick through runBounded, so
the teardown is rate-limited by refill_concurrency and a survivor keeps the
existing retry path. The egress listener still closes at release; the tap
unlock and the volume hold release follow the removal, as on a retried
survivor.
One warmup runner for the golden build and the clone (the golden keeps the
argv for its sidecar stamp), warmClone reads the pool key from the sandbox,
the two-line rationale becomes one, and the CA-install error assigns the
outer variable instead of shadowing it. releaseDelay moves next to the
other node-wide lifecycle scalars, release_delay_seconds takes omitzero
like every sibling int, and the non-negative table follows the struct's
field order. The fake engine latches warmup-after-snapshot as one flag and
the clone test copies the fake's slices before asserting, as the file's
other accessors do.
@CMGS
CMGS merged commit 8152e8a into main Sep 11, 2026
1 check passed
@CMGS
CMGS deleted the feat/refill-warmup branch September 11, 2026 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant