sandboxd: pool warmup, buffered exec endpoint, no_balloon - #148
Merged
Merged
Conversation
A pool entry's warmup argv runs in the golden VM after readiness and before the snapshot, so the files it touches (a Node or Python runtime) are page-cache-resident in every clone instead of being read from disk on the first exec of each claim. Config-owned like egress: PUT /v1/pools rejects it, the seed hash excludes it, and a golden built with a different warmup is rebuilt via a sidecar stamp. Hot path: zero; the exec runs once per golden build.
One command in, its exit code and complete output out as JSON. Clients that cannot hold an upgraded connection (or want to multiplex over HTTP/2 through a TLS proxy) get a single request per command instead of a TCP connection per RPC; the relay stays the streaming path. Timeout closes the guest connection so silkd kills the child. Hot path: none; the relay is untouched.
Boots pool and template VMs with cocoon's --no-balloon so a guest keeps its whole memory; clones inherit it from the golden. cocoon balloons 25% of guest memory by default, and a 16G build guest running a large typecheck thrashes on the remaining 12G before deflate-on-OOM fires (measured: opencode's bun typecheck at 8c/16G froze the guest at ~11.5G used with a 4G balloon inflated).
exec_test.go used context without importing it, so the server package failed typecheck in CI and none of its tests ran. A client that drops the request mid-command now returns silently instead of logging an error and writing 502 to a closed connection; a negative timeout_seconds is a 400. Docs: warmup runs under the 2-minute engine command timeout with only PATH set and a failure fails the golden build; the exec endpoint lists 401 and the negative-timeout 400.
… fields Closing the guest connection only reaches a child that writes: silkd learns the client is gone from a failed write, so a silent command ran to completion behind a 504, and every retry stacked another copy. The handler now keeps the pid from the started frame and sends silkd a kill on a fresh connection whenever it stops collecting output early (timeout, client gone, output cap). The body is decoded strictly, so a client asking for `user`/`session` gets a 400 instead of silently running as root; the API doc states the kill and that output is UTF-8 text.
Go: unreachable EncodeRequest branches, the SetPools 128-bit hash-collision guard, the readoptEgressTap lane re-check, the journal ErrClosed filter and the SignLeaf expiry guard (the notAfter clamp already covers it) are gone; pooledHash scans became an m.pools lookup; handleAgent and handleExec share wakeGuest; the exec output cap is checked before the append; errors.AsType replaces the typed-var errors.As shape in prod and tests; omitzero on every bool/numeric JSON field; the three store/peer nil-client fallbacks use cmp.Or (the asl cmpor gate); the sdk post forwarder is inlined; Connect sits under the Client type; mcp's rpc vocabulary types sit ahead of server; enum-listing field comments and the runCA godoc are dropped; test files lose their no-op nolint:gosec directives, use slices.Concat and strings.Cut. Rust: find reuses one scratch String across the walk; is_valid_pid and is_oversized; boot resolves a disk's serial once and hoists the device path; test files get std/external/crate use groups, helpers ahead of tests and no per-match to_string. Python: Checkpoint.new reuses Client._claim_from (drops the cycle-breaking local import); __exit__ catches APIError only; tests parametrize the malformed-frame cases; ruff 0.16 format. Docs: the warmup environment is silkd's base environment, not PATH alone. Comment lines: +8/-15 (Go+Rust), +0/-2 (Python). Prod net -29, test -3. Gates: make go-lint 10x "0 issues.", make go-test 18 ok, asl 0 findings on both GOOS (5 forwarder advisories kept: guardsEgressLane, matchMethod, skipIdle, refillGated, benignSweepErr), cargo fmt/clippy/test green for silkd and boot/init on macOS and linux/arm64, ruff format/check and pytest green for the three Python packages.
killExec derived a 5-second context but only the wake and the dial honoured it; the kill frame's write and the reply read ran unbounded, so a silkd that accepted the connection and never answered pinned the handler goroutine and the 504 behind it forever. The connection is now closed when the context ends, the same shape as the exec itself, and the reply is decoded: done and not_found mean the child is gone, anything else is logged at the call site together with the wake, dial and write failures.
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #143: the same three additions rebased on
mainafter #142, plus the fixes and the whole-repo review round that followed the PR review. Eight commits, each self-contained.The feature commits (from #143)
warmupargv — runs in the golden VM after readiness and before the snapshot, so what it touches is page-cache-resident in every clone. Measured on a Node flavor: the firstnode -vin a fresh clone cost 150–385 ms (cold read of the node binary), the second 6–37 ms; withwarmup: ["node", "-e", "0"]the first exec is 15–41 ms. Config-owned likeegress(PUT /v1/poolsrejects it, the seed hash excludes it); a.warmupsidecar stamp makes a golden built with a different warmup rebuild. The guest exec PATH now includes/usr/local/{sbin,bin}.POST /v1/sandboxes/{id}/exec— a buffered exec (argv/cwd/env/timeout_seconds in, exit_code/stdout/stderr out) for clients that cannot hold an upgraded connection or want to multiplex over HTTP/2 through a TLS proxy. 8 MiB output cap (413). The relay is untouched and stays the streaming path.no_balloon— boots pool and template VMs with cocoon's--no-balloon(vm: add --no-balloon; whole-repo comment, signature and layout round cocoon#221); clones inherit it from the golden. cocoon balloons 25% of guest memory by default, and an 8-CPU/16G guest running a large typecheck thrashed on page cache and froze for the whole lease. Off by default; only a node that sets it passes the flag, so an older cocoon is unaffected.fix: close buffered exec stdin and scope timeoutandreview: buffered exec test import, client-cancel path, warmup doc bounds— the sandboxd: pool warmup, buffered exec endpoint, no_balloon #143 review fixes.Added after #143
fix: buffered exec kills a command it gives up on and rejects unknown fields— closing the guest connection only reaches a child that writes; a silent one (sleep 600) ran to completion behind the 504 and retries stacked copies. The handler keeps thestartedpid and sends silkdkillon a fresh connection on every give-up path (timeout, client gone, output cap). The body is decoded strictly: an unknown field such asuserwas silently dropped. Regression testsTestExecTimeoutKillsTheCommandandTestExecRejectsUnknownFields.review: whole-repo round after #143— the batch-end round over the whole tree (Go, Rust, Python):wakeGuestshared by the relay and the exec, unreachable error branches and guards dropped,errors.AsType/cmp.Or/slices.Concat/strings.Cut/omitzerowhere they remove code, Rustfindwalks one reusable buffer, Python_claim_fromshared by claim and checkpoint claim. 74 files, prod −29 / test −3, comments +8/−15.fix: bound the exec kill by its timeout and check silkd's reply—killExecderived a 5-second context but only the wake and the dial honoured it; the kill frame's write and the reply read ran unbounded, so a silkd that accepted the connection and never answered pinned the handler and the 504 behind it. The connection is now closed when the context ends, and the reply is decoded:doneandnot_foundmean the child is gone, anything else is logged.Filed as follow-ups rather than folded in: #144 (buffered exec response memory bound and non-UTF-8 output), #145 (silkd spawn failures reported as
internal), #146 (idle sweep vs a live exec).Hot path
Zero on the warm-claim path and the relay path —
wakeGuestis the relay's existing wake+dial sequence moved into a function; warmup runs once per golden build; the exec endpoint is opt-in per request.Evidence
make go-lint: exit 0, 10 ×0 issues.(5 modules × linux/darwin)make go-test: 18 packages ok, race detector onasl ./...andGOOS=linux asl ./...: 0 findings (forwarder advisories recorded in the hygiene ledger)cargo fmt --check,clippy -D warnings,cargo testgreen on macOS and in a linux/arm64 containerruff format --check,ruff check,pytestgreen (155 / 4 / 6)no_balloonwere exercised on the leaderboard dry-run node behind an HTTP/2 proxy (TTI harness at 100 concurrency 382 ms median on the relay → 78–239 ms on the exec endpoint, 100/100 success; the DAX build completes 7/7 phases withno_balloon).