server: buffered exec — bound the response memory and carry non-UTF-8 output faithfully

[CMGS]

POST /v1/sandboxes/{id}/exec (#143) buffers the whole stdout/stderr and returns them as JSON strings. Two properties of that shape need a decision before the endpoint is sold as the "clients that cannot hold an upgraded connection" path.

A. Memory is bounded on the raw bytes only, and nothing bounds concurrency

execOutputCap (8 MiB) caps the decoded bytes. JSON escaping amplifies control bytes 6x: encoding ExecResponse{Stdout: 8 MiB of 0x01} produces a 50,331,688-byte body, and one json.Encoder.Encode of it allocates ~344 MB. Per request the node holds the accumulating slice (up to 2x during append growth), the string(stdout) copy, the scanner's frame buffer and the encoder buffer — roughly 70 MB peak for an 8 MiB output. Nothing limits concurrent buffered execs per node, sandbox or tenant (max_claims counts claims, not requests).

Scenario: one valid sandbox token, 50 concurrent {"argv":["head","-c","8388607","/dev/zero"]} (just under the cap, so every one answers 200) → ~3.5 GB of transient control-plane heap on a node whose RAM is budgeted for VMs. An OOM-killed sandboxd takes the node's claims, relays and egress proxies with it while the VMs live on. The relay path never buffers, so this surface is new.

B. Non-UTF-8 output is rewritten

Stdout/Stderr are Go strings, so encoding/json replaces every byte that is not valid UTF-8 with U+FFFD and still answers 200. Measured: head -c 64 /bin/ls → 11 input bytes come back as 17. The relay and both SDKs' Run deliver exact bytes; this endpoint offers no raw alternative. docs/sandboxd-api.md states the text-only contract for now.

Options

1. Carry stdout/stderr as base64 (the wire's own convention for data frames): bytes exact, amplification drops to 1.33x. Either change the fields (breaking for the one client built against strings) or add stdout_b64/stderr_b64.
2. A small node-wide semaphore on the handler (503 + Retry-After beyond it) and/or a lower cap, with the per-node limit stated in the API doc.
3. Stream-encode the response instead of building the whole body in memory.

1 settles the contract; 2 is orthogonal and cheap. Decide 1 first.

Refs: sandboxd/server/exec.go (collectExec, execOutputCap), docs/sandboxd-api.md "POST /v1/sandboxes/{id}/exec".