Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .github/workflows/python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,14 @@ jobs:
with:
python-version: "3.12"
- name: Install tooling
run: pip install ruff==0.15.20 pytest==8.4.2 && pip install -e sdk/python -e sdk/openai -e sdk/langchain
run: pip install ruff==0.15.20 pytest==8.4.2 mypy==1.19.1 && pip install -e sdk/python -e sdk/openai -e sdk/langchain
- name: Lint
run: ruff format --check . && ruff check .
- name: Test cocoonsandbox
run: cd sdk/python && pytest -q
run: cd sdk/python && python -m pytest -q
- name: Type check cocoonsandbox
run: cd sdk/python && python -m mypy
- name: Test openai adapter
run: cd sdk/openai && pytest -q
run: cd sdk/openai && python -m pytest -q
- name: Test langchain adapter
run: cd sdk/langchain && pytest -q
run: cd sdk/langchain && python -m pytest -q
17 changes: 17 additions & 0 deletions .github/workflows/sandboxd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ on:
paths:
- "sandboxd/**"
- "sdk/go/**"
- "sdk/python/**"
- "mcp/**"
- "e2e/**"
- "protocol/**"
Expand All @@ -17,6 +18,7 @@ on:
paths:
- "sandboxd/**"
- "sdk/go/**"
- "sdk/python/**"
- "mcp/**"
- "e2e/**"
- "protocol/**"
Expand Down Expand Up @@ -48,3 +50,18 @@ jobs:

- name: Test
run: make go-test

- name: Set up Python
uses: actions/setup-python@v7.0.0
with:
python-version: "3.12"

- name: Test TLS edge with Caddy
working-directory: e2e
env:
CADDY_BIN: ${{ runner.temp }}/caddy
run: |
curl -fsSL --retry 3 https://github.com/caddyserver/caddy/releases/download/v2.11.4/caddy_2.11.4_linux_amd64.tar.gz -o "$RUNNER_TEMP/caddy.tar.gz"
printf '%s %s\n' '8220d1f013b6f27510247b2360c9e0ca9f018feebd82515f07635318b34ff9777ccc8fd0b6e6f2486ce3a33fe389fbb7db12d05baa474f4587509fb4f5ebf1c9' "$RUNNER_TEMP/caddy.tar.gz" | sha512sum -c -
tar -xzf "$RUNNER_TEMP/caddy.tar.gz" -C "$RUNNER_TEMP" caddy
GOWORK=off go test -race -count=1 -run TestCaddyTLSCluster -timeout 60s -v .
4 changes: 4 additions & 0 deletions docs/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ A claim always enters at whatever node the client dialed:

The data plane is never proxied between nodes: the claim response carries
`owner_addr` and all sandbox traffic dials the owner directly.
Behind a TLS edge, set each node's `client_advertise` to its own public HTTPS
origin. Owner, redirect, and peer responses use these origins; gossip, peer
probes, and preview forwarding retain the internal `advertise_addr`. See
[TLS termination](deploy.md#tls-for-sdk-clients) for the edge setup.

Node death is honest: a dead node's sandboxes die with it (memory state is
node-local by design). SWIM detects the death and peers stop redirecting to
Expand Down
81 changes: 80 additions & 1 deletion docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,8 @@ sandboxd reads one JSON file (`-config`, default
| `restore_mode` | unset | clone and wake-restore memory mode: `copy`, `ondemand`, or `mmap`; use `mmap` for dense pools |
| `no_direct_io` | false | use buffered writable disks for Cloud Hypervisor cold boots and clones; recommended for dense ephemeral pools to avoid direct-I/O CoW journal contention |
| `no_balloon` | false | boot pool and template VMs without the virtio-balloon (cocoon otherwise returns 25% of guest memory to the host); clones inherit it from the golden. A guest that thrashes before deflate-on-OOM fires — a 16G build tier running a large typecheck — needs its whole memory |
| `advertise_addr` | = `listen` | the host:port clients reach this node at; returned as a claim's owner address and gossiped to peers. Must be routable when `listen` is a wildcard; a node with `mesh` set refuses to load while it names an unspecified host |
| `advertise_addr` | = `listen` | internal HTTP host:port for peer traffic and preview forwarding; also used by clients when `client_advertise` is unset. Must be routable when `listen` is a wildcard; a node with `mesh` set refuses to load while it names an unspecified host |
| `client_advertise` | unset | client-facing HTTP(S) origin for this node, e.g. `https://node-a.sandbox.example.com`; published in owner, redirect, and peer responses. No path, query, fragment, or userinfo |
| `bridges` / `networks` | unset | egress-lane attachment: a list of host bridge devices, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. A Linux bridge holds at most 1024 ports (kernel `BR_MAX_PORTS`), so an N-entry list raises the node's egress ceiling to N×1024 — VMs spread over the list by a stable hash of the VM name, so size it with headroom (the spread is statistical, not exact). `bridges` keeps the raw TAP-on-bridge attachment (taps in the root netns, no per-VM network namespace or CNI plugin execution); `networks` runs the CNI chain per VM. [Guarded egress](egress.md) on the egress lane (an egress-lane pool policy or any tenant policy) needs `bridges` and rejects a CNI network at load; none-lane pool policies ride the proxy on either |
| `volumes` | unset | node-local catalog of operator-managed dataset images: `[ {"name":"imagenet","path":"/srv/datasets/imagenet.img","directio":"off","tenants":["acme"]}, {"name":"scratch-db","path":"/srv/datasets/scratch.img","writable":true} ]`. Names match `^[a-z][a-z0-9_-]{0,19}$` and cannot start with `cocoon-`; paths are absolute; `directio` is `on`, `off`, or `auto` and defaults to `off` for both read-only and writable entries. `tenants` is an optional access list: empty means every authenticated scope, while every listed name must exist in the node's `tenants` config; root always has access. `writable` (default `false`) lets a claim request `mode: "rw"` on that entry — see [Dataset volumes](#dataset-volumes). The catalog is intentionally not part of the cluster digest |
| `secrets` | unset | node-side credentials the egress proxy injects by name: `[{"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}]`. A pool or tenant rule references the name; the value comes from the environment, never this file. See [egress](egress.md) |
Expand Down Expand Up @@ -458,6 +459,84 @@ Add `VOLUME_RW_IMAGE=/srv/datasets/scratch.img` (a second, writable image) to
also run the writable leg: a durable write across release, second-writer
exclusion, and a clean read-only claim afterward.

## TLS for SDK clients

sandboxd serves plain HTTP behind a TLS-terminating proxy. Configure a stable
client origin on **every node** reachable through that proxy:

```json
{
"listen": ":7777",
"advertise_addr": "node-a.internal:7777",
"client_advertise": "https://node-a.sandbox.example.com"
}
```

The mesh gossips both addresses. Client-facing owner replies, claim/template/
checkpoint redirects, and peer discovery use `client_advertise`. Checkpoint
probing, healing, deletion broadcasts, and preview forwarding continue to use
`advertise_addr` over internal HTTP. When `client_advertise` is unset, direct
HTTP deployments keep their existing address behavior. Configure all cluster
members before using external clients; a member without a client origin still
advertises its internal address to clients. Internal SDK users must also be
able to reach the configured client origins.

One proxy can serve the whole cluster, but each owner origin must route to
one particular node. An entry load balancer may choose any node for the initial
claim; a shared random-balancing owner origin cannot route later agent and
release requests to the owning node. Unlike Preview URLs, the SDK API does not
forward arbitrary sandbox requests between nodes.

Caddy reference configuration (node B uses the corresponding hostname and
internal upstream):

```caddyfile
node-a.sandbox.example.com {
reverse_proxy node-a.internal:7777 {
transport http {
versions 1.1
}
}
}

node-b.sandbox.example.com {
reverse_proxy node-b.internal:7777 {
transport http {
versions 1.1
}
}
}
```

For a private development CA, add `tls internal` to each site and provide
Caddy's root certificate to both SDKs using the
[TLS client settings](sdk.md#https-endpoints). For public DNS names, Caddy can
manage the certificates. Keep the upstream listeners and mesh private.

The edge must pass HTTP/1.1 `Connection: Upgrade`, `Upgrade: silkd`, and the
101 response, then relay both byte streams without response buffering. A
WebSocket-only upgrade allowlist is insufficient. Agent connections must not
negotiate HTTP/2. Configure stream/idle timeouts to exceed the longest relay;
Caddy's default stream timeout is unlimited. Configuration reloads may close
active streams; set `stream_close_delay` when reloads need a drain window.
The Upgrade tunnel does not need `flush_interval -1`.

The pinned Caddy integration runs both SDKs against two real sandboxd HTTP
handlers and relays (with a fake VM/guest), with an unreachable internal owner
address. It checks redirects, lookup, exec, port forwarding, half-close,
release, and certificate rejection:

```bash
cd e2e
CADDY_BIN=/path/to/caddy GOWORK=off go test -race -run TestCaddyTLSCluster -v .
```

For hardware acceptance, run the same SDK sequence from outside the node
network, including a guest HTTP server through `proxy_port`, and confirm an
A-to-B claim never dials B's internal address. A client-side TLS bridge alone
does not translate returned owners or redirects; use it only when every
returned endpoint is deliberately mapped through a local bridge.

## Preview URLs

`preview_listen` starts a second HTTP server that serves a sandbox's guest
Expand Down
5 changes: 3 additions & 2 deletions docs/sandboxd-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -550,8 +550,9 @@ silkd error; 504 when the command outlives `timeout_seconds` or the request. A h
## GET /v1/sandboxes/{id}/owner

Auth: the sandbox's own token. Answers `{"owner_addr": "host:port"}` when
this node owns the sandbox, 404 otherwise. Used by the SDK's `Lookup`
scatter.
this node owns the sandbox, 404 otherwise. With `client_advertise` configured,
`owner_addr` is that node's full HTTP(S) origin instead; the same contract applies
to claim and fork responses. Used by the SDK's `Lookup` scatter.

## GET /v1/info

Expand Down
3 changes: 2 additions & 1 deletion docs/sdk-python.md
Original file line number Diff line number Diff line change
Expand Up @@ -462,7 +462,8 @@ zero.
cwd that is not a directory, no exec bit) / `not_found` / `unimplemented` /
`internal`
- `ExitError(code, stderr, stdout)` — non-zero exit from `exec`
- `ProtocolError` — broken stream (EOF, oversized or undecodable frame)
- `ProtocolError` — a failed relay dial or TLS handshake, or a broken stream
(EOF, oversized or undecodable frame)

```python
try:
Expand Down
54 changes: 53 additions & 1 deletion docs/sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,59 @@ client, err := sandbox.Connect("10.0.0.5:7777",
`Info` answer it 403). On a cluster every node shares the same root token
and the same tenants set.
- `WithHTTPClient(client)` — replace the control-plane HTTP client when the
caller needs a custom transport, proxy, or timeout.
caller needs a custom transport, proxy, or timeout. An `http.Transport`'s
TLS configuration also supplies the agent relay's certificate settings;
HTTP proxies and custom dialers apply only to control requests.
- `WithTLSConfig(config)` — set certificate verification for both HTTPS
requests and agent relays. With `WithHTTPClient`, its transport must be an
`*http.Transport`; the SDK clones it rather than changing the caller's client.

### HTTPS endpoints

Both SDKs accept `host:port` (plain HTTP), `http://host[:port]`, or
`https://host[:port]`. The default ports are 80 and 443. IPv6 hosts use brackets.
Endpoints are origins: no userinfo, path prefix, query, or fragment.

For a public certificate, only the address changes:

```go
client, err := sandbox.Connect("https://node-a.sandbox.example.com",
sandbox.WithAPIToken(os.Getenv("SANDBOXD_TOKEN")))
```

For a private CA, load it into an `x509.CertPool` and pass
`sandbox.WithTLSConfig(&tls.Config{RootCAs: roots, MinVersion: tls.VersionTLS12})`.
The same trust configuration covers control requests and every agent relay.
Certificate and hostname verification are enabled by default.

Python uses a standard `ssl.SSLContext`:

```python
import os
import ssl
from cocoonsandbox import Client

client = Client(
"https://node-a.sandbox.example.com",
api_token=os.environ["SANDBOXD_TOKEN"],
ssl_context=ssl.create_default_context(cafile="edge-ca.pem"),
)
with client.new("rt:24.04") as sb:
assert sb.run(["true"]) == 0
```

Omit `ssl_context` for system trust. The SDK configures that context for
HTTP/1.1; use a dedicated context if another caller requires a different ALPN.
TLS handshakes share the existing dial timeout/cancellation budget. The agent
connection then uses HTTP/1.1 `Upgrade: silkd` and remains a bidirectional
stream; the guest protocol and port-forwarding frames do not change.

An explicit scheme in an owner, redirect, peer, or `Attach` address wins;
a bare address inherits the entry client's scheme. Trust settings are shared
across these connections. The SDK does not translate private addresses to
public names: configure each node's `client_advertise` as described in
[TLS deployment](deploy.md#tls-for-sdk-clients). Persist the complete owner URL
with the sandbox ID and token when using `Attach` in another process.

### Connecting to clusters

Expand Down
52 changes: 52 additions & 0 deletions e2e/tls_client.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
"""Exercise both SDK planes through the TLS cluster fixture."""

import socket
import ssl
import sys

from cocoonsandbox import APIError, Client, ProtocolError


def main() -> None:
entry, owner, ca = sys.argv[1:]
context = ssl.create_default_context(cafile=ca)
client = Client(entry, api_token="node-token", ssl_context=context)
with client.new("rt:24.04") as sb:
assert sb.owner == owner, sb.owner
assert sb.exec("echo", "tls") == "tls\n"
assert client.lookup(sb.id, sb.token).owner == owner
assert client.attach(owner.removeprefix("https://"), sb.id, sb.token).exec("echo", "bare") == "bare\n"
with sb.dial_port(5000) as port:
port.send(b"tail")
port.close_write()
output = b""
while chunk := port.recv():
output += chunk
assert output == b"tail", output
with (
sb.proxy_port("127.0.0.1:0", 5000) as proxy,
socket.create_connection(proxy.getsockname(), timeout=5) as conn,
):
conn.sendall(b"proxy")
conn.shutdown(socket.SHUT_WR)
output = b""
while chunk := conn.recv(4096):
output += chunk
assert output == b"proxy", output
untrusted = Client(entry, api_token="node-token")
try:
untrusted.info()
except APIError:
pass
else:
raise AssertionError("control request accepted an untrusted certificate")
try:
untrusted.attach(owner, sb.id, sb.token).exec("echo", "untrusted")
except ProtocolError:
pass
else:
raise AssertionError("relay accepted an untrusted certificate")


if __name__ == "__main__":
main()
Loading