Skip to content

feat: add kubernetes-agent-sandbox template to coder-labs - #1230

Merged
bpmct merged 31 commits into
mainfrom
coder-labs-kubernetes-agent-sandbox
Nov 3, 2026
Merged

bpmct merged 31 commits into
mainfrom
coder-labs-kubernetes-agent-sandbox

Conversation

@bpmct

@bpmct bpmct commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Description

Adds a coder-labs template that runs each workspace in an agent-sandbox Sandbox. You and your AI agent share one sandbox.

  • Kernel isolation: gVisor by default, or a Kata microVM (QEMU or Cloud Hypervisor).
  • Network isolation: a NetworkPolicy per profile: Fully isolated (default, Coder only), GitHub only, or Public internet.
  • Home: persistent (a disk, with a prebuilt preset per profile) or scratch (claimed from a SandboxWarmPool, deleted on stop, for Fully isolated and GitHub only).
  • Image: build/Dockerfile bakes in the Coder agent, code-server, and Claude Code, so a start downloads nothing. Users build it themselves.
  • Cluster setup: cluster/ is a Terraform configuration that creates the namespace, RBAC, a SandboxTemplate and warm pool per runtime and profile, and an optional EKS guardrail. Its coder_variables output is the --variables-file for coder templates push, so the template and pools agree.
  • README: why agent-sandbox, startup speed with a real-time demo GIF, agent-sandbox vs Pods and when each fits, firewall options (NetworkPolicy, ClusterNetworkPolicy, Cilium, a proxy), and ingress options. No parameter or variable tables, since the registry shows them.
  • Icon: .icons/agent-sandbox.svg, the agent-sandbox logo (Apache-2.0) in the project's purple.

Startup on EKS, create to connected agent: 3.5 to 6.5 s from a warm pool, 4 to 7 s from a prebuild, about 17 s cold on gVisor.

Type of Change

  • New module
  • New template
  • Bug fix
  • Feature/enhancement
  • Documentation
  • Other

Template Information

Path: registry/coder-labs/templates/kubernetes-agent-sandbox

Testing & Validation

  • Tests pass locally (terraform test: 36 template runs and 18 cluster/ runs, with mock providers). CI runs .tftest.hcl only for modules, so these do not run in CI.
  • Code formatted (bun fmt:ci, terraform fmt)
  • Changes tested locally (EKS with agent-sandbox v1.0.4, gVisor release-20261102.0, Kata, VPC CNI v1.22.4)
Live test results
Case Agent ready
Scratch home from a warm pool 3.5 to 6.5 s
Persistent home from a prebuild 4 to 7 s
Persistent home, cold, gVisor / Kata QEMU about 17 s / 19 s
Persistent home, cold, with a coder/coder clone about 58 s
  • Fully isolated: only Coder is reachable. DNS, GitHub, public IPs, cloud metadata, the API server, and other Pods are blocked.
  • GitHub only: GitHub works by name and by IP. Everything else is blocked.
  • Public internet: ports 80 and 443 are open. Private ranges, metadata, and the cluster are blocked.
  • Policy check: the agent starts only after the NetworkPolicy is in force, and re-checks every 5 s. When I deleted a workspace's NetworkPolicy, the agent stopped about 5 s later.
  • Web terminal on gVisor: gVisor before release-20261019.0 blocks on a zero-length write to a PTY master, which the agent does on every resize, so the terminal took no input. Fixed upstream in google/gvisor@6b4e208e4. The README requires that release or later. Verified with the stock agent's write path on old and new runsc.
  • After the review fixes: all four preset types (Fully isolated and GitHub only, scratch and persistent) connected, the agent was PID 1 under bash, the web terminal took input, and the profiles blocked and allowed as expected.
  • Not tested live: Kata Cloud Hypervisor, and GKE.
Plan and decisions
  • Why a second Terraform root in cluster/: warm pools, SandboxTemplates, and RBAC are cluster objects that an administrator owns, not per-workspace objects, so the template cannot create them. A Terraform root keeps the admin setup in the same language as the template, and its coder_variables output keeps both in line.
  • Provider: hashicorp/kubernetes for both the template and cluster/. kubernetes_manifest reads CRD schemas at plan time, so the provisioner needs CRD read access. cluster/ sets force_conflicts so it can take over objects created with kubectl.
  • Defaults: gVisor is the default runtime because it works on normal nodes. Presets are the default, and each persistent preset has prebuild_instances prebuilds.
  • Two homes: scratch uses warm pools. Persistent uses prebuilds, because a SandboxClaim cannot carry a disk or be suspended yet (Feature Request: Support volumeClaimTemplates in SandboxTemplate / SandboxClaim kubernetes-sigs/agent-sandbox#225).
  • Scratch profiles: one SandboxTemplate and pool per runtime and profile (coder-<runtime>[-github]), so every Pod in a pool has the same rules. VPC CNI groups Pods by name, so mixing profiles in one pool could leak rules.
  • Comma-separated runtimes and scratch_profiles: Coder checks presets with every template variable as a string, so list variables fail the preset check.
  • Shared files: shared/start-agent.sh (policy check, agent as PID 1) and shared/defaults.json (GitHub ranges, DNS, resources) are read by both the template and cluster/.
  • Policy check: in VPC CNI's default mode, a new Pod is open until its policy applies (up to about 30 s), and the rules can drop while it runs. The check waits until the API server is unreachable, and stops the agent if it becomes reachable. network_policy_gate = false turns it off where a profile can reach the API server.
  • Agent as PID 1: under Kata, the web terminal got no input unless the agent was PID 1.
  • coder_host_ip: Fully isolated has no DNS, so a host name access URL needs an /etc/hosts entry. A precondition fails the build if it is missing.
  • AI Gateway token: the workspace does not get the owner's session token, which has coder:all. The start build uses it once to create a token with only user:read_personal (through hashicorp/http, deferred to apply with timestamp()), and the workspace gets that. The AI Gateway checks that a key is valid, not its scopes, so this works today. Coder deletes the session token before stop, so the scoped token cannot be revoked on stop and expires at the default token lifetime. Prebuilds create none; the claim build does.
  • Dockerfile: no comment before FROM, because prettier-plugin-sh corrupts Dockerfiles with a leading comment block.

Related Issues

None


🤖 This PR was generated by Coder Agents on behalf of @bpmct.

bpmct added 26 commits November 2, 2026 18:40
… Dockerfile header comment that breaks prettier-plugin-sh
… agent-sandbox logo as the icon

The logo is logo.svg from kubernetes-sigs/agent-sandbox (Apache-2.0),
filled with the project's purple instead of white so it shows on light
backgrounds.
…Claude Code env directly instead of using the claude-code module

The claude-code module reads the deprecated coder_task data source through
coder-utils, which adds a deprecation warning to every build. Claude Code
is already in the image, so the template only needs DISABLE_AUTOUPDATER and,
with the AI Gateway, ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN.
…agent as PID 1 so the web terminal works under Kata

With a shell as PID 1 and the agent as its child, the web terminal under
Kata showed a prompt but took no input. The policy re-check now runs in
the background and the shell execs the agent. If the policy is lost, the
re-check sends SIGTERM to PID 1 and then kills the rest, so the container
still restarts into the gate.

Also note in Limits that the web terminal under gVisor needs a Coder agent
that skips empty writes to the PTY.
… Git repository with the git_repo_url variable
…itHub only warm pool that clones the repository
… the cluster YAML with a Terraform installer
… when agent-sandbox fits and what startup speed measures
…the fit guidance in the agent-sandbox roadmap
@bpmct
bpmct marked this pull request as ready for review November 3, 2026 00:02
@bpmct
bpmct requested a review from DevelopmentCats November 3, 2026 00:02
…ent-sandbox

- Drop the Parameters and Variables tables from the README; the registry
  shows them. Use absolute links for repo files, add a warning for the
  tokens in the workspace, and require gVisor or Kata, not only gVisor.
- Add enable_claude_code and dns_servers, order variable fields as
  description then type, and run the agent script with bash.
- Stop the warm pool clone after 5 failures so the Pod restarts.
- Trim the demo GIF to the startup (8.4 MB to 1.1 MB).

@DevelopmentCats DevelopmentCats left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aside from my NIT this LGTM

bpmct added 2 commits November 3, 2026 12:09
…eed details to Speed tuning and clarify the session token
…e pool agent URL from the workspace

Each claim passes data.coder_workspace.me.access_url to its pooled Pod
through an annotation, so cluster/ no longer needs coder_access_url. It
stays as an optional input for the egress port and the coder_host_ip
host name.
@bpmct
bpmct requested a review from DevelopmentCats November 3, 2026 12:14
bpmct and others added 2 commits November 3, 2026 12:32
…e AI Gateway a narrowly scoped token

The workspace got the owner's session token, which has full access to the
Coder API. The start build now uses it once to create a token with only
the user:read_personal scope, which the AI Gateway accepts, and the
workspace gets that token instead.
@bpmct
bpmct enabled auto-merge (squash) November 3, 2026 13:41
@bpmct
bpmct merged commit d5159d5 into main Nov 3, 2026
4 checks passed
@bpmct
bpmct deleted the coder-labs-kubernetes-agent-sandbox branch November 3, 2026 13:43
bpmct added a commit that referenced this pull request Nov 3, 2026
…e token notes from a warning to Limits (#1231)

## Description

Removes the warning box at the top of the kubernetes-agent-sandbox
README. The token issue it described was fixed in #1230: the workspace
now gets a token with only the `user:read_personal` scope, not the
owner's `coder:all` session token. So a warning at the top is too loud.

The two notes now live under Limits:

- The AI Gateway token is scoped, but Coder cannot revoke it on stop.
- A scratch home keeps its agent token in a Pod annotation.

README only. No template change.

## Type of Change

- [ ] New module
- [ ] New template
- [ ] Bug fix
- [ ] Feature/enhancement
- [x] Documentation
- [ ] Other

## Template Information

**Path:** `registry/coder-labs/templates/kubernetes-agent-sandbox`

## Testing & Validation

- [x] README validator passes
- [x] Prettier passes

## Related Issues

Follow-up to #1230

---

🤖 This PR was generated by Coder Agents on behalf of @bpmct.
bpmct added a commit that referenced this pull request Nov 3, 2026
…s) template (#1232)

## Description

Adds a `coder-labs` template that runs each workspace in a plain
Kubernetes Pod, with a host name firewall. It needs no controller and no
CRDs. It is the plain-Pods sibling of [agent-sandbox
(Kubernetes)](https://registry.coder.com/templates/coder-labs/kubernetes-agent-sandbox)
(#1230).

- **Network isolation:** a NetworkPolicy lets the workspace reach only
Coder and its own Squid proxy Pod. The proxy allows host names per
profile: Fully isolated (default, no proxy), GitHub only, GitHub and
package registries, Public internet, or Custom (host names and ports
typed in the form). Workspaces have no DNS. The proxy refuses raw IPs,
private addresses, metadata, and its own cache manager.
- **Kernel isolation:** the node's container runtime by default, the
fastest. List `gvisor`, `kata-qemu`, or `kata-clh` in `runtimes` to
offer them in a "Kernel isolation" dropdown.
- **Speed:** each fixed profile is a preset with prebuilds. A claim
changes no Pod.
- **Shared with #1230:** `build/Dockerfile`, the policy gate in
`start-agent.sh`, the scoped AI Gateway token, and the code-server,
git-clone, and Claude Code setup.
- **Cluster setup:** `cluster/` is a small Terraform configuration for
the namespace and the RBAC.
- **Names:** "Plain Sandbox (Kubernetes)" and "agent-sandbox
(Kubernetes)", which also renames #1230, so the names do not get cut off
on registry cards.
- **Proxy image:** `build/egress-proxy/Dockerfile`, Squid 7 on Alpine,
from the egress firewall in coder/dogfood. Users build it themselves.

Startup on EKS, create to connected agent: 3 to 7 s from a prebuild,
about 15 s cold or start after stop.

## Type of Change

- [ ] New module
- [x] New template
- [ ] Bug fix
- [ ] Feature/enhancement
- [ ] Documentation
- [ ] Other

## Template Information

**Path:** `registry/coder-labs/templates/kubernetes-sandboxed`

## Testing & Validation

- [x] Tests pass locally (`terraform test`: 35 template runs and 3
`cluster/` runs, with mock providers). CI runs `.tftest.hcl` only for
modules, so these do not run in CI.
- [x] Code formatted (`prettier --check`, `terraform fmt`), README
validator passes
- [x] Changes tested locally (EKS, VPC CNI, gVisor `release-20261102.0`)

<details>
<summary>Live test results</summary>

| Case | Result |
|---|---|
| Claim from a prebuild (Fully isolated, GitHub and package registries,
Public internet) | build 1.5 to 1.7 s, agent ready 3.2 to 6.8 s |
| No prebuild (Custom) | ready 15.5 s |
| Stop, then start | stop 5 s, ready 15.2 s |
| gVisor, GitHub only | `4.19.0-gvisor`, GitHub 200, npm blocked, PTY
works |
| GitHub and package registries | GitHub, npm, PyPI 200; example.com
blocked; DNS blocked; direct egress blocked |
| Public internet | example.com 200; raw IP, IPv6 literal, metadata,
cache manager blocked (403 in the Squid log) |
| Custom (`example.com`, `*.wikipedia.org`, 443 and 80) | example.com on
both ports and wikipedia.org 200, GitHub blocked |
| AI Gateway | scoped token set, gateway 200 |

Kata is not tested: the Kata nodes had no free CPU, so the Pod stayed
Pending.

</details>

<details>
<summary>Decisions</summary>

- **`hashicorp/kubernetes`, not `Mastercard/restapi`:** the devrel
prototype used restapi because it starts faster (claim build about 0.8
s). This uses the same provider as #1230, supports `use_kubeconfig`, and
is easy to mock in tests. A claim build takes about 1.6 s.
- **Bare Pods:** no Deployment or controller, so stop and start do not
wait on one. `kubernetes_pod_v1` waits until the Pod runs.
- **Proxy as its own Pod, not a sidecar:** nothing in the workspace can
change its rules. The workspace finds it by the Service's cluster IP,
because it has no DNS.
- **No DNS in the workspace:** the nameserver is `127.0.0.1`, so a
lookup fails at once instead of timing out against a blocked server.
- **Presets set `runtime = "none"`** when `runtimes` is set. Without it,
picking gVisor with a preset claims a prebuild and fails with "Immutable
parameter changed".
- **`coder_egress_port` defaults to 0, not null,** so `coder templates
push` does not ask for it.
- **Pod names end in `-pod` after an id that is new on each start,** for
the VPC CNI rule grouping issue (same as #1230).

</details>

<details>
<summary>Review</summary>

A Claude (Fable) review found no blockers. Fixed: DNS lookups that hung,
Squid `deny manager` and IPv6 literals, a proxy Pod restart on a config
change, `distinct()` on custom ports, README accuracy (the ConfigMap,
"no Pod" changes on claim, Kata numbers, a DNS limit), a stale
Dockerfile comment, and more tests. Not changed: the `custom_problems`
form helper, and the explicit Pod timeouts (the provider default is 5
min).

</details>

## Related Issues

None

---

🤖 This PR was generated by Coder Agents on behalf of @bpmct.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants