Skip to content

feat(registry/coder-labs/templates): add the Plain Sandbox (Kubernetes) template - #1232

Merged
bpmct merged 4 commits into
mainfrom
kubernetes-sandboxed
Nov 3, 2026
Merged

bpmct merged 4 commits into
mainfrom
kubernetes-sandboxed

Conversation

@bpmct

@bpmct bpmct commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Description

Adds a coder-labs template that runs each workspace in a plain Kubernetes Pod, with a host name firewall. It needs no controller and no CRDs. It is the plain-Pods sibling of agent-sandbox (Kubernetes) (#1230).

  • Network isolation: a NetworkPolicy lets the workspace reach only Coder and its own Squid proxy Pod. The proxy allows host names per profile: Fully isolated (default, no proxy), GitHub only, GitHub and package registries, Public internet, or Custom (host names and ports typed in the form). Workspaces have no DNS. The proxy refuses raw IPs, private addresses, metadata, and its own cache manager.
  • Kernel isolation: the node's container runtime by default, the fastest. List gvisor, kata-qemu, or kata-clh in runtimes to offer them in a "Kernel isolation" dropdown.
  • Speed: each fixed profile is a preset with prebuilds. A claim changes no Pod.
  • Shared with feat: add kubernetes-agent-sandbox template to coder-labs #1230: build/Dockerfile, the policy gate in start-agent.sh, the scoped AI Gateway token, and the code-server, git-clone, and Claude Code setup.
  • Cluster setup: cluster/ is a small Terraform configuration for the namespace and the RBAC.
  • Names: "Plain Sandbox (Kubernetes)" and "agent-sandbox (Kubernetes)", which also renames feat: add kubernetes-agent-sandbox template to coder-labs #1230, so the names do not get cut off on registry cards.
  • Proxy image: build/egress-proxy/Dockerfile, Squid 7 on Alpine, from the egress firewall in coder/dogfood. Users build it themselves.

Startup on EKS, create to connected agent: 3 to 7 s from a prebuild, about 15 s cold or start after stop.

Type of Change

  • New module
  • New template
  • Bug fix
  • Feature/enhancement
  • Documentation
  • Other

Template Information

Path: registry/coder-labs/templates/kubernetes-sandboxed

Testing & Validation

  • Tests pass locally (terraform test: 35 template runs and 3 cluster/ runs, with mock providers). CI runs .tftest.hcl only for modules, so these do not run in CI.
  • Code formatted (prettier --check, terraform fmt), README validator passes
  • Changes tested locally (EKS, VPC CNI, gVisor release-20261102.0)
Live test results
Case Result
Claim from a prebuild (Fully isolated, GitHub and package registries, Public internet) build 1.5 to 1.7 s, agent ready 3.2 to 6.8 s
No prebuild (Custom) ready 15.5 s
Stop, then start stop 5 s, ready 15.2 s
gVisor, GitHub only 4.19.0-gvisor, GitHub 200, npm blocked, PTY works
GitHub and package registries GitHub, npm, PyPI 200; example.com blocked; DNS blocked; direct egress blocked
Public internet example.com 200; raw IP, IPv6 literal, metadata, cache manager blocked (403 in the Squid log)
Custom (example.com, *.wikipedia.org, 443 and 80) example.com on both ports and wikipedia.org 200, GitHub blocked
AI Gateway scoped token set, gateway 200

Kata is not tested: the Kata nodes had no free CPU, so the Pod stayed Pending.

Decisions
  • hashicorp/kubernetes, not Mastercard/restapi: the devrel prototype used restapi because it starts faster (claim build about 0.8 s). This uses the same provider as feat: add kubernetes-agent-sandbox template to coder-labs #1230, supports use_kubeconfig, and is easy to mock in tests. A claim build takes about 1.6 s.
  • Bare Pods: no Deployment or controller, so stop and start do not wait on one. kubernetes_pod_v1 waits until the Pod runs.
  • Proxy as its own Pod, not a sidecar: nothing in the workspace can change its rules. The workspace finds it by the Service's cluster IP, because it has no DNS.
  • No DNS in the workspace: the nameserver is 127.0.0.1, so a lookup fails at once instead of timing out against a blocked server.
  • Presets set runtime = "none" when runtimes is set. Without it, picking gVisor with a preset claims a prebuild and fails with "Immutable parameter changed".
  • coder_egress_port defaults to 0, not null, so coder templates push does not ask for it.
  • Pod names end in -pod after an id that is new on each start, for the VPC CNI rule grouping issue (same as feat: add kubernetes-agent-sandbox template to coder-labs #1230).
Review

A Claude (Fable) review found no blockers. Fixed: DNS lookups that hung, Squid deny manager and IPv6 literals, a proxy Pod restart on a config change, distinct() on custom ports, README accuracy (the ConfigMap, "no Pod" changes on claim, Kata numbers, a DNS limit), a stale Dockerfile comment, and more tests. Not changed: the custom_problems form helper, and the explicit Pod timeouts (the provider default is 5 min).

Related Issues

None


🤖 This PR was generated by Coder Agents on behalf of @bpmct.

@bpmct bpmct changed the title feat(registry/coder-labs/templates/kubernetes-sandboxed): add a plain Pod template with a host name firewall feat(registry/coder-labs/templates): add the Plain Sandbox (Kubernetes) template Nov 3, 2026
@bpmct
bpmct marked this pull request as ready for review November 3, 2026 20:14
@bpmct
bpmct force-pushed the kubernetes-sandboxed branch from c2b398e to eecaeba Compare November 3, 2026 20:15

@DevelopmentCats DevelopmentCats left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bpmct
bpmct merged commit aa7ce90 into main Nov 3, 2026
4 checks passed
@bpmct
bpmct deleted the kubernetes-sandboxed branch November 3, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants