Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
492747c
feat: add e2b backend support
Blizzard-cyber Aug 25, 2026
3bb12f0
docs: document e2b backend usage
Blizzard-cyber Aug 25, 2026
b83e0dc
Remove the e2b-template build step: official Templates are pre-built …
Odysseusqs Aug 25, 2026
c462f8a
Pin work container exec user to agent (e2b execs default to the SDK u…
Odysseusqs Aug 25, 2026
e52ea7e
Remove --max-workers; tasks run fully in parallel as on main
Odysseusqs Aug 25, 2026
a421d51
Deploy the judge server on the host as on main; drop the managed e2b …
Odysseusqs Aug 25, 2026
941f894
Restore main's dockerfile apt-mirror handling and image hash recipe
Odysseusqs Aug 25, 2026
9c124e1
Trim non-essential deviations from main in cli, constants, and .gitig…
Odysseusqs Aug 25, 2026
d135301
List submission archives with tar --quoting-style=literal so CJK file…
Odysseusqs Aug 25, 2026
1762391
Add claude-code-2.1.214 agent variant for Claude 5-family models
Odysseusqs Jul 18, 2026
a918b54
Add opencode agent with visualizer support
Odysseusqs Jul 20, 2026
9b85eda
Add unified --effort flag translated per-agent
Odysseusqs Jul 20, 2026
37d3af0
Add codex-0.145.0 pinned agent
Odysseusqs Aug 19, 2026
c43a72c
opencode: chown agent HOME after root npm install so runtime logs are…
Odysseusqs Aug 25, 2026
5d8ade2
feat(e2b): add SFORGE_E2B_TEMPLATE_NAMESPACE for public template reso…
Blizzard-cyber Aug 27, 2026
c4283de
fix(e2b): harden sandbox cleanup and document runtime behavior
Blizzard-cyber Sep 3, 2026
55badf7
docs: add E2B single-task example, condense plan limits, sync agent/C…
Odysseusqs Sep 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ logs/
# Experiments (generated at runtime)
experiments/

# Benchmark task definitions downloaded by sforge fetch-tasks
/tasks/

# IDE
.idea/
.vscode/
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,7 @@ SFORGE_AGENT_API_KEY="sk-xxx" \
**Step-by-step examples:**
- [Single task on local Docker](examples/single-task-docker/) — run one task end-to-end with Docker
- [All tasks on Kubernetes](examples/all-tasks-k8s/) — run the full suite on a K8s cluster with the **official leaderboard setting**
- [E2B backend](https://bytedance-seed.github.io/EdgeBench/en/configuration/container-backends#e2b-backend) — run Work and Judge environments on E2B without operating a cluster

> [!IMPORTANT]
> - **Official setting** — leaderboard numbers use the [official experiment YAMLs](examples/all-tasks-k8s/) unchanged, including the time budget, stop hook, auto-eval, submission cooldowns, and hardware resource limits.
Expand Down
2 changes: 2 additions & 0 deletions docs/.vitepress/config.mts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ export default defineConfig({
text: 'Examples',
items: [
{ text: 'Single Task (Docker)', link: '/en/examples/single-task-docker' },
{ text: 'Single Task (E2B)', link: '/en/examples/single-task-e2b' },
{ text: 'All Tasks (Kubernetes)', link: '/en/examples/all-tasks-k8s' },
],
},
Expand Down Expand Up @@ -105,6 +106,7 @@ export default defineConfig({
text: '示例',
items: [
{ text: '单任务运行 (Docker)', link: '/zh/examples/single-task-docker' },
{ text: '单任务运行 (E2B)', link: '/zh/examples/single-task-e2b' },
{ text: '全部任务 (Kubernetes)', link: '/zh/examples/all-tasks-k8s' },
],
},
Expand Down
133 changes: 117 additions & 16 deletions docs/en/configuration/container-backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,21 @@ title: Container Backends

# Container Backends

SForge uses a container backend to create work containers and judge containers. The default backend is local Docker. Switch to the `k8s` backend when you want SForge workloads to run in a Kubernetes cluster.
SForge uses a container backend to create work and judge environments. The
default backend is local Docker. Use `k8s` for an existing Kubernetes cluster,
or `e2b` for E2B-hosted environments.

## Choosing a Backend

| Backend | Best for | Main requirements |
|---------|----------|-------------------|
| `docker` | Local development, task debugging, single-machine evaluation, small experiments | A working local Docker daemon |
| `k8s` | Shared clusters, high-concurrency batch evaluation, running Work/Judge pods in a cluster | `kubectl` access, images pullable by the cluster, and a Judge URL reachable from pods |
| `e2b` | Remote Work/Judge Sandboxes without operating a cluster | E2B API access, E2B Templates, and a routable Judge Server; see [E2B Backend](#e2b-backend) |

If you are trying a task locally, use the default Docker backend first. The Kubernetes backend is intended for team environments that already have a cluster and container registry.
The E2B backend has a one-time Template preparation step before a new or
changed task image can run.

::: warning Docker backend does not scale to large batches
Each task runs a work container plus ephemeral judge containers, each with its own CPU/memory limits. Running many tasks concurrently on one host (roughly **20+**) causes severe resource contention even on a high-end server. For large batch runs, use the `k8s` backend.
Expand All @@ -29,15 +34,13 @@ sforge run \
--task ad_placement_optimization \
--agent claude-code \
--backend k8s \
--judge-url http://10.0.0.12:8080 --backend k8s
--judge-url http://10.0.0.12:8080

# Override through environment variables
export SFORGE_BACKEND=k8s
sforge run \
--task ad_placement_optimization \
--agent claude-code \
--backend k8s \
--judge-url http://10.0.0.12:8080
--agent claude-code
```

Experiment YAML can also set the default backend:
Expand All @@ -62,9 +65,7 @@ sforge serve
export SFORGE_AGENT_API_KEY="sk-..."
sforge run \
--task ad_placement_optimization \
--agent claude-code \
--backend k8s \
--judge-url http://10.0.0.12:8080
--agent claude-code
```

The Docker backend:
Expand Down Expand Up @@ -222,13 +223,113 @@ Notes:
- Your kubeconfig must be allowed to create and delete `NetworkPolicy` resources.
- DNS, egress gateways, and policy implementations differ between clusters, so validate isolation with a small task first.

## E2B Backend

The E2B backend runs Work, Judge, and Game environments as E2B Sandboxes.
The Judge Server itself runs on a host you control, exactly as with the other
backends: start `sforge serve` on a machine the Sandboxes can reach and pass
its address via `--judge-url`.

### Docker Images and E2B Templates

A Docker image and an E2B Template are different runtime objects: E2B
Sandboxes start from Templates, not from Docker image references. Official
E2B Templates for the published EdgeBench tasks are pre-built and published
by the SForge maintainers, so no template preparation step is required.
Set `SFORGE_E2B_TEMPLATE_NAMESPACE` to the namespace that publishes the
Templates. `sforge run --backend e2b` then derives each task image's public
Template reference automatically (`edgebench.work.foo_bar:abc123` ->
`edgebench/edgebench-work-foo-bar:abc123`). Set `SFORGE_E2B_TEMPLATE_MAP`
only to override this mapping, for example to run self-built Templates for a
modified task.

### Run a Task

Start the Judge Server on a host with a publicly routable address (agents
inside E2B Sandboxes submit to it over the internet):

```bash
uv sync --extra e2b
export E2B_API_KEY=...
export SFORGE_E2B_TEMPLATE_NAMESPACE=edgebench
export SFORGE_ADMIN_SECRET=... # same value on the serve and run hosts

sforge serve --host 0.0.0.0 --port 8080
```

Then run the agent, pointing `--judge-url` at that host:

```bash
export E2B_API_KEY=...
export SFORGE_E2B_TEMPLATE_NAMESPACE=edgebench
export SFORGE_ADMIN_SECRET=... # same value as on the serve host
export SFORGE_AGENT_API_KEY=...
export SFORGE_AGENT_API_BASE_URL=...
export SFORGE_AGENT_MODEL=...

sforge run \
--backend e2b \
--task ad_placement_optimization \
--agent claude-code \
--judge-url http://YOUR_HOST:8080
```

The Judge Server needs `E2B_API_KEY` too: registrations from an e2b run make
it create Judge and Game Sandboxes on E2B (mirroring how a k8s run makes it
schedule judge pods). Work Sandboxes submit to the Judge Server's public URL;
the Judge Server reaches Game Sandboxes through the E2B service gateway.

Each E2B evaluation is scoped to one task. Supplying multiple task IDs only
batch-schedules several independent evaluations; it does not combine tasks into
one Work or Judge environment. The validation described here invoked one task
per `sforge run` command.

### Plan Limits and Run Sizing

E2B plan limits are part of the runtime contract:

- **Resources**: Template CPU and memory are fixed at build time; over-limit
requests fail at Template creation. Rebuild with supported values, since
`--work-*` and `--judge-*` flags cannot resize an existing Template.
- **Sandbox lifetime**: Keep the agent timeout well below the team's maximum
Sandbox lifetime, which must also cover agent install, archive extraction,
Judge evaluation, result collection, and cleanup. At cleanup SForge caps the
remaining lifetime at 60 seconds and deletes the Sandbox, regardless of the
configured TTL.
- **Evaluator timeout**: Judge Sandboxes share the same limit, so a task whose
`judge.eval_timeout` exceeds it may not finish. Prefer a plan with a longer
lifetime over lowering the timeout.
- **Concurrency**: Tasks run fully in parallel. Budget Work, Judge, and Game
Sandboxes for all selected tasks, plus overlap during evaluation and cleanup.
- **Final score**: As on other backends, the score is the best completed agent
or auto-eval submission. `final_archive.tar.gz` is a recovery snapshot, not
an implicit submission at timeout. For short runs, allow at least one
auto-eval to finish, or have the agent call `sforge-submit` before the end.

### Required User Input

| Input | When required |
| --- | --- |
| `E2B_API_KEY` | E2B runs, on both the `sforge run` host and the Judge Server host |
| A publicly routable `--judge-url` | All E2B runs |
| Agent API key, base URL, and model | Agent runs |

Host `HTTP_PROXY` and `HTTPS_PROXY` variables are not copied into E2B Work,
Judge, or Game Sandboxes. A remote-reachable proxy is forwarded only when the
user explicitly configures `SFORGE_HTTP_PROXY` or `SFORGE_HTTPS_PROXY`; isolated
tasks remove proxy variables entirely. See [Network Isolation](/en/features/network-isolation).

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `kubectl cluster-info failed` | Wrong kubeconfig, unreachable cluster, or wrong namespace context | Check `kubectl -n <namespace> cluster-info`; set `SFORGE_K8S_KUBECONFIG` and `SFORGE_K8S_NAMESPACE` if needed |
| Pod never becomes Running | Image pull failure, scheduling failure, or insufficient resources | Run `kubectl -n <namespace> describe pod <pod>` and check events |
| Pod cannot pull image | Image was not pushed to a registry reachable by the cluster, or registry credentials are missing | Run `sforge push`; verify `SFORGE_K8S_IMAGE_REGISTRY`; configure image pull secrets for the namespace |
| Work pod cannot submit evaluations | `--judge-url` is not reachable from pods, or the Judge server is not listening externally | Start `sforge serve` with `--host 0.0.0.0` and pass a pod-reachable IP/Service URL with `--judge-url` |
| `--disable-internet` has no effect | CNI does not enforce NetworkPolicy, or permissions are insufficient | Confirm NetworkPolicy support and verify the current identity can create NetworkPolicies |
| Pod stays Pending after setting a node selector | No nodes match the selector | Check `SFORGE_K8S_NODE_SELECTOR` and node labels |
| Backend | Symptom | Likely cause | Fix |
| --- | --- | --- | --- |
| Kubernetes | `kubectl cluster-info failed` | Wrong kubeconfig, unreachable cluster, or wrong namespace context | Check `kubectl -n <namespace> cluster-info`; set `SFORGE_K8S_KUBECONFIG` and `SFORGE_K8S_NAMESPACE` if needed |
| Kubernetes | Pod never becomes Running | Image pull failure, scheduling failure, or insufficient resources | Run `kubectl -n <namespace> describe pod <pod>` and check events |
| Kubernetes | Pod cannot pull an image | The image is unavailable to cluster nodes, or registry credentials are missing | Run `sforge push`; verify `SFORGE_K8S_IMAGE_REGISTRY`; configure image pull secrets for the namespace |
| Kubernetes | Work cannot reach Judge | The Judge URL is not pod-reachable, or the server is not listening externally | Start `sforge serve` with `--host 0.0.0.0` and pass a pod-reachable IP or Service URL |
| Kubernetes | `--disable-internet` has no effect | CNI does not enforce NetworkPolicy, or permissions are insufficient | Confirm NetworkPolicy support and permissions |
| Kubernetes | Pod remains Pending with a node selector | No nodes match the selector | Check `SFORGE_K8S_NODE_SELECTOR` and node labels |
| E2B | Template not found | The Template namespace is missing or incorrect, the task image version has no published official Template, or the task uses a modified image | Check `SFORGE_E2B_TEMPLATE_NAMESPACE` and the task image version, or point `SFORGE_E2B_TEMPLATE_MAP` at a self-built Template |
| E2B | Agent-side Judge requests intermittently time out | The Work Sandbox could not reach the Judge Server over the internet | Check that the Judge host and port are publicly reachable, then retry the agent submission |
| E2B | A run ends at the configured task timeout with `Sandbox not found` | The team-level maximum Sandbox lifetime was reached before final extraction and cleanup | Shorten the agent timeout, reduce multi-wave batch duration, or use a plan with a longer Sandbox lifetime |
| E2B | A completed run retains a Sandbox for more than 60 seconds | Both the provider-side termination deadline and the immediate deletion request failed | Inspect the lifecycle cleanup error in the run or Judge Server log |
14 changes: 12 additions & 2 deletions docs/en/configuration/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,10 @@ Used when running agents via `sforge run`. These are injected as container envir

| Variable | Purpose |
|----------|---------|
| `SFORGE_AGENT_API_KEY` | API key passed to the selected agent using that agent's expected env var (`ANTHROPIC_AUTH_TOKEN` or `CODEX_API_KEY`) |
| `SFORGE_AGENT_API_KEY` | API key passed to the selected agent using that agent's expected env var (`ANTHROPIC_AUTH_TOKEN` for Claude Code, `CODEX_API_KEY` for Codex, `ANTHROPIC_API_KEY` for OpenCode) |
| `SFORGE_AGENT_API_BASE_URL` | API base URL override passed through the selected agent's base-url env var when it has one |
| `SFORGE_AGENT_MODEL` | Model override for the agent |
| `SFORGE_AGENT_EFFORT` | Reasoning effort (`low`, `medium`, `high`, or `max`) |
| `SFORGE_AGENT_TIMEOUT` | Agent timeout in seconds |
| `SFORGE_AGENT_EXTRA_ENV` | Extra env vars for agent container, format: `"KEY1=VAL1,KEY2=VAL2"` |
| `SFORGE_HTTP_PROXY` | HTTP proxy injected into the agent container at run time. Not recommended for normal use; incompatible with `--disable-internet`. |
Expand All @@ -59,13 +60,19 @@ Used when running agents via `sforge run`. These are injected as container envir
Directly injecting `SFORGE_HTTP_PROXY` / `SFORGE_HTTPS_PROXY` into the agent container gives that container proxy-mediated network access. This is useful only for exceptional run-time dependency downloads and is **not recommended** for LLM API access.

It is also incompatible with network isolation (`--disable-internet`). If the agent needs to call an LLM API through a corporate proxy while network isolation is enabled, run `sforge proxy` on the host and point `SFORGE_AGENT_API_BASE_URL` to `http://host.docker.internal:<port>` instead.

For E2B, generic host `HTTP_PROXY` / `HTTPS_PROXY` variables are not forwarded
to Work Sandboxes. Set the `SFORGE_` variants only when the proxy is reachable
from E2B; isolated E2B runs always remove proxy variables and use the native
host allowlist directly.
:::

## Judge Variables

| Variable | Purpose |
|----------|---------|
| `SFORGE_JUDGE_EXTRA_ENV` | Extra env vars for judge containers, format: `"KEY1=VAL1,KEY2=VAL2"` |
| `SFORGE_ADMIN_SECRET` | Trusted-host credential for Judge session registration and auto-eval. If omitted, SForge creates a mode-`0600` secret under `SFORGE_LOG_DIR`. Never expose it to Work containers. |

## Resource Limit Variables

Expand All @@ -80,11 +87,14 @@ Judge CPU/memory limits are currently set through CLI flags (`--judge-cpu-limit`

| Variable | Default | Purpose |
|----------|---------|---------|
| `SFORGE_BACKEND` | `docker` | Container backend: `docker` or `k8s` |
| `SFORGE_BACKEND` | `docker` | Container backend: `docker`, `k8s`, or official `e2b` |
| `SFORGE_K8S_NAMESPACE` | `default` | Kubernetes namespace |
| `SFORGE_K8S_IMAGE_REGISTRY` | --- | Container registry for K8s image pulls |
| `SFORGE_K8S_KUBECONFIG` | --- | Path to kubeconfig file |
| `SFORGE_K8S_NODE_SELECTOR` | --- | Node selector for K8s pods, format: `"key1=val1,key2=val2"` |
| `SFORGE_E2B_TEMPLATE_MAP` | --- | Explicit E2B image-key to Template mapping, overriding the derived name. Inline `"image_key=template,..."` pairs, or a path ending in `.json` holding a JSON object |
| `SFORGE_E2B_TEMPLATE_NAMESPACE` | --- | Namespace for published E2B Templates (for example, `edgebench`) |
| `SFORGE_E2B_SANDBOX_TTL` | `3600` | E2B Sandbox TTL in seconds |

## Path Variables

Expand Down
5 changes: 3 additions & 2 deletions docs/en/configuration/experiments.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,16 +54,17 @@ Default settings applied to all tasks unless overridden per-task.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `agent` | `string` | --- | Agent name (`claude-code`, `codex`) |
| `agent` | `string` | --- | Agent name (`claude-code`, `codex`, `opencode`, or a pinned variant such as `claude-code-2.1.214`) |
| `model` | `string` | --- | Model override (overrides `model.model` for specific tasks if set per-task) |
| `effort` | `string` | --- | Reasoning effort (`low`, `medium`, `high`, or `max`); mapped to the selected Agent when supported |
| `timeout` | `int` | --- | Agent timeout in seconds |
| `eval_interval` | `int` | --- | Auto-eval interval in seconds |
| `disable_stop_hook` | `bool` | `false` | Disable the agent stop hook |
| `disable_auto_eval` | `bool` | `false` | Disable background auto-evaluation |
| `disable_auto_resume` | `bool` | `false` | Disable auto-resume on abnormal agent exit |
| `internet` | `bool` | --- | Enable/disable internet access |
| `extra_env` | `dict` | --- | Extra environment variables injected into the agent container |
| `backend` | `string` | --- | Container backend (`docker` or `k8s`) |
| `backend` | `string` | --- | Container backend (`docker`, `k8s`, or `e2b`) |
| `judge_url` | `string` | --- | Judge server URL override |
| `max_submissions` | `int` | --- | Maximum number of agent submissions per run |
| `submission_cooldown` | `int` | --- | Minimum seconds between agent submissions |
Expand Down
Loading