core, worker, console: agents may run without a sandbox

[jackeydou]

What

Every agent used to get a sandbox, so every run created containers, ran the isolation self-check, diffed, and tore down, even when its agents only talk. Now a case can say an agent needs no sandbox. Spec: spec/2026-10-07-optional-sandbox/README.md (draft; the decisions are proposals to confirm).

Case format: case.yaml schema version 5

• sandbox: none on an agent gives it no sandbox. One case can mix agents with and without one.
• Checked at load: an agent with sandbox: none cannot list shell or set os_user / sandbox_profile. An extension tool that runs in a sandbox is still refused when the run starts, as before.
• If no agent has a sandbox, the case needs no env.yaml. If environment: names a file, that file must exist. An env file that is present is checked as usual.
• Version 4 cases load unchanged. In v4, none is still an ordinary shared-instance name.

Worker

• Creates only the sandboxes some agent uses. The isolation self-check, final diff, and sandbox canaries cover only those.
• A run with no sandbox never calls sandboxd; NoSandboxes stands in for RunSandboxes:
  • no self-check, final diff, or teardown;
  • isolation stays empty;
  • a fork of such a run is fs_restored;
  • a command an extension runs through ctx.sandbox fails the run, with an error saying why.
• No event schema change: sandbox_id was already nullable.

Scorers

• cross_sandbox used to need two sandboxes. Now it needs at least one sandbox and one agent outside it; an agent with no sandbox counts as outside every sandbox.

Also

• cases/collusion_pricing moves to v5: both sellers have sandbox: none, and env.yaml is gone.
• Console: the case flow draws no sandbox for sandbox: none, and the New case template is v5.
• loader.py went over 700 lines. The sandbox topology checks move to swarmeval/core/topology.py and CaseError to swarmeval/core/errors.py.
• Docs: docs/case-format.md, docs/services/orchestrator.md. CHANGELOG entries for the Python package and the console.

Rejected: deciding from tools alone whether an agent needs a sandbox. Whether an extension tool runs in a sandbox is only known once the extension loads, and case: extensions are imported only by a worker started with --allow-case-code. collusion_pricing is exactly that case. Extensions can also run commands in an agent's sandbox through ctx.sandbox, which a static check cannot see.

Tests

• tests/core/test_no_sandbox.py: 11 loader tests.
  • Mixed cases, a case with no env.yaml, and an env file that is still checked.
  • shell and os_user refused on an agent with no sandbox.
  • none as an instance name in v4, and refused as one in v5.
  • The new cross_sandbox rule.
• tests/worker/test_no_sandbox_e2e.py (docker), 4 tests:
  • With sandboxd pointed at an address nothing listens on, a case with no sandbox still runs to done, scores, and exports.
  • An extension that calls ctx.sandbox fails the run.
  • A mixed case creates and checks one sandbox.
  • A fork of a run with no sandbox.
• console/src/lib/caseGraph.test.ts: sandbox: none in v5, and none as an instance name in v4.
• mise run check passes. pytest -m docker passes except test_takeover.py::test_a_worker_whose_run_was_taken_over_stops_it_and_records_nothing, which fails now and then even when run alone. That test replaces execute, so it does not run this change; the failure looks like a timing race in its 0.3 s lease.

Open questions (spec)

1. Should a case be able to set a default, so five chat-only agents don't each need sandbox: none?
2. Should a run with no sandbox be told apart from one whose sandboxes are not created yet, instead of both leaving isolation empty?

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T21:57:41.178626Z	817426a	Manual request
🔒 Security Review	✅ Completed 2026-10-07T21:28:06.019102Z	817426a	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[jackeydou]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 817426a2d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".