Repository navigation
feat(openenv): serve every bundle task as an OpenEnv env - #230
Conversation
build_app(bundle) returns the OpenEnv server for a bundle's tasks: each session plays one episode through the Episode API, with the env's MCP tools plus an end_turn tool, and the task's verifier score as the reward. Bundles are prepared as agent-env run prepares them, without the agent a client plays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onment's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e's loop The playground calls the sync step in a worker thread; the episode's tool sessions belong to the loop its reset ran on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e app's lifespan OpenEnv's HTTP MCP sessions and web playground can call one env concurrently; a step during a reset or end_turn dereferenced the turn's unopened tools, and a close racing a turn's opening left its tool sessions open. Reset, step and close now take turns. Routing is on from each startup to its shutdown, so a failed create_app no longer leaves it on and a second run of the app has it. Reset defaults to the first split's first task as OpenEnv's importers do, with all listed first; an eval named all is refused; a Path is always a folder; a dotted bundle name is served dashed, since OpenEnv's web interface reads the env name as a module path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The extra installed fastapi and uvicorn, which core now installs, so agent-env up no longer checks for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e limit allows Each entry's lock holds a file open, so agent-env run crashed with EMFILE on a bundle of over about 1000 entries on Linux, or 250 on macOS, such as a generated task pack. The soft limit is raised toward the hard one to fit them, as Go's runtime does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
…set skips an empty split A call cancelled while end_turn or reset waited for the next turn left a live episode with no open tools, so the next step dereferenced None; it now ends the episode as an error observation. A bare reset plays the first task served even when the first split has none, and the validator test asserts that OpenEnv's validation passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alizes it jaraco.classes, which openenv's keyring needs, ships a wheel whose filename keeps the dot, so its index page was jaraco.classes and uv found no jaraco-classes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_bundle and EpisodeSpec.start built the same three-key user_overrides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lifecycle helpers say what they do Startup turns reward= into one function per task (a user's function, or the score at a step), so the episode applies it in one place. _end/_close/_ended/_ended_episode/_fail become _abandon/_abandon_when_free/_error_observation/_why_stopped/_end_with_error, and done lives only in the episode's State. A missing score now reads as the reward's own error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's Episode API now plays an outside agent through a proxy (#270), so the OpenEnv adapter rides on it: - an env tool's failure reaches the adapter as the proxy's error result, an execution error with the proxy's text; a timeout or a lost connection between the adapter and the proxy is still its own error type; - EpisodeSpec.prompts reads each outside turn's system prompt as its agent gets it, from its deploy or the last prompt that set one; - a cancelled episode is recorded in its task's run scope, as before. Bundle preparation keeps main's --agent, --user-model and --judge-model overrides beside the played-agent skip, and one helper builds the sandbox overrides for agent-env run --sandbox and EpisodeSpec.start(sandbox=). The explorer extra stays retired (README, AGENTS.md, THIRD_PARTY_NOTICES.md), and the notices list the merged lock's runtime dependencies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… proxy reports it Since the episode plays its agent through a proxy, an env tool call that can't reach its env fails in the proxy, which answers with an error result; the adapter serves it as an execution error with the proxy's text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ce 0.10.0 OpenEnv 0.10.0 came out on 2026-10-22, past the range's top, so the range and the lock move together to the two minors OpenEnv keeps working, and the notices list 0.10.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ves that teardown tracked The session's abandon awaited its tracked teardown directly, so a reset cancelled during that wait cancelled the tracked task: it left Served.ending while the episode's teardown ran on, so the server's shutdown no longer waited for it, and the next reset could deploy before it finished. The wait is shielded, and the session keeps the teardown until it's done, so the next reset waits for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…played through bash Since #270 the Episode API serves a task whose agent works in its sandbox, so build_app does too. The test serves one that loads a file into its agent and checks what the agent left: through OpenEnv's own client, bash comes first in the tools and runs as the agent in its folder, the check scores what it left, and nothing is left running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Every task in a bundle is now an OpenEnv env.
agent_env.openenv.build_app(bundle)returns the OpenEnv server for a bundle's tasks. Any OpenEnv client (OpenEnv's ownGenericEnvClientandMCPToolClient, verifiers, a TRL wrapper) resets an episode, acts through the env's MCP tools and gets the task's verifier score as its reward, with no per-task code.openenv>=0.9,<0.11becomes a hard dependency, the waymcpanda2a-sdkare. That range is OpenEnv's two-minor deprecation window, moved up when OpenEnv 0.10.0 came out on 2026-10-22. Onlyagent_env/openenv/imports it, so no other command pays the cost of loading it.Two fixes outside
openenv/came out of testing on PortSim's packs:agent-env runcrashes withEMFILEon a bundle of more than about 1,000 entries (Linux) or 250 (macOS), such as PortSim's generated 1,050-task pack.explorerextra is retired. OpenEnv's fastapi makes it install nothing core lacks.Merged onto main after #270 (2026-10-26)
Main's Episode API now plays an outside agent through a ProxyAgent (#270): env tools are called by a proxy, served in this process for a task whose agent works in no sandbox. The adapter rides on it unchanged in shape:
execution_errorwith the proxy's text (error: the tool call timed out after 600s,error: ConnectError: …). A timeout or lost connection between the adapter and the proxy is stilltimeoutortransport_error, and an unknown tooltool_not_found.EpisodeSpec.promptsreads each outside turn's system prompt as its agent gets it: from its deploy, or the last prompt that set one.@localtask's lands where its instance is.--agent,--user-modeland--judge-modeloverrides beside the played-agent skip, and one helper (sandbox_overrides) builds the sandbox overrides foragent-env run --sandboxandEpisodeSpec.start(sandbox=).build_appserves those tasks too: a proxy is deployed in the agent's place, andbash, first in the tools, runs as the agent in its folder. A new integration test covers it, and the chaos below runs it.How a task maps onto OpenEnv
/splits,/tasks,/task)all, then a split per eval, or a mapping's keys. Each task listing carries{id, name, index, prompt, system_prompt, agent}, read without deploying.reset(split, index)EpisodeSpec) and returns once its first turn is open (deploy and setup done). The observation carries the prompt and system prompt. With neither argument, it plays the first task served, as OpenEnv's ORS and verifiers importers do.step(ListToolsAction / CallToolAction)MCPEnvironment's does.end_turn(response)(an added tool)done=Trueand every score inmetadata["scores"].SUPPORTS_CONCURRENT_SESSIONS, andmax_concurrent_envs(default 8) caps them.The choices follow OpenEnv's own envs:
jupyter_envandterminus_envcreate their sandbox in reset.close().end_turnin flight.done, no reward, andmetadata["status"] == "error", asterminus_envreports a sandbox that can't start.ToolError, and the episode goes on.resetorend_turnwaited for the turn to open ends the episode at the next step, in the same error shape.reset,step,state,closeorend_turnare refused, asMCPEnvironmentrefuses them.reward=<step id>or a function of the scores, checked over every task at startup.agent-env rundoes before its first task, except that the agent the client plays is never deployed, so it needn't exist.@localids resolve while it serves.Changes outside
agent_env/openenv/agent_env.bundle.prepare_bundle(..., played=False):run_bundle's work before its first task, moved out unchanged, soagent-env runbehaves exactly as before.played=Truedrops the references of each task's only prompted agent'sdeploy_agentsteps, keeping its envs, and preflight skips those deploys.locate_bundlemoves fromcli/run.pyintobundle/installed.py.EpisodeSpec.start(sandbox=)setsuser_overridesthe wayrun_bundledoes, through a sharedtask_step.context.sandbox_overrides.EpisodeSpec.promptsgives each outside turn's prompt and system prompt before anything runs.Episode.finishedis added.cancelledmark is now written in the task's run scope, asrun_bundle's_mark_cancelleddoes. Before, an@localtask's abandoned episode stayedrunningin the local store, because the mark went to the configured store.holding_locksraises the soft open-file limit to fit its locks, as Go's runtime does.agent-env runwithOSError: [Errno 24] Too many open files.agent-env runon PortSim's generateddock-v1-train(1,050 tasks, Linux default limit 1024) anddock-v1-train-standard(266 tasks at macOS's default of 256).explorerextra is gone.agent-env updrops its check for them, and the README and AGENTS.md drop the extra.pip install 'agentenv-framework[explorer]'still works, with pip's warning that the extra doesn't exist.jaraco.classes(needed through openenv's keyring) ships a wheel whose filename keeps the dot, so uv found nojaraco-classes.THIRD_PARTY_NOTICES.md: lists the runtime dependenciesuv.lockresolves for core and its documented extras, withoutexplorer: main's entries, the new packages, and the versions that moved (openenv0.10.0,hf-xet,huggingface_hub,pandas,brotli), andPyJWTandtensorlake, which already disagreed with main's lock.Notes for review
agent-env run's own error. Only the tasks the Episode API refuses are left out (and logged), since planning attributes problems to the bundle, not to individual tasks. So a task the Episode API would refuse (say, one prompting two agents) must still passagent-env run's checks: its agents must exist.POST /resetdeploys an episode and tears it down at once, as it does for every OpenEnv env that sets up in reset. Real episodes go over/wssessions, whichmax_concurrent_envscaps..is served with-(dock.v1at/dock-v1/...). OpenEnv's web interface reads the env name as a module path, so a dotted name failedcreate_appunderENABLE_WEB_INTERFACE, whichopenenv pushsets in a Space.allis refused, since it would hide the split that holds every task./mcpendpoint answers a plainEnvironment'stools/callwith the whole observation, not the tool's data. That's true for every plain-Environmentenv, including the ORS importer's. SoMCPToolClient.call_toolreturns the observation for these envs.Verification
Tests: unit tests in
tst/unit/openenv/(25), plus extensions to the Episode API, bundle, ledger and installed-bundle tests. Integration tests intst/integration/openenv/(6) drivebuild_appon local Docker through OpenEnv's ownGenericEnvClientandMCPToolClient; they passed twice on a Linux box.After merging main (head
6d7fe038, macOS, local Docker):tst/integration/openenv/: 5 of 6 pass, twice, on OpenEnv 0.10.0. The sixth builds the echo agent for linux/amd64, which this Mac's Docker can't without buildx; CI runs it.After the merge, on a Linux EC2 box (c6i.8xlarge, local Docker, the head with OpenEnv 0.10.0 and agentenv-portsim 0.5.5):
build_app({"train": dock-v1-train, "eval": dock-v1-eval})serves 1,050 + 50 tasks in 7.3 s cold and 2.0 s warm.GenericEnvClient, the optimal, naive and empty plans of the first standard train week score 1.0, 0.238456 and 0.0, exactly what the same plays score through the Episode API in process.execution_error(error: ConnectError: …);end_turnis then the error observationmax_concurrent_envs=3CAPACITY_REACHEDverify_sandbox). Every scenario passes:bashruns asagentin/appand reads the loaded file; every record completedexecution_error,bashstill works, and the task still scores 1.0bashis atransport_error;end_turnis the error observation onceprompt_agentgives up on the agent (about 60 s)tst/integration/openenv/passes 7 of 7 on the box, andserver_env_local_test.py(the Episode API's container tests) passes too.The reset and close chaos again, on the final head
11695a4c(a fresh box, after the cancelled-reset fix). Every scenario passes, two of them new:PortSim proof, before the merge (agentenv-portsim 0.5.5, a Linux EC2 box, local Docker):
build_app({"train": dock-v1-train, "eval": dock-v1-eval})serves 1,050 + 50 tasks after 11 s cold, 5 s warm.GenericEnvClient, score the same as the same episode run through the Episode API in process: 1.0, 0.200181, 0.0.Chaos, before the merge (
dock-v1-train-standard, 266 tasks, c6i.8xlarge). All 21 checks pass:cancelled, the lastcompleted. Never more than one deployment.transport_errorToolError(since the merge, anexecution_error: the proxy makes the call).end_turnis then the error observation (done, no reward,status: error).cancelledand torn down; stopped in 0.8 s.max_concurrent_envs=3CAPACITY_REACHED.OpenEnv's own tools against the PortSim server:
openenv validate --url ... --level runtime: PASS, 6 of 6 criteria.ENABLE_WEB_INTERFACE=true): the page serves, and an eval task resets with its four tools;end_turnscores it.Regression (main
705a178and this branch on the same Linux box):make unit-testmake int-test-fastmake int-test-slowrun_docker_container_test.py::test_a_context_zip_from_a_non_s3_store_unpacks_in_a_real_local_sandbox.local_sandbox_test.py::test_teardown_stops_what_a_command_left_running_and_only_thatfailed once in this PR's CI. It failed the same way on another PR's CI today, and passed in both full runs on the box.Follow-ups (separate PRs)
explorerextra once this is released, and a page on serving tasks as OpenEnv envs is added.agent-env hf publish, with envs deployed to a remote sandbox.AutoEnv;🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge, with no new actionable issues found.Summary
Adds
build_app(bundle)to serve bundle tasks through OpenEnv clients, with task splits, tools, rewards, and session cleanup.agent-env upno longer needs an explorer extra.Diagram
Reviews (7) · Last reviewed commit: "test(openenv): a task whose agent works ..." · Reviewed by Greptile