RFC: Environment Services — brokered, budgeted access to external services (training, GPU jobs, evaluators, simulators) #1382

Problem Statement

More and more environments give the agent real services, not just in-env tools. Examples: post-training tasks (PostTrain Arena #898, PostTrainBench, RSI-Index), kernel and compiler hill-climbing, Blender and Isaac simulation, and queued science solvers. #898 explicitly leaves "hosted-verifier, GPU/TPU-only, multi-agent, or simulated-user packages" out of scope "without a defined runner and policy". This RFC proposes that policy.

Today each env answers the same five questions ad hoc:

1. Who holds the credential? Raw keys reach the agent environment. Examples: Harbor environment.env, Prime sandbox secrets, and OpenHands exporting secrets into the agent's shell.

2. Who pays? Existing caps cover LLM tokens only and are checked after spending, so they overshoot (OpenHands max_budget_per_run, LiteLLM MCP cost, RSIGym). There's no budget per rollout, per group or per run.

3. What happens to a job that outlives the episode? There's no job handle, no cancel on reset and no orphan handling. kernrl runs its GPU evaluator inside step.

4. Can the agent reach the grader? Several frameworks grade inside the agent's sandbox:

   • ROLL and rLLM upload tests into the agent's sandbox after the episode.
   • In AReaL, a rollout's own key can set its reward.

   METR saw reward hacking in 30.4% of RE-Bench runs vs 0.7% on HCAST. The same concern drives Verify sandbox environments outside the agent's sandbox #1232 and Measure OpenEnv's reward-hack defenses and prototype a Karotte bridge #1357.

5. What was called, and what did it produce? No framework links service calls to trajectory steps.

Proposed Solution

Add an Environment Services layer on openenvd (RFC 009, #1053). It has three rules: the agent never holds a real credential, never reaches the grader, and never spends without a reservation.

v1 scope: two profiles, batch job and evaluator, plus the per-rollout budget. That covers post-training and kernel optimisation. The other profiles are future work (see the appendix) until an env needs them.

agent sandbox ──MCP svc.* / jobs.* / artifacts.*──▶ openenvd (per env)
 (run token only)                                     ledger · audit · artifacts · token exchange
                                                          │ audience-bound token (RFC 8693)
                                                          ▼
                                    shared brokers: jobs (HF Jobs) · eval
grader runtime ◀── artifact handles + score endpoints ────┘

1. Service manifest. An env declares its services by profile:

services:
  - {name: jobs, profile: batch_job, surface: agent, flavors: [a10g-small], max_timeout: 2h, meter: [usd, gpu_s]}
  - {name: kernel_eval, profile: evaluator, surface: {agent: dev_evaluate, grader: score},
     hardware: {class: h100, exclusive: true}}
budget: {usd: 50, gpu_s: 7200, jobs: 10, scope: rollout}

2. Service profiles (v1). Each profile adds one requirement to the contract. Domains combine profiles.

Post-training uses batch job plus a grader-only evaluator: in v1 the agent trains by submitting scripts to HF Jobs. Kernel optimisation uses evaluator and batch job.

3. Budget ledger. It reserves before a call and settles after. A job's reservation is the hardware rate × the timeout. Budgets are scoped per rollout and allocated by the orchestrator, which keeps GRPO groups comparable. Exclusive GPUs are leases, separate from money. The agent sees its remaining budget as a fact, never as reward. Running out ends the episode with a typed reason.

Spend is published under a stable key. It is for accounting and for planning budgets and concurrency, never a sampling weight: dividing a task's sampling weight by its cost would starve expensive, hard tasks. RFC 013's sampler (#1256) never reads the env; the trainer reads this entry from the done observation and can report one value per rollout as costs to update(task, rewards, costs):

obs.metadata["service_spend"] = {
    "version": 1,
    "settled": {"usd": 0.62, "gpu_s": 410.0, "calls": 37},  # cumulative for the episode
    "reserved": {},   # open reservations; empty once every job has settled
    "final": True,    # false if a job ended orphaned or its cancel was unconfirmed
}

Units come from a fixed vocabulary that matches the manifest's meter field: usd, gpu_s, cpu_s, tokens, calls and jobs, always in base units. The manifest's meter list tells trainers which cost units to report (#1256, open question 6); spend is never a sampling weight.

4. Jobs. These are tools that return a handle: jobs.submit, status, wait and cancel. jobs.submit takes an idempotency key, so a retried submit never pays twice. States are queued → running → succeeded | failed | cancelled and cancelling → orphaned, each with a reason. done, reset and close cancel the episode's jobs, and an unconfirmed cancel blocks the next episode. Support for the MCP Tasks extension (SEP-2663) is opt-in, once /mcp speaks 2026-07-28 (decision 4).

Prior art: OpenMLE Sandbox (OpenRSI) already runs this job shape for MLE agents:

• submit, status, logs and cancel endpoints;
• states queued/running/completed/failed/cancelled, with typed result reasons;
• idempotency keys;
• CPU/GPU priority queues;
• a router across several controllers.

It is also a backend a broker could front, but it scores on the same worker that ran the candidate, from a shared mount where the private answers are readable. Its README says this is "not a security boundary". The broker adds the missing layers: scoring in a fresh container, network policy, run tokens and the ledger.

5. Credentials. We follow MCP 2026-07-28 auth: audience-bound tokens and no passthrough.

• The daemon injects credentials, using the OpenShell pattern, which is RFC 009's credential_isolation future work.
• Brokers get short-lived tokens through RFC 8693 token exchange, with a hard cap on the provider side as a backstop (an HF Resource Group spend limit or a LiteLLM key budget).
• The HF Jobs broker holds a fine-grained token with only job.write ("Start and manage Jobs"). It calls run_job/run_uv_job with explicit flavor, namespace and secrets, and never shells out to hf jobs uv run, whose script header can pull in the caller's HF_TOKEN and set the namespace.
• The broker never passes its own token into a job: a job.write token inside a job can start further jobs that skip the broker's admission and budget checks. Checkpoint pushes get a separate repo-scoped token.
• HF tokens carry no spend cap, so broker jobs are billed to a Resource Group with a Jobs spend limit (Enterprise) as the backstop.

6. Artifacts and audit. Artifacts stay in the daemon. The agent gets openenv-artifact://episode/job/name resource_links that carry a sha256 and size. Graders receive them through VerifyResult.artifacts. Checkpoints are signed with OpenSSF model signing v1.0. Every service call emits an audit record keyed to (episode, step, tool_call_id) as an OTel GenAI/MCP span, pinned to a specific commit.

7. Evaluator contract. The evaluator runs a cascade: admit (hash, dedupe, static scan), then validate, verify, measure on exclusive hardware, and grade.

• dev_evaluate is rate-limited and budgeted. It returns at most 5 metrics, size-capped feedback, status, ci and evaluator_version.
• score is grader-only. It uses hidden inputs, re-measures, and audits the trace.

Results are cached by candidate hash, evaluator version, input set, hardware class and toolchain. Each evaluator ships a red-team record; #1357's attack panel is a natural seed. Timing needs an exclusive lease: shared GPUs inflate kernel timings 1.5–3.2× (KREX), and MIG/MPS don't qualify.

Not allowed:

• raw provider keys in the sandbox;
• grading in the agent's sandbox;
• reward writes the agent can reach;
• passing the broker's own token into jobs;
• judges reading unescaped trajectories;
• building on MCP sampling or x402 (MCP has no official cost or payment extension).

Alignment with Principles

Principles Addressed

• Minimize lifecycle deltas. Training and production eval use the same manifest and brokers. Only the budget values change.
• Minimize human-agent divergence. Agents use the same scoped, budgeted tools a human operator would.
• Be hands-on. This starts as an env-only prototype; see the implementation sketch.
• Design for LLMs. Quota, denial and budget states are typed, and jobs return handles instead of blocking calls.

It keeps the key invariants:

• Agents cannot reset. Approval, cancel and budget top-up are orchestrator-only.
• Rewards inside environment. score runs in the env's grader zone.
• Dual API boundary. svc.*, jobs.* and artifacts.* are agent tools, while the ledger and lifecycle stay on the infra side.

Trade-offs

Impact

Files/Systems Affected

• src/openenv/core/env_server/mcp_types.py: RESERVED_TOOL_NAMES is exact-match only. It needs a prefix rule for svc.*, jobs.* and artifacts.*.
• src/openenv/core/env_server/http_server.py: _cleanup_session_resources swallows errors, so job-cancel failures never surface.
• src/openenv/core/env_server/types.py: the idle reaper should be on whenever services are declared.
• src/openenv/core/harness/rollout.py: VerifyResult.artifacts should carry artifact handles.
• src/openenv/core/env_server/http_server.py /mcp and pyproject.toml: speak MCP 2026-07-28 while still serving 2025 clients, which means bumping fastmcp to 4 (decision 4; later, via the RFC 003 amendment).
• New client-safe models (manifest, ledger, job, artifact) that live outside server/.
• openenvd (feat(openenvd): contract and pluggable enforcement backends (RFC 009, 1/2) #1354, feat(openenvd): daemon, surfaces and declared processes (RFC 009, 2/2) #1355): broker token exchange and the grader-side score service. A jobs observer stream needs its own justification, because RFC 009 rejected a resource stream.

Related RFCs

• RFC 009 openenvd (RFC: openenvd — Privileged Environment Sidecar with Policy-Scoped Surfaces #1053, feat(openenvd): contract and pluggable enforcement backends (RFC 009, 1/2) #1354, feat(openenvd): daemon, surfaces and declared processes (RFC 009, 2/2) #1355): the base for this proposal.
• RFC 013 task sampling (RFC 013: Task sampling and in-training evaluation #1256): the trainer reports service_spend as costs, for accounting and budgets, not as a sampling weight.
• RFC 003: a later, backward-compatible amendment moves /mcp to MCP 2026-07-28, which enables the Tasks extension.
• RFC 004: score feeds rubrics.
• RFC 005 / 006: LLM proxying, i.e. the future model-call profile.
• RFC 008: contract checks for declared services, plus Measure OpenEnv's reward-hack defenses and prototype a Karotte bridge #1357's adversarial panel.
• RFC 012: separate-verifier precedent.
• Issues: Spec-agnostic task validation: openenv validate-task for external task.md packages (PostTrain Arena / BenchFlow) #898, Measure OpenEnv's reward-hack defenses and prototype a Karotte bridge #1357, Verify sandbox environments outside the agent's sandbox #1232, tbench2_env: expose the task's execution budgets to the client #1026 (exposing execution budgets), RFC: Support multi-threading (e.g. subagents, multi-agents) #779 and RFC: OpenShell as an external ContainerProvider backend #1299.

Implementation Sketch

1. First PR: RFC doc plus env-only prototype, no core change. It adds rfcs/014-environment-services.md and a posttrain_env whose server holds an HF Jobs broker and an evaluator, the way kernrl holds its evaluator. The env runs PostTrain-style T0 tasks (Spec-agnostic task validation: openenv validate-task for external task.md packages (PostTrain Arena / BenchFlow) #898).
2. Core types. Manifest, ledger, job and artifact models; the reserved-name prefix rule; cleanup semantics.
3. openenvd integration. Credential custody, token exchange, the grader zone, and MCP Tasks.

job = await env.call_tool("jobs.submit", script="train.py", flavor="a10g-small", timeout="1h")
# -> {"job_id": "...", "reserved": {"usd": 1.05}, "budget_remaining": {"usd": 48.95}}
res = await env.call_tool("jobs.wait", job_id=job["job_id"], timeout_s=600)
# -> {"state": "succeeded", "artifacts": [{"uri": "openenv-artifact://ep/j1/adapter", "sha256": "..."}],
#     "settled": {"usd": 0.62}}

Decisions

1. Long-running services and GRPO: a service runs live at train time only if it finishes within the trainer's staleness window. For example, TRL's experimental AsyncGRPOTrainer drops samples more than max_staleness weight updates old (default 4). Slower services declare a stand-in in the manifest, and the real service runs only at eval.

2. HF Spaces: no services on Spaces for now. A Space gives no isolation guarantee, so it must not advertise services that hold credentials.

3. Shared brokers are operator-run services outside the env. They have the same standing as RFC 009's backend gateways, such as the OpenShell gateway. RFC 009's "one daemon per env" still holds: the broker serves many tenants, the daemon doesn't. This follows existing practice:

   • RSIGym puts training, inference, eval and sandbox services behind one auth server with per-run budgeted keys.
   • LiteLLM gives each team or run a virtual key with its own budget.
   • OpenMLE Sandbox routes jobs to shared controllers by API key.

   Concretely:

   • Trust: the orchestrator's run config lists each broker's URL and expected TLS identity. That registration is the trust anchor; hardware attestation is out of scope.
   • Tokens: openenvd holds the broker credential as a backend credential and exchanges it for per-rollout tokens bound to one broker (RFC 8707). Brokers act as MCP resource servers (RFC 9728 metadata).
   • Budget: each broker enforces a hard cap per token and reports settled amounts. The daemon's per-rollout ledger stays authoritative.

4. Transport: target MCP 2026-07-28. The revision removes protocol sessions. Servers that need cross-call state use server-minted handles passed as arguments (SEP-2567).

   • No conflict with today's /mcp. It is a custom JSON-RPC handler that serves only tools/list and tools/call, with no initialize. openenv/session/create already returns a server-minted id that clients pass explicitly.
   • What the upgrade adds: server/discover, the per-request _meta protocol version, resultType, the Mcp-Method/Mcp-Name headers, and cacheable tools/list results.
   • Libraries: MCP Python SDK v2 and FastMCP 4 serve 2025- and 2026-era clients from one server, and FastMCP 4 ships the Tasks extension (fastmcp-tasks). OpenEnv pins fastmcp>=3,<4, so this includes a major bump.
   • Jobs: they stay as tools that return handles, which is the revision's own pattern for cross-call state. MCP Tasks is opt-in on top.
   • Timing: the upgrade comes later, as a backward-compatible RFC 003 amendment, so existing envs need no changes. It isn't part of this RFC's first PR, and v1 jobs work without it.

Appendix: future service profiles

Each is added when an env needs it.