From 7ebdea30cfce8886713b1c6bf5ce49a29d0b5c3c Mon Sep 17 00:00:00 2001 From: swappy <59965507+rycerzes@users.noreply.github.com> Date: Fri, 16 Oct 2026 23:23:42 +0530 Subject: [PATCH 01/11] Add RFC 014: Environment Services RFC-only extraction from original commit 637310bdec712f5bc47be7e9471d2411b58c26eb in openenv/OpenEnv#1500. --- rfcs/014-environment-services.md | 207 +++++++++++++++++++++++++++++++ rfcs/README.md | 3 + 2 files changed, 210 insertions(+) create mode 100644 rfcs/014-environment-services.md diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md new file mode 100644 index 0000000000..a1898fb728 --- /dev/null +++ b/rfcs/014-environment-services.md @@ -0,0 +1,207 @@ +# RFC: Environment Services + +**Status**: In Review +**Created**: 2026-10-16 +**Authors**: @rycerzes +**RFC ID**: 014 + +Discussion: [#1382](https://github.com/huggingface/OpenEnv/issues/1382). Prototype: [`envs/posttrain_env`](../envs/posttrain_env/). + +## Summary + +More environments give the agent real services instead of in-env tools: post-training tasks that train models on rented GPUs, hill-climbing tasks that call an evaluator on dedicated hardware, simulators and render farms. Today each environment decides alone who holds the credentials, who pays, what happens to a job that outlives the episode and whether the agent can reach the grader. + +This RFC defines **Environment Services**: a brokered layer between the agent and external services, with three rules. The agent never holds a real credential. The agent never reaches the grader. Nothing is spent without a reservation. v1 covers two service profiles, **batch job** and **evaluator**, and a **per-rollout budget**, which is enough for post-training and kernel optimisation. `envs/posttrain_env` implements v1 on HF Jobs without core changes. + +## Motivation + +### Problem Statement + +Environments answer five questions ad hoc: + +1. **Who holds the credential?** Raw keys reach the agent environment. Examples: Harbor `environment.env`, Prime sandbox `secrets`, and OpenHands exporting secrets into the agent's shell. +2. **Who pays?** Existing caps cover LLM tokens only and are checked after spending, so they overshoot (OpenHands `max_budget_per_run`, LiteLLM MCP cost, RSIGym). Nothing defines a budget per rollout, per group or per run. +3. **What happens to a job that outlives the episode?** There is no job handle, no cancel on reset and no orphan handling. `kernrl` runs its GPU evaluator inside `step`. +4. **Can the agent reach the grader?** [ROLL](https://github.com/alibaba/ROLL/blob/581046abce143e643221f8e292895c46a3b20112/roll/pipeline/agentic/env/rock/sanbox_manager.py#L1230-L1332) and [rLLM](https://github.com/rllm-org/rllm/blob/4e828c81680e5a85b638ceedeaf09d1e90109ca8/rllm/eval/script_evaluator.py#L73-L94) upload tests into the agent's sandbox after the episode; in AReaL a rollout's own key can set its reward. METR measured [reward hacking in 30.4% of RE-Bench runs vs 0.7% on HCAST](https://metr.org/blog/2025-06-05-recent-reward-hacking). #1232 and #1357 raise the same concern for OpenEnv. +5. **What was called, and what did it produce?** No framework links service calls to trajectory steps. + +### Goals + +1. A service contract an environment declares, instead of per-env plumbing. +2. Credentials that stay with a broker; the agent and its jobs hold none. +3. A per-rollout budget that reserves before spending and settles after, published in a stable form for cost-aware samplers (RFC 013). +4. Jobs with handles, cancellation at episode end and a defined orphan state. +5. Evaluation split into an agent-callable dev endpoint and a grader-only score. + +### Non-Goals + +- Hosting services. Brokers wrap existing ones (HF Jobs first). +- Profiles beyond batch job and evaluator in v1 (see [Future Work](#future-work)). +- Changing `/mcp` in this RFC. The move to MCP 2026-07-28 is a later RFC 003 amendment (decision 4). + +## Design + +### Architecture Overview + +``` +agent sandbox ──MCP tools──▶ environment / openenvd (per env) + (no credentials) ledger · artifacts · audit · token exchange + │ broker credential, audience-bound + ▼ + shared brokers: jobs (HF Jobs) · eval +grader ◀── artifact handles + score jobs ──┘ +``` + +Without openenvd (RFC 009), the environment server is the broker, as in `posttrain_env`. With openenvd, the daemon holds the broker credential as a backend credential and the environment only sees run tokens. + +### Core Abstractions + +**Service manifest.** An environment declares its services by profile: + +```yaml +services: + - {name: jobs, profile: batch_job, surface: agent, flavors: [a10g-small], max_timeout: 60m, meter: [usd, gpu_s]} + - {name: kernel_eval, profile: evaluator, surface: {agent: dev_evaluate, grader: score}, + hardware: {class: h100, exclusive: true}} +budget: {usd: 3.0, scope: rollout} +``` + +`posttrain_env` keeps this in its task definitions (`server/tasks.py`); the manifest becomes a core type in step 2 of the plan below. + +**Service profiles (v1).** + +| Profile | Examples | Adds to the contract | +|---|---|---| +| batch job | HF Jobs; later Modal, Slurm, render farms | job handle, worst-case reservation, cancel, orphans | +| evaluator | held-out model eval, kernel timing | `dev_evaluate` vs `score`, exclusive hardware, cache | + +**Budget ledger.** Money is integer micro-USD. Before a job starts the broker reserves its worst case, the flavor's per-minute price times the job's timeout, and refuses the job with a typed `budget_exhausted` error if it does not fit. When the job ends the reservation is replaced by the real cost, the per-minute price times the minutes the job ran, rounded up to whole minutes as HF Jobs bills. Budgets are per rollout and set by the orchestrator, so every rollout in a GRPO group faces the same limit. The agent sees its remaining budget as data, never as reward. The ledger is the broker's estimate; the provider's bill is authoritative, and a provider-side cap is the backstop (see Credentials). + +Every observation carries a stable spend entry. RFC 013's sampler never reads the environment: the trainer reads this entry from the final observation and forwards one value per rollout as `costs` to `update(task, rewards, costs)`. + +```python +obs.metadata["service_spend"] = { + "version": 1, + "settled": {"usd": 0.62, "gpu_s": 410.0, "jobs": 2, "calls": 1}, # cumulative for the episode + "reserved": {"usd": 0.0}, # open reservations; empty once every job has settled + "final": True, # false while a job is live or was orphaned +} +``` + +Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. `jobs` counts agent jobs and `calls` counts dev evaluations. Grader evaluations are not charged to the agent. + +**Jobs.** Tools return a handle. States are `queued → running → succeeded | failed | cancelled`, plus `cancelling` while a cancel is pending and `orphaned` when the provider does not confirm one in time. An orphaned job is settled at its full reservation, keeps `final: false` and blocks the next `reset` with an error, because it may still be running and billing. `reset`, `close`, `submit`, the step limit and the time limit cancel the episode's live jobs. Submissions take an idempotency key; a retry with the same key returns the same job and makes no second reservation. + +**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; anything else, including a score outside [0, 1], counts as `invalid` with score 0. + +**Artifacts.** An artifact is the output folder of a succeeded job, exposed as `openenv-artifact:////outputs` with each file's path, size and content hash. Graders mount the folder read-only; nothing is copied through the agent. + +### Key Design Decisions + +#### D1. The broker is the only credential holder + +- **Chosen approach**: The broker starts jobs with `HfApi.run_uv_job`, with explicit flavor, namespace, timeout and labels and an empty `secrets` map. It never runs `hf jobs uv run`, whose script header can pull the caller's `HF_TOKEN` into the job and set the namespace; the Python API [ignores that header](https://github.com/huggingface/huggingface_hub/blob/bb2ef7579c3d2dc2eb49b16ccf15a070fe5b7023/src/huggingface_hub/hf_api.py#L12665-L12668). +- **Rationale**: A fine-grained token with only `job.write` cannot touch repos, but a job holding it can start further jobs that skip the broker's budget checks (verified on HF Jobs). So the token never enters a job. +- **Broker token**: "Start and manage Jobs" plus write access to the namespace's `jobs-artifacts` bucket. A `job.write`-only token is refused (HTTP 403) when the broker uploads the job script to that bucket. +- **Backstop**: HF tokens carry no spend cap. Billing broker jobs to a Resource Group with a Jobs spend limit (Enterprise) caps the total; over the limit, HF rejects new jobs and cancels running ones. + +#### D2. Artifacts move through the HF Jobs artifacts bucket + +- **Chosen approach**: Follow `run_uv_job`'s own convention. It uploads the script to the namespace's private `jobs-artifacts` bucket and mounts that job's folder read-write at `/data`. Jobs write outputs to `/data/outputs/`. Evaluator jobs mount `/outputs` read-only at `/model`. +- **Rationale**: No token inside the job, no extra repository, and HF's existing behaviour. Content hashes come from the bucket listing (Xet hashes). +- **Trade-offs**: Every agent job and evaluation creates a bucket folder; cleanup follows the bucket's retention. + +#### D3. Tool names use an underscore prefix + +- **Chosen approach**: `jobs_submit`, `jobs_status`, `jobs_wait`, `jobs_logs`, `jobs_cancel`, `dev_evaluate`, `submit` and `budget`. +- **Rationale**: OpenAI-compatible function names allow only `[a-zA-Z0-9_-]`, and most trainers convert MCP tools to that form, so `jobs.submit` breaks them. The reserved-name prefix rule in core (step 2) covers `jobs_`. + +#### D4. Long calls block with a declared timeout + +- **Chosen approach**: `submit` waits for the score job and returns the reward in the same step; `jobs_wait` waits up to a capped number of seconds. The environment raises its tool timeout for these calls and reports `score_timeout_s` at `reset` so clients can set their message deadline (the same need as #1026). +- **Rationale**: The reward stays inside the environment (RFC 004) and the episode ends with it, without a side channel. + +### Decisions from the discussion + +1. **Long-running services and GRPO.** A service runs live at train time only if it finishes within the trainer's staleness window; for example, TRL's experimental `AsyncGRPOTrainer` drops samples more than [`max_staleness`](https://github.com/huggingface/trl/blob/0dcd18c48bf7e63b49a23da0cb030b0a93a2f39f/trl/experimental/async_grpo/async_grpo_config.py#L126-L128) weight updates old (default 4). Slower services declare a stand-in and the real service runs only at eval. Post-training tasks are multi-hour episodes, so their near-term use is evaluation. +2. **HF Spaces.** No services on Spaces for now. A Space gives no isolation guarantee, so it must not hold a broker credential. +3. **Shared brokers are operator-run services outside the environment**, with the standing of RFC 009's backend gateways. The orchestrator's run config lists each broker's URL and expected TLS identity; that registration is the trust anchor. The daemon exchanges its broker credential for per-rollout tokens bound to one broker ([RFC 8693](https://www.rfc-editor.org/rfc/rfc8693), [RFC 8707](https://www.rfc-editor.org/rfc/rfc8707)); brokers are MCP resource servers ([RFC 9728](https://www.rfc-editor.org/rfc/rfc9728)) that enforce a hard cap per token. This follows RSIGym, LiteLLM virtual keys and OpenMLE Sandbox. +4. **Transport targets MCP 2026-07-28**, which [removes protocol sessions](https://github.com/modelcontextprotocol/modelcontextprotocol/blob/86f6e0c675afea9a8f233e09b6faf566b126086c/docs/specification/2026-07-28/changelog.mdx) in favour of server-minted handles passed as arguments ([SEP-2567](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2567)). That matches `openenv/session/create` and the job handles above. The upgrade adds `server/discover`, per-request protocol versions, `resultType`, the `Mcp-Method`/`Mcp-Name` headers and cacheable `tools/list`, and enables the [Tasks extension](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2663). It comes later as a backward-compatible RFC 003 amendment; v1 jobs do not need it. + +### Not allowed + +- Raw provider keys in the agent sandbox or in agent jobs. +- Grading in the agent's sandbox or jobs. +- Reward writes the agent can reach. +- Passing the broker's token into jobs. +- Judges reading unescaped trajectories. +- Building on MCP sampling or payment protocols; MCP has no official cost or payment extension. + +## Examples + +`posttrain_env` with task `t0_arithmetic_format`, driven by any agent loop: + +```python +from posttrain_env import PostTrainEnv + +with PostTrainEnv(base_url="http://localhost:8000", message_timeout_s=3600).sync() as env: + result = env.reset(task_id="t0_arithmetic_format") + print(result.observation.metadata["prompt"]) + print(env.call_tool("budget")) + # {'remaining_usd': 3.0, 'flavors': {'t4-small': {'usd_per_hour': 0.4}, ...}, 'jobs_left': 6, ...} + + job = env.call_tool("jobs_submit", script=train_py, flavor="t4-small", timeout_minutes=30) + status = env.call_tool("jobs_wait", job_id=job["job_id"], timeout_s=300) + # {'state': 'succeeded', 'cost_usd': 0.04, + # 'artifact': {'uri': 'openenv-artifact:////outputs', 'files': [...]}} + + dev = env.call_tool("dev_evaluate", job_id=job["job_id"]) + env.call_tool("jobs_wait", job_id=dev["job_id"], timeout_s=300) # result: {'score': ...} + + final = env.call_tool("submit", job_id=job["job_id"]) # ends the episode; reward = hidden-split score +``` + +The final observation carries `reward`, `metadata["service_spend"]` and `metadata["episode_end"]`. + +## Prototype results + +`envs/posttrain_env` runs this design on HF Jobs. Its unit tests replace HF Jobs with an in-memory fake, with no network access. Two live runs on 2026-10-16: + +| Task | Agent job | Dev eval | Hidden-split reward | Rollout spend | +|---|---|---|---|---| +| `t0_artifact` | `cpu-basic`, 4 s | 1.0 | 1.0 | $0.000334 | +| `t0_arithmetic_format` | SFT of SmolLM2-135M-Instruct, `t4-small`, 333 s | 0.94 | 0.95 | $0.041 (`gpu_s` 333) | + +Both runs ended with no orphaned jobs and `final: true`. The score jobs were not charged to the agent. A run with a `job.write`-only broker token was refused with HTTP 403 when the script upload reached the `jobs-artifacts` bucket (D1). + +## Plan + +1. **This PR**: the RFC and `envs/posttrain_env`, with no core changes. Two T0 tasks: `t0_artifact` checks the plumbing for a fraction of a cent; `t0_arithmetic_format` is a real SFT task with a hidden test split. +2. **Core types**: manifest, ledger, job and artifact models outside `server/`; the reserved-name prefix rule for `jobs_`; cleanup that surfaces cancel failures (`_cleanup_session_resources` swallows errors today) and the idle reaper on when services are declared. +3. **openenvd integration** (RFC 009): credential custody, token exchange, the grader zone, and MCP Tasks after the RFC 003 amendment. + +## Future Work + +Profiles added when an environment needs one: + +| Profile | Examples | Adds | +|---|---|---| +| model call | inference, LLM/VLM judge | token metering | +| training session | Tinker-compatible APIs | session state, checkpoints as artifacts | +| interactive simulator | Blender, Isaac, browser, VM | lease, snapshot/fork, hardware capability | +| sub-sandbox | E2B, Daytona | nested isolation, per-service network policy | +| data access | search, paper/chem DBs | licensing, contamination audit, caching | +| licensed tool | FlexLM EDA, quantum queues | non-money leases (seats, daily quotas) | +| real-world actuation | cloud lab, robot, deploy | irreversible spend, orchestrator approval | +| human in the loop | expert review | hours-to-days latency, scarcity | +| artifact registry | HF Hub push | write-scoped credentials, provenance | +| peer agent | A2A | budget passed down to the peer (#779) | + +Also: audit records keyed to `(episode, step, tool_call_id)` as OpenTelemetry GenAI/MCP spans, signed checkpoints (OpenSSF model signing), and exclusive-hardware leases for timing evaluators. + +## Related + +- RFC 009 openenvd (#1053, #1354, #1355): the base for credential custody and the grader zone. +- RFC 013 task sampling (#1256): consumes `service_spend`. +- RFC 003: the MCP 2026-07-28 amendment. RFC 004: `score` feeds rubrics. RFC 005/006: LLM proxying is the future model-call profile. RFC 008: contract checks for declared services, plus #1357's adversarial panel. RFC 012: separate-verifier precedent. +- Issues: #898, #1026, #1232, #1299, #1357, #779. diff --git a/rfcs/README.md b/rfcs/README.md index 8c2824b2a5..836a05f962 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -101,6 +101,9 @@ Each RFC should include the following sections: ### World Modeling - [010-echo-env-token-world-model.md](./010-echo-env-token-world-model.md) - Env-token World Modeling (ECHO): trajectory token-role masks + an optimizer world-loss seam +### Environment Services +- [014-environment-services.md](./014-environment-services.md) - Environment Services: brokered, budgeted access to external services (HF Jobs batch jobs and evaluators in v1) + ### Discovery & Distribution - [011-ard-catalog-discovery.md](./011-ard-catalog-discovery.md) - Versioned environment metadata, ARD interchange, and a read-only producer-consumer path From 11c971365abad8de07660eb9a90c3124bcca46dc Mon Sep 17 00:00:00 2001 From: swappy <59965507+rycerzes@users.noreply.github.com> Date: Sat, 17 Oct 2026 14:35:46 +0530 Subject: [PATCH 02/11] Score only the checkpoint's weights in the posttrain_env evaluator RFC-only extraction from original commit a77ae5eee743907c5619c2f5f05422d886e2ba21 in openenv/OpenEnv#1500. --- rfcs/014-environment-services.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index a1898fb728..e891025c6c 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -92,7 +92,7 @@ Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. **Jobs.** Tools return a handle. States are `queued → running → succeeded | failed | cancelled`, plus `cancelling` while a cancel is pending and `orphaned` when the provider does not confirm one in time. An orphaned job is settled at its full reservation, keeps `final: false` and blocks the next `reset` with an error, because it may still be running and billing. `reset`, `close`, `submit`, the step limit and the time limit cancel the episode's live jobs. Submissions take an idempotency key; a retry with the same key returns the same job and makes no second reservation. -**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; anything else, including a score outside [0, 1], counts as `invalid` with score 0. +**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; anything else, including a score outside [0, 1], counts as `invalid` with score 0. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer and chat template at a pinned revision, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release. **Artifacts.** An artifact is the output folder of a succeeded job, exposed as `openenv-artifact:////outputs` with each file's path, size and content hash. Graders mount the folder read-only; nothing is copied through the agent. From dcad63800b49ee409141df232b0f473fbbbb8c42 Mon Sep 17 00:00:00 2001 From: swappy <59965507+rycerzes@users.noreply.github.com> Date: Mon, 19 Oct 2026 00:19:03 +0530 Subject: [PATCH 03/11] Keep spend out of sampling weights in RFC 014 RFC-only extraction from original commit 0bd799196c9b46916eee9ce1dee3ef99c8eeb99b in openenv/OpenEnv#1500. --- rfcs/014-environment-services.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index e891025c6c..2e6cf8a270 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -29,7 +29,7 @@ Environments answer five questions ad hoc: 1. A service contract an environment declares, instead of per-env plumbing. 2. Credentials that stay with a broker; the agent and its jobs hold none. -3. A per-rollout budget that reserves before spending and settles after, published in a stable form for cost-aware samplers (RFC 013). +3. A per-rollout budget that reserves before spending and settles after, published in a stable form so trainers can account for spend and plan budgets and concurrency. 4. Jobs with handles, cancellation at episode end and a defined orphan state. 5. Evaluation split into an agent-callable dev endpoint and a grader-only score. @@ -77,7 +77,7 @@ budget: {usd: 3.0, scope: rollout} **Budget ledger.** Money is integer micro-USD. Before a job starts the broker reserves its worst case, the flavor's per-minute price times the job's timeout, and refuses the job with a typed `budget_exhausted` error if it does not fit. When the job ends the reservation is replaced by the real cost, the per-minute price times the minutes the job ran, rounded up to whole minutes as HF Jobs bills. Budgets are per rollout and set by the orchestrator, so every rollout in a GRPO group faces the same limit. The agent sees its remaining budget as data, never as reward. The ledger is the broker's estimate; the provider's bill is authoritative, and a provider-side cap is the backstop (see Credentials). -Every observation carries a stable spend entry. RFC 013's sampler never reads the environment: the trainer reads this entry from the final observation and forwards one value per rollout as `costs` to `update(task, rewards, costs)`. +Every observation carries a stable spend entry. It is for accounting and for planning budgets and concurrency, never a sampling weight: dividing a task's sampling weight by its cost would starve expensive, hard tasks. RFC 013's sampler never reads the environment; the trainer reads this entry from the final observation and can report one value per rollout as `costs` to `update(task, rewards, costs)`. ```python obs.metadata["service_spend"] = { @@ -202,6 +202,6 @@ Also: audit records keyed to `(episode, step, tool_call_id)` as OpenTelemetry Ge ## Related - RFC 009 openenvd (#1053, #1354, #1355): the base for credential custody and the grader zone. -- RFC 013 task sampling (#1256): consumes `service_spend`. +- RFC 013 task sampling (#1256): reports `service_spend` as cost, not as a sampling weight. - RFC 003: the MCP 2026-07-28 amendment. RFC 004: `score` feeds rubrics. RFC 005/006: LLM proxying is the future model-call profile. RFC 008: contract checks for declared services, plus #1357's adversarial panel. RFC 012: separate-verifier precedent. - Issues: #898, #1026, #1232, #1299, #1357, #779. From 9291dfcddbf0144a1f2eac93d7c6e3861b69b817 Mon Sep 17 00:00:00 2001 From: swappy <59965507+rycerzes@users.noreply.github.com> Date: Tue, 27 Oct 2026 01:36:30 +0530 Subject: [PATCH 04/11] End posttrain_env episodes unscored when the score job fails, and install CPU-only torch in its evaluator RFC-only extraction from original commit 434cbbd78040e355fafcb2949bbb1d47c4537e8b in openenv/OpenEnv#1500. --- rfcs/014-environment-services.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index 2e6cf8a270..d58b7615a2 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -92,7 +92,7 @@ Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. **Jobs.** Tools return a handle. States are `queued → running → succeeded | failed | cancelled`, plus `cancelling` while a cancel is pending and `orphaned` when the provider does not confirm one in time. An orphaned job is settled at its full reservation, keeps `final: false` and blocks the next `reset` with an error, because it may still be running and billing. `reset`, `close`, `submit`, the step limit and the time limit cancel the episode's live jobs. Submissions take an idempotency key; a retry with the same key returns the same job and makes no second reservation. -**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; anything else, including a score outside [0, 1], counts as `invalid` with score 0. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer and chat template at a pinned revision, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release. +**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; anything else, including a score outside [0, 1], counts as `invalid` with score 0. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer and chat template at a pinned revision, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release, and as a CPU job it installs CPU-only torch. A score job that fails, times out or is cancelled ends the episode unscored (`reward` is `None`, `episode_end.reason` is `score_failed`): the fault is the grader's, and RFC 013 treats a missing reward as missing, not zero. **Artifacts.** An artifact is the output folder of a succeeded job, exposed as `openenv-artifact:////outputs` with each file's path, size and content hash. Graders mount the folder read-only; nothing is copied through the agent. From 2bece18d99634afc65d3f2a5e39363fbb65b197f Mon Sep 17 00:00:00 2001 From: swappy <59965507+rycerzes@users.noreply.github.com> Date: Tue, 27 Oct 2026 10:01:06 +0530 Subject: [PATCH 05/11] Add the network policy and new runs to RFC 014 RFC-only extraction from original commit c32dbb29f50f69c732f97b46feb1395f5bf5d1a6 in openenv/OpenEnv#1500. --- rfcs/014-environment-services.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index d58b7615a2..2b05f56ec1 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -92,6 +92,8 @@ Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. **Jobs.** Tools return a handle. States are `queued → running → succeeded | failed | cancelled`, plus `cancelling` while a cancel is pending and `orphaned` when the provider does not confirm one in time. An orphaned job is settled at its full reservation, keeps `final: false` and blocks the next `reset` with an error, because it may still be running and billing. `reset`, `close`, `submit`, the step limit and the time limit cancel the episode's live jobs. Submissions take an idempotency key; a retry with the same key returns the same job and makes no second reservation. +**Network.** Agent jobs run without network access. A job that can reach the internet can download a model already trained for the task, or the evaluation's own test data; [PostTrainBench](https://arxiv.org/abs/2603.08640) reports both. Inputs come as read-only mounts instead: the task's base model at a pinned revision, and the public, ungated Hub datasets the agent names, at their current revision, minus a per-task denylist. Packages are installed before the agent's code runs, from wheels only, so no package code runs while the job still has network access. HF Jobs has no egress control, and its containers can't create network namespaces or firewall rules (we checked on `cpu-basic` and `a10g-small`). So `posttrain_env` runs the agent's script inside a launcher that installs a seccomp filter refusing every socket except `AF_UNIX`, and io_uring and cross-process memory access, then drops to an unprivileged uid; systemd's `RestrictAddressFamilies=` works the same way. On hosts the operator controls, openenvd's `openshell` backend (RFC 009) gives the same `egress_control` guarantee with a network namespace. A denylist only catches the known copies of a test set, so tasks whose test data is public also need a contamination check after the run. + **Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; anything else, including a score outside [0, 1], counts as `invalid` with score 0. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer and chat template at a pinned revision, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release, and as a CPU job it installs CPU-only torch. A score job that fails, times out or is cancelled ends the episode unscored (`reward` is `None`, `episode_end.reason` is `score_failed`): the fault is the grader's, and RFC 013 treats a missing reward as missing, not zero. **Artifacts.** An artifact is the output folder of a succeeded job, exposed as `openenv-artifact:////outputs` with each file's path, size and content hash. Graders mount the folder read-only; nothing is copied through the agent. @@ -131,6 +133,7 @@ Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. ### Not allowed - Raw provider keys in the agent sandbox or in agent jobs. +- Network access from agent jobs. - Grading in the agent's sandbox or jobs. - Reward writes the agent can reach. - Passing the broker's token into jobs. @@ -165,18 +168,25 @@ The final observation carries `reward`, `metadata["service_spend"]` and `metadat ## Prototype results -`envs/posttrain_env` runs this design on HF Jobs. Its unit tests replace HF Jobs with an in-memory fake, with no network access. Two live runs on 2026-10-16: +`envs/posttrain_env` runs this design on HF Jobs. Its unit tests replace HF Jobs with an in-memory fake, with no network access. Live runs: + +| Date | Task | Driver | Agent jobs' network | Dev eval | Reward | Rollout spend | +|---|---|---|---|---|---|---| +| 2026-10-16 | `t0_artifact` | script | open | 1.0 | 1.0 | $0.000334 | +| 2026-10-16 | `t0_arithmetic_format` | script: SFT on `t4-small`, 333 s | open | 0.94 | 0.95 | $0.041 (`gpu_s` 333) | +| 2026-10-26 | `t0_arithmetic_format` | Qwen3.8 through HF Inference Providers, 25 and 29 turns | open | 1.0 | 1.0, 1.0 | $0.21, $0.34 | +| 2026-10-27 | `t0_artifact` | script | none | | 1.0 | $0.000167 | +| 2026-10-27 | `t0_arithmetic_format` | script: SFT on `t4-small`, 333 s | none | 0.94 | 0.93 | $0.041 (`gpu_s` 333) | -| Task | Agent job | Dev eval | Hidden-split reward | Rollout spend | -|---|---|---|---|---| -| `t0_artifact` | `cpu-basic`, 4 s | 1.0 | 1.0 | $0.000334 | -| `t0_arithmetic_format` | SFT of SmolLM2-135M-Instruct, `t4-small`, 333 s | 0.94 | 0.95 | $0.041 (`gpu_s` 333) | +Spend is the agent's HF Jobs spend; LLM tokens are not included. Every run ended with no orphaned jobs and `final: true`, and score jobs were not charged to the agent. A run with a `job.write`-only broker token was refused with HTTP 403 when the script upload reached the `jobs-artifacts` bucket (D1). -Both runs ended with no orphaned jobs and `final: true`. The score jobs were not charged to the agent. A run with a `job.write`-only broker token was refused with HTTP 403 when the script upload reached the `jobs-artifacts` bucket (D1). +- **No network.** In the 27 Oct runs the agent's script could not open a socket and ran as uid 65534. CUDA training, DataLoader workers and writing `/data/outputs/` worked as before. Loading the base model from its mount instead of the Hub also cut the score job from about 20 minutes to 140 s. +- **T0 measures plumbing, not training skill.** Qwen3.8 trained on all 10,000 `(a, b)` pairs, so a hidden split of an input space that small hides nothing. Real tasks need inputs that cannot be enumerated, or test data the agent cannot reach. +- **A failed score job is not a zero.** One Qwen3.8 run scored 1.0 on dev, and its score job printed 1.0 and then hit its timeout; the episode got reward 0. Such episodes now end unscored (`score_failed`), and the evaluator installs CPU-only torch. ## Plan -1. **This PR**: the RFC and `envs/posttrain_env`, with no core changes. Two T0 tasks: `t0_artifact` checks the plumbing for a fraction of a cent; `t0_arithmetic_format` is a real SFT task with a hidden test split. +1. **This PR**: the RFC and `envs/posttrain_env`, with no core changes and no network in agent jobs. Two T0 tasks: `t0_artifact` checks the plumbing for a fraction of a cent; `t0_arithmetic_format` is a real SFT task with a hidden test split. 2. **Core types**: manifest, ledger, job and artifact models outside `server/`; the reserved-name prefix rule for `jobs_`; cleanup that surfaces cancel failures (`_cleanup_session_resources` swallows errors today) and the idle reaper on when services are declared. 3. **openenvd integration** (RFC 009): credential custody, token exchange, the grader zone, and MCP Tasks after the RFC 003 amendment. From f176553dab92663da587ac6ecc870091278f5b86 Mon Sep 17 00:00:00 2001 From: swappy <59965507+rycerzes@users.noreply.github.com> Date: Tue, 27 Oct 2026 10:57:52 +0530 Subject: [PATCH 06/11] Describe what the network policy does and doesn't stop in RFC 014 RFC-only extraction from original commit 60ca41128b9009d187eb5f99c28850373b9da8f9 in openenv/OpenEnv#1500. --- rfcs/014-environment-services.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index 2b05f56ec1..afe36c5281 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -92,7 +92,7 @@ Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. **Jobs.** Tools return a handle. States are `queued → running → succeeded | failed | cancelled`, plus `cancelling` while a cancel is pending and `orphaned` when the provider does not confirm one in time. An orphaned job is settled at its full reservation, keeps `final: false` and blocks the next `reset` with an error, because it may still be running and billing. `reset`, `close`, `submit`, the step limit and the time limit cancel the episode's live jobs. Submissions take an idempotency key; a retry with the same key returns the same job and makes no second reservation. -**Network.** Agent jobs run without network access. A job that can reach the internet can download a model already trained for the task, or the evaluation's own test data; [PostTrainBench](https://arxiv.org/abs/2603.08640) reports both. Inputs come as read-only mounts instead: the task's base model at a pinned revision, and the public, ungated Hub datasets the agent names, at their current revision, minus a per-task denylist. Packages are installed before the agent's code runs, from wheels only, so no package code runs while the job still has network access. HF Jobs has no egress control, and its containers can't create network namespaces or firewall rules (we checked on `cpu-basic` and `a10g-small`). So `posttrain_env` runs the agent's script inside a launcher that installs a seccomp filter refusing every socket except `AF_UNIX`, and io_uring and cross-process memory access, then drops to an unprivileged uid; systemd's `RestrictAddressFamilies=` works the same way. On hosts the operator controls, openenvd's `openshell` backend (RFC 009) gives the same `egress_control` guarantee with a network namespace. A denylist only catches the known copies of a test set, so tasks whose test data is public also need a contamination check after the run. +**Network.** Agent jobs run without network access. A job that can reach the internet can download a model already trained for the task, or the evaluation's own test data; [PostTrainBench](https://arxiv.org/abs/2603.08640) reports both. Inputs come as read-only mounts instead: the task's base model at a pinned revision, and the public, ungated Hub datasets the agent names, at their current revision, minus a per-task denylist. Packages are installed into a fresh venv before the agent's code runs, from wheels only; installing a wheel runs none of its code, so no package code runs while the job still has network access. HF Jobs has no egress control, and its containers can't create network namespaces or firewall rules (we checked on `cpu-basic` and `a10g-small`). So `posttrain_env` runs the agent's script inside a launcher that installs a seccomp filter refusing every socket except `AF_UNIX`, and io_uring and cross-process memory access, then closes inherited file descriptors and drops to a uid no other process in the job uses; systemd's `RestrictAddressFamilies=` works the same way. On hosts the operator controls, openenvd's `openshell` backend (RFC 009) gives the same `egress_control` guarantee with a network namespace. This narrows what a job can fetch to declared, recorded inputs, but it doesn't make them clean: a public dataset or a PyPI wheel can still hold weights or test data, and a denylist only catches known copies. Tasks whose test data is public also need a contamination check after the run. **Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; anything else, including a score outside [0, 1], counts as `invalid` with score 0. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer and chat template at a pinned revision, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release, and as a CPU job it installs CPU-only torch. A score job that fails, times out or is cancelled ends the episode unscored (`reward` is `None`, `episode_end.reason` is `score_failed`): the fault is the grader's, and RFC 013 treats a missing reward as missing, not zero. @@ -180,7 +180,7 @@ The final observation carries `reward`, `metadata["service_spend"]` and `metadat Spend is the agent's HF Jobs spend; LLM tokens are not included. Every run ended with no orphaned jobs and `final: true`, and score jobs were not charged to the agent. A run with a `job.write`-only broker token was refused with HTTP 403 when the script upload reached the `jobs-artifacts` bucket (D1). -- **No network.** In the 27 Oct runs the agent's script could not open a socket and ran as uid 65534. CUDA training, DataLoader workers and writing `/data/outputs/` worked as before. Loading the base model from its mount instead of the Hub also cut the score job from about 20 minutes to 140 s. +- **No network.** In the 27 Oct runs the agent's script could not open a socket and ran as an unprivileged uid. CUDA training, DataLoader workers and writing `/data/outputs/` worked as before. Loading the base model from its mount instead of the Hub also cut the score job from about 20 minutes to 140 s. - **T0 measures plumbing, not training skill.** Qwen3.8 trained on all 10,000 `(a, b)` pairs, so a hidden split of an input space that small hides nothing. Real tasks need inputs that cannot be enumerated, or test data the agent cannot reach. - **A failed score job is not a zero.** One Qwen3.8 run scored 1.0 on dev, and its score job printed 1.0 and then hit its timeout; the episode got reward 0. Such episodes now end unscored (`score_failed`), and the evaluator installs CPU-only torch. From 58d8fb249c8c4a94cb19f122659d82402e54aed1 Mon Sep 17 00:00:00 2001 From: swappy <59965507+rycerzes@users.noreply.github.com> Date: Tue, 27 Oct 2026 11:22:46 +0530 Subject: [PATCH 07/11] Add the no-network Qwen3.8 run to RFC 014 RFC-only extraction from original commit 0dda477d69988177270e190c82b0445b0963f018 in openenv/OpenEnv#1500. --- rfcs/014-environment-services.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index afe36c5281..6ce63f14b7 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -177,10 +177,11 @@ The final observation carries `reward`, `metadata["service_spend"]` and `metadat | 2026-10-26 | `t0_arithmetic_format` | Qwen3.8 through HF Inference Providers, 25 and 29 turns | open | 1.0 | 1.0, 1.0 | $0.21, $0.34 | | 2026-10-27 | `t0_artifact` | script | none | | 1.0 | $0.000167 | | 2026-10-27 | `t0_arithmetic_format` | script: SFT on `t4-small`, 333 s | none | 0.94 | 0.93 | $0.041 (`gpu_s` 333) | +| 2026-10-27 | `t0_arithmetic_format` | Qwen3.8 through HF Inference Providers, 23 turns | none | 1.0 | 1.0 | $0.12 | Spend is the agent's HF Jobs spend; LLM tokens are not included. Every run ended with no orphaned jobs and `final: true`, and score jobs were not charged to the agent. A run with a `job.write`-only broker token was refused with HTTP 403 when the script upload reached the `jobs-artifacts` bucket (D1). -- **No network.** In the 27 Oct runs the agent's script could not open a socket and ran as an unprivileged uid. CUDA training, DataLoader workers and writing `/data/outputs/` worked as before. Loading the base model from its mount instead of the Hub also cut the score job from about 20 minutes to 140 s. +- **No network.** In the 27 Oct runs the agent's script could not open a socket and ran as an unprivileged uid. CUDA training, DataLoader workers and writing `/data/outputs/` worked as before. Qwen3.8 loaded the base model from `/base_model` from its first script on; its three failed jobs were its own bugs and one CUDA allocation error, none from the network policy. Loading the base model from its mount instead of the Hub also cut the score job from about 20 minutes to 140 s. - **T0 measures plumbing, not training skill.** Qwen3.8 trained on all 10,000 `(a, b)` pairs, so a hidden split of an input space that small hides nothing. Real tasks need inputs that cannot be enumerated, or test data the agent cannot reach. - **A failed score job is not a zero.** One Qwen3.8 run scored 1.0 on dev, and its score job printed 1.0 and then hit its timeout; the episode got reward 0. Such episodes now end unscored (`score_failed`), and the evaluator installs CPU-only torch. From 615404ccfe6e648866cba331bc03d9558cf76560 Mon Sep 17 00:00:00 2001 From: swappy <59965507+rycerzes@users.noreply.github.com> Date: Tue, 27 Oct 2026 11:56:06 +0530 Subject: [PATCH 08/11] Scope RFC 014 to the agreed first slice and align it with RFC 013 RFC-only extraction from original commit 073ebcdd61a3043896f5c4c28a102378ebe3536f in openenv/OpenEnv#1500. --- rfcs/014-environment-services.md | 48 +++++++++++++++----------------- 1 file changed, 23 insertions(+), 25 deletions(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index 6ce63f14b7..7c3071f3a3 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -37,22 +37,23 @@ Environments answer five questions ad hoc: - Hosting services. Brokers wrap existing ones (HF Jobs first). - Profiles beyond batch job and evaluator in v1 (see [Future Work](#future-work)). -- Changing `/mcp` in this RFC. The move to MCP 2026-07-28 is a later RFC 003 amendment (decision 4). +- Changing `/mcp` or the core types in this RFC. ## Design ### Architecture Overview ``` -agent sandbox ──MCP tools──▶ environment / openenvd (per env) - (no credentials) ledger · artifacts · audit · token exchange - │ broker credential, audience-bound - ▼ - shared brokers: jobs (HF Jobs) · eval -grader ◀── artifact handles + score jobs ──┘ +agent ──MCP tools──▶ environment server = broker (holds the HF token) + (no credentials) ledger · job handles · artifacts + │ run_uv_job, no secrets + ▼ + HF Jobs: agent jobs (no network) · dev_evaluate · score + │ + score job ◀────┘ artifact mounted read-only; its score is the reward ``` -Without openenvd (RFC 009), the environment server is the broker, as in `posttrain_env`. With openenvd, the daemon holds the broker credential as a backend credential and the environment only sees run tokens. +In v1 the environment server is the broker, as in `posttrain_env`. Moving the credential to openenvd (RFC 009) is later work. ### Core Abstractions @@ -73,11 +74,11 @@ budget: {usd: 3.0, scope: rollout} | Profile | Examples | Adds to the contract | |---|---|---| | batch job | HF Jobs; later Modal, Slurm, render farms | job handle, worst-case reservation, cancel, orphans | -| evaluator | held-out model eval, kernel timing | `dev_evaluate` vs `score`, exclusive hardware, cache | +| evaluator | held-out model eval, kernel timing | `dev_evaluate` vs `score`; exclusive hardware and caching for timing (not in the prototype) | -**Budget ledger.** Money is integer micro-USD. Before a job starts the broker reserves its worst case, the flavor's per-minute price times the job's timeout, and refuses the job with a typed `budget_exhausted` error if it does not fit. When the job ends the reservation is replaced by the real cost, the per-minute price times the minutes the job ran, rounded up to whole minutes as HF Jobs bills. Budgets are per rollout and set by the orchestrator, so every rollout in a GRPO group faces the same limit. The agent sees its remaining budget as data, never as reward. The ledger is the broker's estimate; the provider's bill is authoritative, and a provider-side cap is the backstop (see Credentials). +**Budget ledger.** Money is integer micro-USD. Before a job starts the broker reserves its worst case, the flavor's per-minute price times the job's timeout, and refuses the job with a typed `budget_exhausted` error if it does not fit. When the job ends the reservation is replaced by the real cost, the per-minute price times the minutes the job ran, rounded up to whole minutes as HF Jobs bills. Budgets are per rollout and set by the orchestrator, so every rollout in a GRPO group faces the same limit. The agent sees its remaining budget as data, never as reward. The ledger is the broker's estimate; the provider's bill is authoritative, and a provider-side cap is the backstop (D1). -Every observation carries a stable spend entry. It is for accounting and for planning budgets and concurrency, never a sampling weight: dividing a task's sampling weight by its cost would starve expensive, hard tasks. RFC 013's sampler never reads the environment; the trainer reads this entry from the final observation and can report one value per rollout as `costs` to `update(task, rewards, costs)`. +Every observation carries a stable spend entry. It is for accounting, and for budgets, concurrency and rollout allocation, never a sampling weight: dividing a task's sampling weight by its cost would starve expensive, hard tasks. RFC 013's sampler never reads the environment; the trainer reads this entry from the final observation and can report one value per rollout as `costs` to `update(task, rewards, costs)`. RFC 013 takes costs for every group or for none, so an episode that ends unscored still reports its spend. ```python obs.metadata["service_spend"] = { @@ -88,13 +89,13 @@ obs.metadata["service_spend"] = { } ``` -Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. `jobs` counts agent jobs and `calls` counts dev evaluations. Grader evaluations are not charged to the agent. +v1 reports `usd`, `gpu_s`, `jobs` and `calls`. `jobs` counts agent jobs and `calls` counts dev evaluations. Grader evaluations are not charged to the agent. **Jobs.** Tools return a handle. States are `queued → running → succeeded | failed | cancelled`, plus `cancelling` while a cancel is pending and `orphaned` when the provider does not confirm one in time. An orphaned job is settled at its full reservation, keeps `final: false` and blocks the next `reset` with an error, because it may still be running and billing. `reset`, `close`, `submit`, the step limit and the time limit cancel the episode's live jobs. Submissions take an idempotency key; a retry with the same key returns the same job and makes no second reservation. -**Network.** Agent jobs run without network access. A job that can reach the internet can download a model already trained for the task, or the evaluation's own test data; [PostTrainBench](https://arxiv.org/abs/2603.08640) reports both. Inputs come as read-only mounts instead: the task's base model at a pinned revision, and the public, ungated Hub datasets the agent names, at their current revision, minus a per-task denylist. Packages are installed into a fresh venv before the agent's code runs, from wheels only; installing a wheel runs none of its code, so no package code runs while the job still has network access. HF Jobs has no egress control, and its containers can't create network namespaces or firewall rules (we checked on `cpu-basic` and `a10g-small`). So `posttrain_env` runs the agent's script inside a launcher that installs a seccomp filter refusing every socket except `AF_UNIX`, and io_uring and cross-process memory access, then closes inherited file descriptors and drops to a uid no other process in the job uses; systemd's `RestrictAddressFamilies=` works the same way. On hosts the operator controls, openenvd's `openshell` backend (RFC 009) gives the same `egress_control` guarantee with a network namespace. This narrows what a job can fetch to declared, recorded inputs, but it doesn't make them clean: a public dataset or a PyPI wheel can still hold weights or test data, and a denylist only catches known copies. Tasks whose test data is public also need a contamination check after the run. +**Network.** Agent jobs run without network access. A job that can reach the internet can download a model already trained for the task, or the evaluation's own test data; [PostTrainBench](https://arxiv.org/abs/2603.08640) reports both. Inputs come as read-only mounts instead: the task's base model at a pinned revision, and the public, ungated Hub datasets the agent names, at their current revision, minus a per-task denylist. Packages are installed into a fresh venv from wheels only, which runs none of their code, before the agent's code runs. HF Jobs has no egress control, and its containers can't create network namespaces or firewall rules (checked on `cpu-basic` and `a10g-small`). So `posttrain_env` runs the agent's script inside a launcher that installs a seccomp filter refusing every socket except `AF_UNIX`, and io_uring and cross-process memory access, then closes inherited file descriptors and drops to a uid no other process in the job uses; systemd's `RestrictAddressFamilies=` works the same way. This limits a job to declared, recorded inputs, but doesn't make them clean: a public dataset or a PyPI wheel can still hold weights or test data. Tasks whose test data is public also need a contamination check after the run. -**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; anything else, including a score outside [0, 1], counts as `invalid` with score 0. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer and chat template at a pinned revision, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release, and as a CPU job it installs CPU-only torch. A score job that fails, times out or is cancelled ends the episode unscored (`reward` is `None`, `episode_end.reason` is `score_failed`): the fault is the grader's, and RFC 013 treats a missing reward as missing, not zero. +**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward; the hidden split's seed reaches it as a secret. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; a checkpoint it can't use scores 0 as `invalid`. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer, chat template and generation settings from the base model mount, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release. A score job that fails, times out, is cancelled or prints no valid score ends the episode unscored (`reward` is `None`, `episode_end.reason` is `score_failed`): the fault is the grader's, and RFC 013's `update` takes `None` for a rollout the environment did not score, which samplers leave out rather than count as zero. **Artifacts.** An artifact is the output folder of a succeeded job, exposed as `openenv-artifact:////outputs` with each file's path, size and content hash. Graders mount the folder read-only; nothing is copied through the agent. @@ -102,16 +103,16 @@ Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. #### D1. The broker is the only credential holder -- **Chosen approach**: The broker starts jobs with `HfApi.run_uv_job`, with explicit flavor, namespace, timeout and labels and an empty `secrets` map. It never runs `hf jobs uv run`, whose script header can pull the caller's `HF_TOKEN` into the job and set the namespace; the Python API [ignores that header](https://github.com/huggingface/huggingface_hub/blob/bb2ef7579c3d2dc2eb49b16ccf15a070fe5b7023/src/huggingface_hub/hf_api.py#L12665-L12668). +- **Chosen approach**: The broker starts jobs with `HfApi.run_uv_job`, with explicit flavor, namespace, timeout and labels. Agent jobs get an empty `secrets` map. It never runs `hf jobs uv run`, whose script header can [request secrets that resolve from the caller's environment](https://github.com/huggingface/huggingface_hub/blob/841c22b46cdc25b55a2bfb1c2ad97f418162c282/src/huggingface_hub/cli/jobs.py#L311-L323), `HF_TOKEN` included, and set the namespace; the Python API [ignores that header](https://github.com/huggingface/huggingface_hub/blob/841c22b46cdc25b55a2bfb1c2ad97f418162c282/src/huggingface_hub/hf_api.py#L12901-L12902). - **Rationale**: A fine-grained token with only `job.write` cannot touch repos, but a job holding it can start further jobs that skip the broker's budget checks (verified on HF Jobs). So the token never enters a job. - **Broker token**: "Start and manage Jobs" plus write access to the namespace's `jobs-artifacts` bucket. A `job.write`-only token is refused (HTTP 403) when the broker uploads the job script to that bucket. -- **Backstop**: HF tokens carry no spend cap. Billing broker jobs to a Resource Group with a Jobs spend limit (Enterprise) caps the total; over the limit, HF rejects new jobs and cancels running ones. +- **Backstop**: A token has no spend limit of its own; spend limits are set on a [Resource Group](https://huggingface.co/docs/hub/en/enterprise-resource-groups) (Enterprise). Billing broker jobs to a group with a Jobs spend limit caps the total; over the limit, HF refuses new jobs and cancels running ones. #### D2. Artifacts move through the HF Jobs artifacts bucket - **Chosen approach**: Follow `run_uv_job`'s own convention. It uploads the script to the namespace's private `jobs-artifacts` bucket and mounts that job's folder read-write at `/data`. Jobs write outputs to `/data/outputs/`. Evaluator jobs mount `/outputs` read-only at `/model`. - **Rationale**: No token inside the job, no extra repository, and HF's existing behaviour. Content hashes come from the bucket listing (Xet hashes). -- **Trade-offs**: Every agent job and evaluation creates a bucket folder; cleanup follows the bucket's retention. +- **Trade-offs**: Every agent job and evaluation creates a bucket folder; cleanup follows the bucket's retention, and what a job writes there is neither capped nor charged to the budget. #### D3. Tool names use an underscore prefix @@ -127,8 +128,6 @@ Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. 1. **Long-running services and GRPO.** A service runs live at train time only if it finishes within the trainer's staleness window; for example, TRL's experimental `AsyncGRPOTrainer` drops samples more than [`max_staleness`](https://github.com/huggingface/trl/blob/0dcd18c48bf7e63b49a23da0cb030b0a93a2f39f/trl/experimental/async_grpo/async_grpo_config.py#L126-L128) weight updates old (default 4). Slower services declare a stand-in and the real service runs only at eval. Post-training tasks are multi-hour episodes, so their near-term use is evaluation. 2. **HF Spaces.** No services on Spaces for now. A Space gives no isolation guarantee, so it must not hold a broker credential. -3. **Shared brokers are operator-run services outside the environment**, with the standing of RFC 009's backend gateways. The orchestrator's run config lists each broker's URL and expected TLS identity; that registration is the trust anchor. The daemon exchanges its broker credential for per-rollout tokens bound to one broker ([RFC 8693](https://www.rfc-editor.org/rfc/rfc8693), [RFC 8707](https://www.rfc-editor.org/rfc/rfc8707)); brokers are MCP resource servers ([RFC 9728](https://www.rfc-editor.org/rfc/rfc9728)) that enforce a hard cap per token. This follows RSIGym, LiteLLM virtual keys and OpenMLE Sandbox. -4. **Transport targets MCP 2026-07-28**, which [removes protocol sessions](https://github.com/modelcontextprotocol/modelcontextprotocol/blob/86f6e0c675afea9a8f233e09b6faf566b126086c/docs/specification/2026-07-28/changelog.mdx) in favour of server-minted handles passed as arguments ([SEP-2567](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2567)). That matches `openenv/session/create` and the job handles above. The upgrade adds `server/discover`, per-request protocol versions, `resultType`, the `Mcp-Method`/`Mcp-Name` headers and cacheable `tools/list`, and enables the [Tasks extension](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2663). It comes later as a backward-compatible RFC 003 amendment; v1 jobs do not need it. ### Not allowed @@ -137,7 +136,6 @@ Units are `usd`, `gpu_s`, `cpu_s`, `tokens`, `calls` and `jobs`, in base units. - Grading in the agent's sandbox or jobs. - Reward writes the agent can reach. - Passing the broker's token into jobs. -- Judges reading unescaped trajectories. - Building on MCP sampling or payment protocols; MCP has no official cost or payment extension. ## Examples @@ -187,9 +185,9 @@ Spend is the agent's HF Jobs spend; LLM tokens are not included. Every run ended ## Plan -1. **This PR**: the RFC and `envs/posttrain_env`, with no core changes and no network in agent jobs. Two T0 tasks: `t0_artifact` checks the plumbing for a fraction of a cent; `t0_arithmetic_format` is a real SFT task with a hidden test split. +1. **This PR**: the RFC and `envs/posttrain_env`, with no core changes and no network in agent jobs. Two T0 tasks that check the plumbing: `t0_artifact` for a fraction of a cent, and `t0_arithmetic_format`, a small SFT task with a hidden test split. 2. **Core types**: manifest, ledger, job and artifact models outside `server/`; the reserved-name prefix rule for `jobs_`; cleanup that surfaces cancel failures (`_cleanup_session_resources` swallows errors today) and the idle reaper on when services are declared. -3. **openenvd integration** (RFC 009): credential custody, token exchange, the grader zone, and MCP Tasks after the RFC 003 amendment. +3. **openenvd integration** (RFC 009): credential custody and the grader zone. ## Future Work @@ -208,11 +206,11 @@ Profiles added when an environment needs one: | artifact registry | HF Hub push | write-scoped credentials, provenance | | peer agent | A2A | budget passed down to the peer (#779) | -Also: audit records keyed to `(episode, step, tool_call_id)` as OpenTelemetry GenAI/MCP spans, signed checkpoints (OpenSSF model signing), and exclusive-hardware leases for timing evaluators. +Also: shared brokers outside the environment with per-rollout tokens ([RFC 8693](https://www.rfc-editor.org/rfc/rfc8693) token exchange), the MCP 2026-07-28 transport through an RFC 003 amendment, audit records keyed to `(episode, step, tool_call_id)` as OpenTelemetry GenAI/MCP spans, signed checkpoints (OpenSSF model signing), and exclusive-hardware leases for timing evaluators. ## Related - RFC 009 openenvd (#1053, #1354, #1355): the base for credential custody and the grader zone. -- RFC 013 task sampling (#1256): reports `service_spend` as cost, not as a sampling weight. -- RFC 003: the MCP 2026-07-28 amendment. RFC 004: `score` feeds rubrics. RFC 005/006: LLM proxying is the future model-call profile. RFC 008: contract checks for declared services, plus #1357's adversarial panel. RFC 012: separate-verifier precedent. +- RFC 013 task sampling (#1256): `service_spend` is one source of the trainer's `costs`, never a sampling weight; an unscored episode's `None` reward is left out of the sampler's estimates. +- RFC 004: `score` feeds rubrics. RFC 005/006: LLM proxying is the future model-call profile. RFC 008: contract checks for declared services, plus #1357's adversarial panel. RFC 012: separate-verifier precedent. - Issues: #898, #1026, #1232, #1299, #1357, #779. From d0efe88bbf041ce1da1b8fa4c46aa95d2c08daea Mon Sep 17 00:00:00 2001 From: swappy <59965507+rycerzes@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:29:51 +0530 Subject: [PATCH 09/11] Keep posttrain_env episode teardown off the event loop and address review notes Ending an episode on the WebSocket path waited for HF Jobs to confirm cancels on the shared event loop. Run it in a thread instead. The evaluator now rejects checkpoints with unexpected keys, and RFC 014 links to openenv/OpenEnv and describes trainer staleness without naming one framework. RFC-only extraction from original commit eb21148dbaac4947a471d5e7f30f8326017afbd7 in openenv/OpenEnv#1500. --- rfcs/014-environment-services.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index 7c3071f3a3..fcd1e649e3 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -5,7 +5,7 @@ **Authors**: @rycerzes **RFC ID**: 014 -Discussion: [#1382](https://github.com/huggingface/OpenEnv/issues/1382). Prototype: [`envs/posttrain_env`](../envs/posttrain_env/). +Discussion: [#1382](https://github.com/openenv/OpenEnv/issues/1382). Prototype: [`envs/posttrain_env`](../envs/posttrain_env/). ## Summary @@ -126,7 +126,7 @@ v1 reports `usd`, `gpu_s`, `jobs` and `calls`. `jobs` counts agent jobs and `cal ### Decisions from the discussion -1. **Long-running services and GRPO.** A service runs live at train time only if it finishes within the trainer's staleness window; for example, TRL's experimental `AsyncGRPOTrainer` drops samples more than [`max_staleness`](https://github.com/huggingface/trl/blob/0dcd18c48bf7e63b49a23da0cb030b0a93a2f39f/trl/experimental/async_grpo/async_grpo_config.py#L126-L128) weight updates old (default 4). Slower services declare a stand-in and the real service runs only at eval. Post-training tasks are multi-hour episodes, so their near-term use is evaluation. +1. **Long-running services and GRPO.** A service runs live at train time only if it finishes within the trainer's staleness window. Asynchronous GRPO trainers drop samples generated more than a few weight updates ago, so an episode has minutes, not hours. Slower services declare a stand-in and the real service runs only at eval. Post-training tasks are multi-hour episodes, so their near-term use is evaluation. 2. **HF Spaces.** No services on Spaces for now. A Space gives no isolation guarantee, so it must not hold a broker credential. ### Not allowed From db049f47bbd8b044d3fa679e2a9f93348a67e8a7 Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Tue, 27 Oct 2026 17:52:02 +0100 Subject: [PATCH 10/11] Clarify RFC 014 provider prerequisites and split experimental prototype --- rfcs/014-environment-services.md | 149 ++++++++++++++++++++++++++----- 1 file changed, 126 insertions(+), 23 deletions(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index fcd1e649e3..a65d86a8a1 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -5,7 +5,7 @@ **Authors**: @rycerzes **RFC ID**: 014 -Discussion: [#1382](https://github.com/openenv/OpenEnv/issues/1382). Prototype: [`envs/posttrain_env`](../envs/posttrain_env/). +Discussion: [#1382](https://github.com/openenv/OpenEnv/issues/1382). Prototype: [experimental `posttrain_env` PR #1500](https://github.com/openenv/OpenEnv/pull/1500). ## Summary @@ -67,7 +67,7 @@ services: budget: {usd: 3.0, scope: rollout} ``` -`posttrain_env` keeps this in its task definitions (`server/tasks.py`); the manifest becomes a core type in step 2 of the plan below. +`posttrain_env` keeps this in its task definitions (`server/tasks.py`); a possible core manifest is deferred to step 2, after the provider contract and second backend have been validated. **Service profiles (v1).** @@ -83,9 +83,14 @@ Every observation carries a stable spend entry. It is for accounting, and for bu ```python obs.metadata["service_spend"] = { "version": 1, - "settled": {"usd": 0.62, "gpu_s": 410.0, "jobs": 2, "calls": 1}, # cumulative for the episode + "settled": { + "usd": 0.62, + "gpu_s": 410.0, + "jobs": 2, + "calls": 1, + }, # cumulative for the episode "reserved": {"usd": 0.0}, # open reservations; empty once every job has settled - "final": True, # false while a job is live or was orphaned + "final": True, # false while a job is live or was orphaned } ``` @@ -95,10 +100,45 @@ v1 reports `usd`, `gpu_s`, `jobs` and `calls`. `jobs` counts agent jobs and `cal **Network.** Agent jobs run without network access. A job that can reach the internet can download a model already trained for the task, or the evaluation's own test data; [PostTrainBench](https://arxiv.org/abs/2603.08640) reports both. Inputs come as read-only mounts instead: the task's base model at a pinned revision, and the public, ungated Hub datasets the agent names, at their current revision, minus a per-task denylist. Packages are installed into a fresh venv from wheels only, which runs none of their code, before the agent's code runs. HF Jobs has no egress control, and its containers can't create network namespaces or firewall rules (checked on `cpu-basic` and `a10g-small`). So `posttrain_env` runs the agent's script inside a launcher that installs a seccomp filter refusing every socket except `AF_UNIX`, and io_uring and cross-process memory access, then closes inherited file descriptors and drops to a uid no other process in the job uses; systemd's `RestrictAddressFamilies=` works the same way. This limits a job to declared, recorded inputs, but doesn't make them clean: a public dataset or a PyPI wheel can still hold weights or test data. Tasks whose test data is public also need a contamination check after the run. -**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward; the hidden split's seed reaches it as a secret. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; a checkpoint it can't use scores 0 as `invalid`. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer, chat template and generation settings from the base model mount, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release. A score job that fails, times out, is cancelled or prints no valid score ends the episode unscored (`reward` is `None`, `episode_end.reason` is `score_failed`): the fault is the grader's, and RFC 013's `update` takes `None` for a rollout the environment did not score, which samplers leave out rather than count as zero. +**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward; the hidden split's seed reaches it as a secret. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; a checkpoint it can't use scores 0 as `invalid`. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer, chat template and generation settings from the base model mount, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release. A score job that fails, times out, is cancelled or prints no valid score ends the episode unscored (`reward` is `None`, `episode_end.reason` is `score_failed`): the fault is the grader's, and RFC 013's `update` takes `None` for a rollout the environment did not score, which samplers leave out rather than count as zero. This `None`-reward integration depends on acceptance of RFC 013 (#1256) and must follow its final unscored-rollout semantics; it is not an independently accepted core contract. **Artifacts.** An artifact is the output folder of a succeeded job, exposed as `openenv-artifact:////outputs` with each file's path, size and content hash. Graders mount the folder read-only; nothing is copied through the agent. +### Minimal batch-job provider contract (proposed) + +HF Jobs is the first implementation, not the definition of the interface. A +provider must offer these operations to the broker, without exposing its +credentials to agent code: + +- **Submit**: accept a workload, resource choice, deadline and declared inputs; + return a stable job handle and support reconciliation of ambiguous submissions + without duplicate work or spend. +- **Inspect**: report state and execution duration/usage for settlement, including + terminal failures and pending cancellation. +- **Cancel**: request termination and allow confirmation of a terminal state; + an unconfirmed cancellation remains an orphan that may still bill. +- **Logs**: read bounded diagnostics for the episode's own jobs, separately from + the authoritative evaluator result. +- **Price**: expose units, billing granularity and a conservative cost bound that + can be reserved before execution, then settled from observed usage. +- **Artifacts**: enumerate outputs and their identity/size, and provide read-only + access to an isolated grader without passing credentials through the agent. + +Per-minute billing, `run_uv_job`, the `jobs-artifacts` bucket, Resource Group +caps and the seccomp launcher are HF implementation choices. Other providers +must preserve reservations, credential/grader separation and no-network agent +execution using their own facilities. + +**Prerequisite before step 2:** sketch and validate this contract against a +second backend before proposing core types or reserved names. A local Docker +backend is proposed for no-spend end-to-end tests: container handles and inspect, +stop, logs, isolated output volumes, and an explicit local resource-price policy +would exercise these operations. Modal is another proposed check against +HF-shaped assumptions, especially network isolation. Neither backend is +implemented or adopted by this RFC. Native provider no-network support remains +a prerequisite for step 2; track HF support in +[huggingface_hub#5261](https://github.com/huggingface/huggingface_hub/issues/5261). + ### Key Design Decisions #### D1. The broker is the only credential holder @@ -117,7 +157,7 @@ v1 reports `usd`, `gpu_s`, `jobs` and `calls`. `jobs` counts agent jobs and `cal #### D3. Tool names use an underscore prefix - **Chosen approach**: `jobs_submit`, `jobs_status`, `jobs_wait`, `jobs_logs`, `jobs_cancel`, `dev_evaluate`, `submit` and `budget`. -- **Rationale**: OpenAI-compatible function names allow only `[a-zA-Z0-9_-]`, and most trainers convert MCP tools to that form, so `jobs.submit` breaks them. The reserved-name prefix rule in core (step 2) covers `jobs_`. +- **Rationale**: OpenAI-compatible function names allow only `[a-zA-Z0-9_-]`, and most trainers convert MCP tools to that form, so `jobs.submit` breaks them. These are prototype names only. Any reserved-name prefix rule in core is deferred until the provider contract and a second backend have been validated before step 2. #### D4. Long calls block with a declared timeout @@ -143,26 +183,89 @@ v1 reports `usd`, `gpu_s`, `jobs` and `calls`. `jobs` counts agent jobs and `cal `posttrain_env` with task `t0_arithmetic_format`, driven by any agent loop: ```python +from openenv.core.env_server.mcp_types import CallToolAction from posttrain_env import PostTrainEnv -with PostTrainEnv(base_url="http://localhost:8000", message_timeout_s=3600).sync() as env: +with PostTrainEnv( + base_url="http://localhost:8000", message_timeout_s=3600 +).sync() as env: result = env.reset(task_id="t0_arithmetic_format") print(result.observation.metadata["prompt"]) - print(env.call_tool("budget")) - # {'remaining_usd': 3.0, 'flavors': {'t4-small': {'usd_per_hour': 0.4}, ...}, 'jobs_left': 6, ...} - - job = env.call_tool("jobs_submit", script=train_py, flavor="t4-small", timeout_minutes=30) - status = env.call_tool("jobs_wait", job_id=job["job_id"], timeout_s=300) - # {'state': 'succeeded', 'cost_usd': 0.04, - # 'artifact': {'uri': 'openenv-artifact:////outputs', 'files': [...]}} - - dev = env.call_tool("dev_evaluate", job_id=job["job_id"]) - env.call_tool("jobs_wait", job_id=dev["job_id"], timeout_s=300) # result: {'score': ...} - - final = env.call_tool("submit", job_id=job["job_id"]) # ends the episode; reward = hidden-split score + result = env.step(CallToolAction(tool_name="budget", arguments={})) + print(result.observation.result["data"]) + + result = env.step( + CallToolAction( + tool_name="jobs_submit", + arguments={ + "script": train_py, + "flavor": "t4-small", + "timeout_minutes": 30, + }, + ) + ) + assert result.observation.error is None + job = result.observation.result["data"] + assert "error" not in job, job + status = "queued" + while status in {"queued", "running"}: + result = env.step( + CallToolAction( + tool_name="jobs_wait", + arguments={ + "job_id": job["job_id"], + "timeout_s": 300, + }, + ) + ) + assert not result.done, result.observation.metadata + status = result.observation.result["data"]["state"] + assert status == "succeeded", result.observation.result + + dev = env.step( + CallToolAction( + tool_name="dev_evaluate", + arguments={ + "job_id": job["job_id"], + }, + ) + ) + dev_job = dev.observation.result["data"] + assert "error" not in dev_job, dev_job + status = "queued" + while status in {"queued", "running"}: + result = env.step( + CallToolAction( + tool_name="jobs_wait", + arguments={ + "job_id": dev_job["job_id"], + "timeout_s": 300, + }, + ) + ) + assert not result.done, result.observation.metadata + status = result.observation.result["data"]["state"] + assert status == "succeeded", result.observation.result + print(result.observation.result["data"]["result"]) + + final = env.step( + CallToolAction( + tool_name="submit", + arguments={ + "job_id": job["job_id"], + }, + ) + ) + print(final.done, final.reward, final.observation.metadata["service_spend"]) ``` -The final observation carries `reward`, `metadata["service_spend"]` and `metadata["episode_end"]`. +The final step result carries `reward`, and its observation carries +`metadata["service_spend"]` and `metadata["episode_end"]`. Direct `call_tool` +uses `/mcp`, bypasses episode step/time limits and does not return this final +step observation; core enforcement is tracked in [#1507](https://github.com/openenv/OpenEnv/issues/1507). +`service_spend["final"]` means all jobs have settled, not that the episode ended. +`submit` blocks for up to reset metadata's `score_timeout_s` (2700 s for this +task), so allow additional cancellation/transport time in client deadlines. ## Prototype results @@ -185,8 +288,8 @@ Spend is the agent's HF Jobs spend; LLM tokens are not included. Every run ended ## Plan -1. **This PR**: the RFC and `envs/posttrain_env`, with no core changes and no network in agent jobs. Two T0 tasks that check the plumbing: `t0_artifact` for a fraction of a cent, and `t0_arithmetic_format`, a small SFT task with a hidden test split. -2. **Core types**: manifest, ledger, job and artifact models outside `server/`; the reserved-name prefix rule for `jobs_`; cleanup that surfaces cancel failures (`_cleanup_session_resources` swallows errors today) and the idle reaper on when services are declared. +1. **Separate design and prototype PRs**: this PR contains only RFC 014 and its index entry; [#1500](https://github.com/openenv/OpenEnv/pull/1500) contains experimental `posttrain_env`, with no core changes and no network in agent jobs. Two T0 tasks that check the plumbing: `t0_artifact` for a fraction of a cent, and `t0_arithmetic_format`, a small SFT task with a hidden test split. +2. **Proposed core types, after prerequisites**: first validate the provider contract with a second backend and obtain native no-network support. Only then propose manifest, ledger, job and artifact models outside `server/`, any reserved-name prefix rule, cleanup that surfaces cancel failures and an idle reaper for declared services, in a separate PR. None of these core changes or names is adopted here. 3. **openenvd integration** (RFC 009): credential custody and the grader zone. ## Future Work @@ -211,6 +314,6 @@ Also: shared brokers outside the environment with per-rollout tokens ([RFC 8693] ## Related - RFC 009 openenvd (#1053, #1354, #1355): the base for credential custody and the grader zone. -- RFC 013 task sampling (#1256): `service_spend` is one source of the trainer's `costs`, never a sampling weight; an unscored episode's `None` reward is left out of the sampler's estimates. +- RFC 013 task sampling (#1256): `service_spend` is one source of the trainer's `costs`, never a sampling weight; the proposed treatment of an unscored episode's `None` reward depends on RFC 013 acceptance and its final semantics. - RFC 004: `score` feeds rubrics. RFC 005/006: LLM proxying is the future model-call profile. RFC 008: contract checks for declared services, plus #1357's adversarial panel. RFC 012: separate-verifier precedent. - Issues: #898, #1026, #1232, #1299, #1357, #779. From 2be7239eaee23ca40c1e279bb4b11701e9be242d Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Wed, 28 Oct 2026 17:17:15 +0100 Subject: [PATCH 11/11] docs: clarify RFC 014 secrets, score transport and public inputs --- rfcs/014-environment-services.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/rfcs/014-environment-services.md b/rfcs/014-environment-services.md index a65d86a8a1..44ab6ef969 100644 --- a/rfcs/014-environment-services.md +++ b/rfcs/014-environment-services.md @@ -98,9 +98,9 @@ v1 reports `usd`, `gpu_s`, `jobs` and `calls`. `jobs` counts agent jobs and `cal **Jobs.** Tools return a handle. States are `queued → running → succeeded | failed | cancelled`, plus `cancelling` while a cancel is pending and `orphaned` when the provider does not confirm one in time. An orphaned job is settled at its full reservation, keeps `final: false` and blocks the next `reset` with an error, because it may still be running and billing. `reset`, `close`, `submit`, the step limit and the time limit cancel the episode's live jobs. Submissions take an idempotency key; a retry with the same key returns the same job and makes no second reservation. -**Network.** Agent jobs run without network access. A job that can reach the internet can download a model already trained for the task, or the evaluation's own test data; [PostTrainBench](https://arxiv.org/abs/2603.08640) reports both. Inputs come as read-only mounts instead: the task's base model at a pinned revision, and the public, ungated Hub datasets the agent names, at their current revision, minus a per-task denylist. Packages are installed into a fresh venv from wheels only, which runs none of their code, before the agent's code runs. HF Jobs has no egress control, and its containers can't create network namespaces or firewall rules (checked on `cpu-basic` and `a10g-small`). So `posttrain_env` runs the agent's script inside a launcher that installs a seccomp filter refusing every socket except `AF_UNIX`, and io_uring and cross-process memory access, then closes inherited file descriptors and drops to a uid no other process in the job uses; systemd's `RestrictAddressFamilies=` works the same way. This limits a job to declared, recorded inputs, but doesn't make them clean: a public dataset or a PyPI wheel can still hold weights or test data. Tasks whose test data is public also need a contamination check after the run. +**Network.** Agent jobs run without network access. A job that can reach the internet can download a model already trained for the task, or the evaluation's own test data; [PostTrainBench](https://arxiv.org/abs/2603.08640) reports both. Inputs come as read-only mounts instead: the task's base model at a pinned revision, and the public, ungated Hub datasets the agent names, at their current revision, minus a per-task denylist. Packages are installed into a fresh venv from wheels only, which runs none of their code, before the agent's code runs. HF Jobs has no egress control, and its containers can't create network namespaces or firewall rules (checked on `cpu-basic` and `a10g-small`). So `posttrain_env` runs the agent's script inside a launcher that installs a seccomp filter refusing every socket except `AF_UNIX`, and io_uring and cross-process memory access, then closes inherited file descriptors and drops to a uid no other process in the job uses; systemd's `RestrictAddressFamilies=` works the same way. This limits a job to declared, recorded inputs, but doesn't make them clean: a public dataset or a PyPI wheel can still hold weights or test data. Tasks whose test data is public also need a contamination check after the run or an allowlist of permitted inputs. -**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward; the hidden split's seed reaches it as a secret. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The evaluator prints one `POSTTRAIN_SCORE {"score": ...}` line; a checkpoint it can't use scores 0 as `invalid`. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer, chat template and generation settings from the base model mount, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release. A score job that fails, times out, is cancelled or prints no valid score ends the episode unscored (`reward` is `None`, `episode_end.reason` is `score_failed`): the fault is the grader's, and RFC 013's `update` takes `None` for a rollout the environment did not score, which samplers leave out rather than count as zero. This `None`-reward integration depends on acceptance of RFC 013 (#1256) and must follow its final unscored-rollout semantics; it is not an independently accepted core contract. +**Evaluator.** Two endpoints. `dev_evaluate` runs on the public dev split, is rate-limited and charged to the agent, and returns at most five metrics and short feedback. `score` runs on the hidden split as a grader-only job and is the reward; the hidden split's seed reaches it as a secret. Both run the environment's evaluator script as their own job, with the artifact under test mounted read-only. The current prototype reads the evaluator's `POSTTRAIN_SCORE {"score": ...}` line from the logs; a checkpoint it can't use scores 0 as `invalid`. A planned follow-up, not yet implemented, will have the evaluator write `/data/result.json` and the broker read it separately from diagnostic logs. The evaluator takes only data from the artifact, never code or anything that shapes the evaluator's own input: `t0_arithmetic_format` loads safetensors weights into the base model's config, with the base tokenizer, chat template and generation settings from the base model mount, since a chat template could compute the answer into the prompt. Its dependencies are pinned too, so scores don't change with a new release. A score job that fails, times out, is cancelled or prints no valid score ends the episode unscored (`reward` is `None`, `episode_end.reason` is `score_failed`): the fault is the grader's, and RFC 013's `update` takes `None` for a rollout the environment did not score, which samplers leave out rather than count as zero. This `None`-reward integration depends on acceptance of RFC 013 (#1256) and must follow its final unscored-rollout semantics; it is not an independently accepted core contract. **Artifacts.** An artifact is the output folder of a succeeded job, exposed as `openenv-artifact:////outputs` with each file's path, size and content hash. Graders mount the folder read-only; nothing is copied through the agent. @@ -111,8 +111,9 @@ provider must offer these operations to the broker, without exposing its credentials to agent code: - **Submit**: accept a workload, resource choice, deadline and declared inputs; - return a stable job handle and support reconciliation of ambiguous submissions - without duplicate work or spend. + support secrets for grader jobs only, for the hidden split's seed; agent jobs + receive no secrets. Return a stable job handle and support reconciliation of + ambiguous submissions without duplicate work or spend. - **Inspect**: report state and execution duration/usage for settlement, including terminal failures and pending cancellation. - **Cancel**: request termination and allow confirmation of a terminal state;