Skip to content

RFC 014: Environment Services (design only) - #1508

Merged
burtenshaw merged 12 commits into
openenv:mainfrom
burtenshaw:ben/rfc014-services-review
Oct 28, 2026
Merged

burtenshaw merged 12 commits into
openenv:mainfrom
burtenshaw:ben/rfc014-services-review

Conversation

@burtenshaw

@burtenshaw burtenshaw commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Separates @rycerzes's RFC 014 design from the experimental implementation in #1500, as requested in #1500 (review). Discussion and origin: #1382.

rfcs/
  014-environment-services.md  design, still In Review
  README.md                    index entry

The nine original RFC revisions are replayed with their author/date preserved and original commit SHAs recorded in each commit. The author credit and prototype results remain. The prototype link points to #1500.

Adds the proposed submit/inspect/cancel/logs/price/artifacts provider contract. A second backend and native no-network support are prerequisites before step 2; Docker and Modal remain proposals. Core manifest types and reserved jobs_ names are deferred. Unscored None rewards depend on RFC 013 acceptance and its final semantics. Examples use step(CallToolAction(...)); direct-MCP episode enforcement is tracked in #1507.

Evidence

  • Before: RFC and implementation shared Add experimental posttrain_env prototype (RFC 014) #1500; examples used direct MCP calls.
    After: diff against main contains exactly the two RFC files; example executed with fake HF Jobs through server serialization/client parsing and returned final reward/spend. No paid jobs.
  • Test plan: whitespace checks and RFC example execution pass. The env validation is reported on Add experimental posttrain_env prototype (RFC 014) #1500. Full local pytest selection: 3958 passed, 150 skipped, 79 deselected, 3 failed (259 s). Failures are in unchanged main files: two harness-process tests raised macOS os.killpg PermissionError during the full run, but passed in isolation on main; the Harbor occupied-port test failed identically on main (expected "already in use", observed startup timeout). These results do not establish a regression from this change. Full lint has 47 pre-existing main formatting findings plus the two known usort files; changed files pass. The test hook itself lacks timeout on this host, so the same pytest selection ran directly. CI was pending at the post-push check; no monitoring requested. Repository formatting has existing failures on main; no Python or runtime files change here.

Merge Danger

Door: two-way

Documentation only; this PR does not mark the RFC accepted or adopt core APIs.

Blast Radius: design

Provider contract, second-backend validation and RFC 013 semantics remain review decisions.

rycerzes and others added 10 commits October 27, 2026 17:47
RFC-only extraction from original commit 637310b in openenv#1500.
RFC-only extraction from original commit 0bd7991 in openenv#1500.
…tall CPU-only torch in its evaluator

RFC-only extraction from original commit 434cbbd in openenv#1500.
RFC-only extraction from original commit c32dbb2 in openenv#1500.
RFC-only extraction from original commit 0dda477 in openenv#1500.
…view notes

Ending an episode on the WebSocket path waited for HF Jobs to confirm cancels
on the shared event loop. Run it in a thread instead. The evaluator now rejects
checkpoints with unexpected keys, and RFC 014 links to openenv/OpenEnv and
describes trainer staleness without naming one framework.

RFC-only extraction from original commit eb21148 in openenv#1500.
@rycerzes

Copy link
Copy Markdown
Contributor

Thanks @burtenshaw! This covers everything @sergiopaniego asked for in #1500. Three small things:

  1. Submit should require secrets for grader jobs only. The hidden split's seed depends on it.
  2. The Evaluator section still reads the score from the logs, but Logs says the result comes separately. One line on the planned /data/result.json would fix it.
  3. For public data, Sergio suggested "a contamination check or an allowlist". Only the check is in the RFC.

@burtenshaw

Copy link
Copy Markdown
Collaborator Author

@rycerzes Addressed the three wording fixes from #1508 (comment) in 2be7239:

  1. Submit now requires support for secrets for grader jobs only, for the hidden split's seed; agent jobs receive no secrets.
  2. Evaluator explicitly says the current prototype reads POSTTRAIN_SCORE from logs. Writing /data/result.json and reading it separately from diagnostic logs is a planned follow-up, not implemented.
  3. Network now requires a contamination check after the run or an allowlist of permitted inputs for tasks with public test data.

Only rfcs/014-environment-services.md changed. RFC status remains In Review; attribution, history, prototype results, provider neutrality, second-backend/native no-network prerequisites and the RFC 013 caveat are preserved. No environment or core-type changes.

Validation: diff whitespace, Markdown structure/format, unchanged links/local anchors and single-file scope checks pass. No runtime tests run for these wording-only edits. At the post-push snapshot, no checks or reviews were reported; CI/review remain pending. No monitoring requested.

@burtenshaw
burtenshaw merged commit aca76d9 into openenv:main Oct 28, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants