[burtenshaw]

Summary

Restores #1247's seed checks, fresh-session and fresh-container replay, and subject-emitted trajectory validation. Recreated after #1319; part 5 of #1177, building on #1320.

explicit UNKNOWN_TYPE → named session_telemetry SKIP; refusal/malformed reply → FAIL

Unsupported telemetry is distinct from collection errors in evidence and reproduction artifacts, including changed-seed replays. Successful seed-control evidence says seeds were observed as forwarded to reset; empirical determinism is a separate finding. Merges the updated #1320 parent without rebasing, retaining network checks and #1447 cleanup.

Evidence

• Before: Unsupported telemetry is conflated with authorization refusal; new collector/grader/replay regressions fail.
  After: 104 focused tests pass; a subsequent 98-test run also passes the explicit forwarding-wording assertion. The clean Python 3.12 validation run at 07a867eff263e647535b1a9901c8071df7b65e44 passes 679 tests (8 skipped; 26 Docker/network tests deselected), using its current-source installed wheel. Final head b9a5e080f6ce4ad65a6c76dd65da02f4b49335ba changes only the README acceptance count; all test and production sources are identical.
• Changed-file Ruff/import-sort checks and diff whitespace checks pass. Repository-wide formatting still flags 49 files unchanged from the original Restore session telemetry #1320 head. The standard full-suite hook lacks timeout on this host; an equivalent bounded run stops at collection because optional smolagents is absent. Docker/network acceptance was not rerun. Fresh GitHub checks are separate from this local evidence.

Merge Danger

Door: Two-way. Blast Radius: Replay.

Depends on #1320. RFC008 remains in review. Leave open for human review; do not merge. No existing approval is treated as approval of this new head.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Alignment Review Report

Automated Checks

• Lint: PASS — usort check src/ tests/, ruff format src/ tests/ envs/ --check, and ruff check src/ tests/ all pass for every file this PR touches (graders/runtime/repeatability.py, runtime/replay.py, runtime/artifacts.py, runtime/contracts.py, providers/docker.py, runner.py, types.py, and the test/fixture files). The only findings are pre-existing and unrelated to this diff: usort flags tests/envs/test_grid_world.py and tests/envs/test_julia_env.py (documented as pre-existing in AGENTS.md), and ruff format flags ~58 files under envs/ plus src/openenv/core/harness/README.md — none are part of this PR's changed-file set.
• Debug code: CLEAN — no print/breakpoint/debugger statements or new TODO/FIXME were introduced by this diff. check-debug.sh's whole-repo scan still surfaces older print/TODO lines elsewhere, but all are in files untouched by this PR.
• Tests: Ran the targeted suites (test_runtime_repeatability.py, test_runtime_replay.py, test_runtime_artifacts.py, test_runtime_execution.py, integration/test_runtime_cli.py, integration/test_runtime_process.py): 166 passed, 16 skipped (skips are the expected Docker-only cases in this non-Docker review environment). Full suite: 3618 passed, 2 failed, 169 skipped — the 2 failures are in tests/envs/test_websockets.py (echo_env state step_count assertions); confirmed via git diff that these files are untouched by this PR and the failures reproduce in isolation, so they're pre-existing/environmental, not caused by this change.

Open RFCs Context

• RFC 008 (Environment Auto-Validation) — Status: In Review, author @zkwentz. This PR is squarely in RFC 008's territory: it implements the "Runtime replay evidence (PR5)" slice (seed-control, episode-determinism, and trajectory-record graders; the fresh-session/fresh-container replay collector; replays.json artifact) that follows PR4's session-telemetry protocol (base branch of this PR). The PR also edits rfcs/008-environment-auto-validation.md itself to replace the "future protocol slice must authorize access..." hedge with "The protocol authorizes access..." and to flesh out the previously-stubbed "Later runtime evidence contracts" section into concrete, shipped behavior (300s/32MiB budgets, container-identity invariants, null-reward handling, judged-sample counts). I cross-checked every claim added to the RFC text against the implementation (budgets in replay.py, container/image identity checks, the empty volatile-exclusion set, null-vs-terminal reward handling in repeatability.py, replays.json contents in artifacts.py) and found them faithfully matched — this reads as planned, sequential RFC execution rather than a surprise deviation.
• Other Draft/In-Review RFCs (000, 001, 002, 003, 005, 006, 010, 011, 012) do not appear touched by this diff — this PR stays inside src/openenv/validation/ and its tests, with no changes to the Gym-like API, MCP surface, client/server boundary, or reward computation.

Tier 1: Fixes Required

None. No mechanical/lint/debug issues found in the diff.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. The new code is validation/grading logic that reads already-computed rewards and telemetry to check determinism and tolerance bounds declared in the manifest — it does not compute rewards itself, so "rewards inside environment" is not implicated. Session telemetry stays orchestrator-only, is carried over the same validation WebSocket connection, and is never exposed as an MCP tool, consistent with the "agent isolation" invariant. No client code imports server internals or vice versa.

RFC Conflicts

ALIGNMENT FLAG: PR amends RFC 008's own prose while the RFC is still "In Review"

• Principle/RFC at stake: RFC 008 (Environment Auto-Validation), status In Review
• The concern: This PR changes RFC 008's text from forward-looking/hedged language ("A future protocol slice must authorize access...") to present-tense, implemented-fact language, and expands the "Later runtime evidence contracts" section into fully specified behavior. The implementation matches the new text in every detail I checked, so this isn't a technical conflict — but since the RFC is still formally "In Review" under a different primary author, it's worth the RFC owner explicitly signing off that this PR's wording is the intended canonical description of PR5 (and that no further changes to this contract are expected before PR6+).
• Suggested reviewer: @zkwentz (RFC 008 author)

Summary

• 0 mechanical issues to fix
• 1 alignment point for human review (RFC-text sign-off, not a code concern)
• 0 RFC conflicts to discuss (implementation matches the RFC text it adds)

  

_{Sent by Cursor Automation: Pre-review}

[rycerzes]

Reviewed at 88232843 and ran the pinned lab (Python 3.12.13, uv 0.9.3, --require-complete) on Docker 29.7.2, linux/amd64. fast, protocol and docker (20/20) pass.

I also validated maze_env built from its own Dockerfile with this stack. state_contract and episode_determinism correctly caught maze ignoring episode_id. One comment inline about how subjects on an older core are graded.

[rycerzes]

Should fix or decide. A subject on an openenv-core without telemetry answers validation_open with an UNKNOWN_TYPE error envelope. The collector (from #1320) records any non-terminal reply as telemetry_error = "session telemetry unavailable or authorization refused", and this line turns that into FAIL. No released openenv has telemetry, and real env locks pin 0.3.1/0.4.1, so every env built from its own Dockerfile fails seed_control and trajectory_record (observed on maze_env). The RFC says missing subject features are distinct outcomes, so classifying UNKNOWN_TYPE as unsupported and SKIPping with the cause ("subject openenv-core does not implement session telemetry") seems right, while keeping FAIL for refusals and malformed replies.

[rycerzes]

Note. This checks that the seed was forwarded, not that it changes behaviour. An env that accepts seed but ignores it (finqa before #1252) passes here and is caught by episode_determinism instead. That's fine, but authors may read a PASS here as "seeding works"; the evidence text could say "forwarded".