RFC 008: Level 2 implementation #1177

Track RFC 008 Level Two runtime validation, following #778 and the merged validation foundation.

Status refreshed: 2026-10-05 after the selective revert. #1319 reverted only #1246, #1247, #1248 and #1294. Their implementations are available again in the replacement PRs below for review. The replacements are open and have not been merged. #1178, #1179, #1181 and the Docker/Podman container-cleanup fix #1259 remain on main.

Current main implements four of sixteen runtime checks. Seven additional graders are implemented in the replacement stack but are pending review and merge. Five qualification graders remain unimplemented. The release decisions on reset validation, numeric step rewards and deferred credential delivery remain in main through #1181.

Delivery and dependency state

Replacement dependency order:

main → #1320 → #1321 → #1322 → #1323

These four PRs are linked as formal GitHub Stack 1324, visible in #1320's stack navigator. The revert PR is not a member.

The original PRs remain recorded as merged in GitHub; reverting their commits does not reopen them. Their replacement PRs are the review targets. Related #1259 remains merged, preserving Docker 29/Podman container-removal verification and closing #1257.

Recovery verification

The revert's exact head 6e659607 passed all 15 GitHub checks, including Linux Docker acceptance, Bugbot and pre-review, with no unresolved review threads. Fresh rollback workflow and retained evidence. Its merge commit is 3f143556; the resulting tree exactly matches the retained #1181 merge 80309e07.

Each replacement is a fresh commit based on that revert, with its source tree and incremental patch independently verified against the original slice. Fresh local pinned fast/protocol suites passed on each replacement head, with artifact hashes checked: telemetry 501/28, repeatability 604/31, discovery 753/41, cleanup 791/41. Tested replacement heads are 215859d9, 88232843, cad68dd3 and 7d84e20b, respectively. These are macOS arm64/process results, not fresh Docker evidence. New GitHub CI runs assess the replacement heads separately; these PRs remain open for human review and are not authorized for automatic merge.

Coverage and remaining scope

Implemented on post-revert main:

• runtime.startup
• runtime.reward_well_formed
• runtime.observation_schema
• runtime.state_contract

Implemented in the replacement stack, pending review and merge:

• runtime.seed_control
• runtime.episode_determinism
• runtime.trajectory_record
• runtime.tool_declaration_accuracy
• runtime.task_declaration_accuracy
• runtime.rubric_introspectable
• runtime.reward_attribution

Session telemetry supplies the evidence these seven graders consume. Optional image cleanup adds lifecycle acceptance cases, not another runtime grader. When requested and applicable, these seven checks currently produce named SKIP results on main because their graders are absent there; their implementation exists in the open replacement PRs.

Unimplemented in both main and the replacement stack:

Post-revert main therefore has twelve check IDs without graders on main: seven with implementations pending review and five without implementations. These checks produce named SKIP results when requested and applicable. Completing the replacement stack would restore eleven implemented checks, with the five remaining qualification checks still reporting SKIP when requested and applicable. Neither state provides complete Level Two conformance. Provider hardening flags and passing Docker tests do not substitute for the missing graders. --require-complete checks the selected acceptance-test inventory, not implementation of every RFC check.

Release decisions retained on main

The decisions recorded on #778 and implemented in #1181 remain in effect:

• Reset validation remains mandatory. /schema.reset_observation declares reset output separately from step output. Legacy servers without that field use the step schema; malformed reset schemas do not silently fall back.
• Level Two is stricter than the core API. Every step, including intermediate steps, must return a finite, non-boolean numeric reward in the declared range. Reset reward may be null. The core reward type remains compatible; validation never converts null to zero.
• Credential delivery is deferred for this release. validation.execution.requires_credentials: true declares a prerequisite without names or values. After plan validation, runtime checks SKIP with credential_delivery named before build/launch. Host credentials are not forwarded. Self-contained judges are not skipped solely for being LLM-judged.

#1181 also retains episode-budget request timeouts, safe schema diagnostics, collection-failure classification, native-wheel selection and malformed-envelope handling.

The real Echo canary remains a known certification incompatibility: its separately declared reset output passes observation-schema validation, but null intermediate rewards fail the Level Two reward check. Expected result: schema PASS, reward FAIL, report FAIL, CLI exit 1.

Historical evidence — not replacement-PR validation

These results belong to the published revisions tested before the selective revert. They remain reproducibility evidence for those revisions, but do not establish that post-revert main or any replacement PR has passed CI. Replacement checks, reviews and artifact verification must be assessed on their own current heads.

Counts are cumulative per branch and suites are distinct. The historical Linux amd64 bundles verify required cases and artifact hashes; local Docker Desktop arm64 and process-only HF Jobs have different scope. Recorded artifact retention ends on 2026-10-19.

For reproduction, use the recorded revision and its pinned lab instructions, acceptance inventory and implementation plan. These links describe the historical full-stack revision, not current main.

Reproducing the replacement stack

Use the new cleanup head's pinned lab and acceptance inventory. Check out the exact member head for slice-specific evidence; use Python 3.12.13 and uv 0.9.3. Run uv sync --project tests/validation_runtime --frozen, then scripts/validation/reproduce.py --suite {fast|protocol|docker} --require-complete through that project's uv run. Docker requires a running Linux engine. Verify each bundle with scripts/validation/verify_artifacts.py from the same revision.

Completion checklist

• Contracts, schemas and shared cases — Define Level 2 contracts #1178 retained
• Docker supervisor, pinned lab and Linux CI — Add Docker validation lab #1179 retained
• Basic runtime probes, scheduling, raw evidence and CLI — Run Level 2 validation probes #1181 retained
• Docker 29/Podman container-cleanup portability — fix(validation): verify container cleanup without matching engine error text #1259 retained
• Reset-schema, numeric-step-reward and deferred-credential release decisions implemented
• Selectively revert Add session telemetry #1246, Validate runtime repeatability #1247, Validate discovery and rubric attribution #1248 and Clean up validation images #1294 — Revert validation follow-ups for renewed review #1319
• Open replacement stack for review — Restore session telemetry #1320 → Restore runtime repeatability #1321 → Restore discovery validation #1322 → Restore validation image cleanup #1323
• Review and merge session telemetry — Restore session telemetry #1320
• Review and merge repeatability and trajectory recording — Restore runtime repeatability #1321
• Review and merge discovery, rubric introspection and attribution — Restore discovery validation #1322
• Review and merge optional owned-image cleanup — Restore validation image cleanup #1323
• Align applicable environments' step rewards with Level Two requirements
• Implement the five remaining qualification graders listed above
• Complete Level Two acceptance across all sixteen runtime check IDs

RFC 008 remains In Review. RFC/core alignment and RFC 011 discovery/report coordination remain separate from the accepted release decisions. Level Three semantics, statistical validation, operator certification and publication gating remain outside this delivery.