[burtenshaw]

Summary

Restores #1248's tool/task declaration checks, rubric introspection and per-step reward attribution. Recreated after #1319; part 6 of #1177, building on #1321.

explicit unsupported discovery/telemetry -> named SKIP
legacy tools/list -32603 + exact non-MCP message -> named tool_discovery SKIP
other collection errors / malformed snapshots -> FAIL

Discovery recognizes structured UNKNOWN_TYPE, JSON-RPC METHOD_NOT_FOUND, and the verified legacy INTERNAL_ERROR with exactly Environment does not support MCP, a matching valid envelope/id and absent/null result/error details. New non-MCP servers emit METHOD_NOT_FOUND. Explicit empty declarations remain assertions; discovery failure cannot become an empty success. Rubric/attribution graders retain named session-telemetry skips. The controlled discovery-failure fixture injects a genuine internal error. Parent merges preserve network validation and #1447 cleanup.

Evidence

• Before: At exact head 478bdebb74353fa9a7bae07caf181ffb96902616, new regressions produce 2 failed, 72 passed: the legacy non-MCP reply fails an explicitly empty declaration.
  After: 451200e16a4a4c13110c98801e9ded0d2f9781c8 passes 173 focused tests, repeated against the rebuilt, installed current wheel without PYTHONPATH. Negative controls cover arbitrary internal/auth errors, near-match messages, incorrect code types/IDs/versions/envelopes, non-null results/error details, malformed telemetry and rubric exceptions.
• Ran the real legacy /ws server from main e9218f48cca2cef765a1896eba638fdb0d3f1fbb: non-MCP tools/list emits {"code":-32603,"message":"Environment does not support MCP","data":null}; validation_open emits UNKNOWN_TYPE. The latter already yields unsupported capability: session_telemetry for both rubric checks.
• Historical telemetry source 215859d9 emits Rubric introspection unavailable from broad exception handlers, including invalid/cyclic trees. It is not an explicit unsupported-capability signal. That marker, authorization refusals and malformed snapshots deliberately remain FAIL. The review thread stays open for human assessment of this ambiguity.
• Test plan executed with Python 3.12: tests/test_validation -m 'not docker and not network': 879 passed, 34 deselected; tests/core plus tests/test_core -m 'not integration and not docker and not network': 1,418 passed, 13 skipped, 15 deselected. Focused real WebSocket tests are included. Docker/network acceptance was not rerun.
• Changed-file Ruff format/import sorting, repository Ruff rules and diff whitespace checks pass. Repository formatting still reports 47 unchanged files and two unchanged import-sort failures. Standard test hook cannot find timeout; an equivalent bounded full-suite run stops at collection on missing optional smolagents. Fresh CI is separate from local evidence.

Merge Danger

Door: Two-way. Blast Radius: Grading.

Depends on #1321. Exact-head patch preserves the stacked base; main was fetched but not merged into this review layer. No public API or core-server changes in this follow-up; no new RFC is required. RFC008 remains in review. Leave open for explicit human review; do not merge. Existing approvals do not approve this head. #1323 is untouched; its parent refresh is separate.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Alignment Review Report

Automated Checks

• Lint: PASS — usort check, ruff format --check, and ruff check all pass clean on every file this PR touches (src/openenv/validation/graders/runtime/discovery.py, manifest.py, runner.py, runtime/{artifacts,collector,contracts,discovery,replay}.py, and all new/changed test files). The two pre-existing usort findings (tests/envs/test_grid_world.py, tests/envs/test_julia_env.py) and the 58 ruff format findings on unrelated markdown/env files are identical on the base commit (88232843) — none of this PR's changes.
• Debug code: CLEAN — no print(), TODO/FIXME/XXX/HACK, or debugger statements in any file this PR touches.
• Tests: tests/test_validation/ — 786 passed, 36 skipped. Full repo suite — 3777 passed, 177 skipped, 2 pre-existing failures in tests/envs/test_websockets.py (test_protocol_client_state, test_protocol_client_multiple_episodes) confirmed identical on base 88232843 — unrelated to this change (echo_env step_count tracking, outside the validation module).

Open RFCs Context

• RFC 008 (Environment Auto-Validation) — In Review, authored by @zkwentz. This PR directly implements RFC 008's "Declaration and rubric evidence (PR6)" section, which the PR itself adds to the RFC text alongside the implementation. The grader semantics (weighted_sum/sequential/gate aggregation, fail-fast on zero, gate threshold direction, weights summing to 1.0 with abs_tol=1e-6) were cross-checked against the production Rubric/Sequential/Gate/WeightedSum implementations in src/openenv/core/rubrics/containers.py and match exactly. The RubricNode wire schema this PR consumes (name, class_name, children, aggregation, config, config_available, dotted child-naming convention) matches src/openenv/core/env_server/session_telemetry.py byte-for-byte, which predates this PR (stacked on #1246/#1247 in this same chain). No conflict.
• RFC 004 (Rubrics) — In Review. Aggregation semantics match as above. No conflict.
• RFC 011 (ARD-backed catalog discovery) — Draft, authored by @thegovind. Different feature despite the shared word "discovery": RFC 011 is about environment-catalog/package discovery for selecting an unfamiliar environment, while this PR's ToolDeclarationAccuracyGrader/TaskDeclarationAccuracyGrader validate an already-selected subject's declared MCP tools/tasks against runtime reality. RFC 011 explicitly notes "Basic discovery does not depend on RFC 008 validation reports; any claim of a validated property does," so the two are anticipated to coexist. No conflict, flagging only for naming-overlap awareness.
• Other open RFCs (000–003, 005, 006, 010, 012) are unrelated to this change's surface area.

Tier 1: Fixes Required

None — lint, formatting, and debug-code checks are clean on every file this PR touches.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. This is validation/test infrastructure under src/openenv/validation/, not the core Environment/EnvClient API:

• Agent isolation / dual API boundary: tool discovery calls tools/list only (never tools/call) over the validator's own measured WebSocket session — it inspects the subject server for conformance, exactly like the existing reset/step/state replay checks already do. It does not expose reset()/step() to an agent and does not change what any agent can call.
• Client-server separation: all changes stay inside src/openenv/validation/; no cross-boundary imports.
• Rewards in environment: RewardAttributionGrader only verifies that the subject's own emitted reward matches its own rubric tree's root score — it computes nothing externally.
• No credential exposure: new code is unusually careful here — contains_credential() checks (including Unicode-escape evasion), the discovery.json artifact redaction path, and _json_float's rejection of non-finite JSON numbers are all covered by dedicated tests (test_discovery_credential_echo_is_discarded_even_with_unicode_escapes, test_discovery_artifact_redacts_tool_task_and_error_credentials, etc.).
• SSRF/network hygiene in runtime/discovery.py: trust_env=False, follow_redirects=False, explicit validation of the /list_environments namespace string (rejects ., .., /, \) before building the next request path, and byte/response-size/time budgets throughout.

RFC Conflicts

None identified beyond the naming-overlap note above (RFC 011, Draft, no implementation yet — no actual collision).

Summary

• 0 mechanical issues to fix
• 0 alignment points for human review
• 0 RFC conflicts to discuss

This is a clean, well-tested, incremental completion of RFC 008's declaration/rubric-attribution checks, consistent with the existing rubric and session-telemetry contracts it depends on.

  

_{Sent by Cursor Automation: Pre-review}

[rycerzes]

Reviewed at cad68dd3 and ran the pinned lab on Docker 29.7.2, linux/amd64. fast, protocol and docker (28/28) pass. Two comments inline.

[rycerzes]

Should fix or decide. Any tools_error/telemetry_error becomes FAIL here. A non-MCP env answers tools/list with a JSON-RPC error (on any core; WS mcp exists since 0.2.1), so declaring declared_tools: [] fails tool_declaration_accuracy with "tool discovery collection failed" (observed on maze_env). A rubric_tree: true subject on an older core fails the rubric checks the same way. Agreed that unavailable discovery mustn't become an empty passing set, but per the RFC a missing subject feature is a distinct outcome, so a named SKIP with the cause fits better than a generic FAIL. Same shape as my comment on #1321.

[rycerzes]

Nit. These report counts only. The missing names come from the author's own manifest, so listing them is safe and saves a round trip.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Unsupported discovery treated as failure
  • Legacy unsupported MCP, telemetry, and rubric-introspection responses now produce named skips while genuine collection errors still fail.

Or push these changes by commenting:

@cursor push 08a327ed87

Preview (08a327ed87)

diff --git a/src/openenv/validation/graders/runtime/discovery.py b/src/openenv/validation/graders/runtime/discovery.py
--- a/src/openenv/validation/graders/runtime/discovery.py
+++ b/src/openenv/validation/graders/runtime/discovery.py
@@ -236,7 +236,7 @@
         if type(payload["schema_version"]) is not int or payload["schema_version"] != 1:
             raise ValueError("unsupported telemetry version")
         if payload.get("rubric_error") is not None:
-            return ["subject reported unavailable rubric introspection"], None
+            return [], "unsupported capability: rubric_introspection"
         nodes = _rubric_nodes(payload["rubric"])
         return (
             ["rubric configuration is unavailable"]
@@ -250,7 +250,9 @@
     depends_on = ("runtime.startup", "runtime.rubric_introspectable")
 
     def check(self, subject, payload):
-        problems, _ = super().check(subject, payload)
+        problems, incomplete = super().check(subject, payload)
+        if incomplete:
+            return problems, incomplete
         if subject.runtime_evidence.failure_reason:
             problems.append("wire replay is incomplete")
         if problems:

diff --git a/src/openenv/validation/runtime/collector.py b/src/openenv/validation/runtime/collector.py
--- a/src/openenv/validation/runtime/collector.py
+++ b/src/openenv/validation/runtime/collector.py
@@ -258,7 +258,10 @@
                         and 16 <= len(data["capability"]) <= 256
                     ):
                         capability = data["capability"]
-                    elif code == WSErrorCode.UNKNOWN_TYPE:
+                    elif code in {
+                        WSErrorCode.UNKNOWN_TYPE,
+                        WSErrorCode.VALIDATION_ERROR,
+                    }:
                         telemetry_unavailable = (
                             "unsupported capability: session_telemetry"
                         )
@@ -313,8 +316,15 @@
                         if (
                             isinstance(error, dict)
                             and type(error.get("code")) is int
-                            and error["code"] == -32601
                             and isinstance(error.get("message"), str)
+                            and (
+                                error["code"] == -32601
+                                or (
+                                    error["code"] == -32603
+                                    and error["message"]
+                                    == "Environment does not support MCP"
+                                )
+                            )
                             and rpc.get("result") is None
                         ):
                             tools_unavailable = "unsupported capability: tool_discovery"

diff --git a/tests/test_validation/test_discovery_collection.py b/tests/test_validation/test_discovery_collection.py
--- a/tests/test_validation/test_discovery_collection.py
+++ b/tests/test_validation/test_discovery_collection.py
@@ -300,7 +300,7 @@
                     },
                 },
             },
-            "fail",
+            "skip",
         ),
         (
             {

diff --git a/tests/test_validation/test_runtime_discovery.py b/tests/test_validation/test_runtime_discovery.py
--- a/tests/test_validation/test_runtime_discovery.py
+++ b/tests/test_validation/test_runtime_discovery.py
@@ -364,7 +364,6 @@
         "nonfinite_score",
         "stale_score",
         "bad_version",
-        "rubric_error",
         "empty_child_segment",
     ],
 )
@@ -387,14 +386,22 @@
     elif change == "empty_child_segment":
         payload["rubric"][0]["children"][0] = "root."
         payload["rubric"][1]["name"] = "root."
-    else:
-        payload["rubric_error"] = "private-error"
     result = RubricIntrospectableGrader().run(subject(tmp_path, snapshot=payload))
     assert result.status is CheckStatus.FAIL
-    assert "private-error" not in result.model_dump_json()
 
 
 @pytest.mark.parametrize(
+    "grader", [RubricIntrospectableGrader, RewardAttributionGrader]
+)
+def test_unavailable_rubric_introspection_is_a_named_skip(tmp_path, grader):
+    payload = telemetry()
+    payload["rubric_error"] = "private-error"
+    result = grader().run(subject(tmp_path, snapshot=payload))
+    assert result.status is CheckStatus.SKIP
+    assert result.evidence == ["unsupported capability: rubric_introspection"]
+
+
+@pytest.mark.parametrize(
     "change",
     [
         "root_reward",

diff --git a/tests/test_validation/test_runtime_telemetry_collector.py b/tests/test_validation/test_runtime_telemetry_collector.py
--- a/tests/test_validation/test_runtime_telemetry_collector.py
+++ b/tests/test_validation/test_runtime_telemetry_collector.py
@@ -196,8 +196,12 @@
     assert TOKEN not in repr(result)
 
 
-@pytest.mark.parametrize("code", ["VALIDATION_ERROR", TOKEN])
-def test_optional_opening_refusal_does_not_fail_the_episode(monkeypatch, code):
+@pytest.mark.parametrize(
+    "code,unsupported", [("VALIDATION_ERROR", True), (TOKEN, False)]
+)
+def test_optional_opening_refusal_does_not_fail_the_episode(
+    monkeypatch, code, unsupported
+):
     result = collect(
         monkeypatch,
         opening_reply=json.dumps(
@@ -206,10 +210,17 @@
     )
     assert result.failure_reason is None
     assert result.telemetry_json is None
-    assert (
-        result.telemetry_error
-        == "session telemetry unavailable or authorization refused"
-    )
+    if unsupported:
+        assert result.telemetry_error is None
+        assert (
+            result.telemetry_unavailable == "unsupported capability: session_telemetry"
+        )
+    else:
+        assert (
+            result.telemetry_error
+            == "session telemetry unavailable or authorization refused"
+        )
+        assert result.telemetry_unavailable is None
     assert [row.operation for row in result.exchanges] == [
         "reset",
         "state",

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 478bdeb. Configure here.}