Skip to content

tri-mcp: the tri_test tool only lists a spec's tests and reports success whatever they did #8140

Open

Description

@gHashTag

What is wrong

cli/tri-mcp/src/main.rs cmd_test (the MCP tool tri_test, "Runs tests for a .t27 spec file using t27c.") runs t27c test <spec>. t27c test only LISTS the tests; it does not run them. The tool then reports "success": true from that exit code, so an agent asking tri-mcp whether a spec's tests pass is told yes even when one FAILs.

This is the same defect #7369 fixes for the tri test CLI command; #7369 records it for tri-mcp and leaves it unchanged ("cli/tri-mcp/src/main.rs cmd_test (around line 725) runs the same vacuous t27c test; noted, not changed here").

What it should do

tri_test runs t27c test-report <spec> and takes its verdict from the same rules as tri test: specs/tri/test/report.t27, through its t27c gen-rust output gen/rust/tri/test/report.rs (added in #7400). tri-mcp includes that generated module; it gets only I/O glue, no parsing or verdict logic of its own.

  • green (built, at least one test ran, none failed) -> isError: false
  • a FAIL, BLOCKED, no test, or a report that cannot be believed -> isError: true, with the verdict's words from report::why, the totals and the failing names

Boundary

Done when

  • tri_test on a spec whose tests all pass returns isError: false; on a copy with one planted failing test it returns isError: true and names the failing test
  • a stand-in t27c that only lists tests (the old behaviour) is not read as a pass
  • cargo test -p tri-mcp and cargo test -p tri pass

Stacked on #7400.

User Scenarios

  • Given a .t27 spec file whose tests all pass, When an agent calls the tri_test tool with that spec, Then the tool returns isError: false.
  • Given a .t27 spec file that has at least one failing test, When an agent calls the tri_test tool with that spec, Then the tool returns isError: true and includes the failing test name and the verdict's why text.
  • Given a .t27 spec file that has no tests, When an agent calls the tri_test tool with that spec, Then the tool returns isError: true with a message indicating no tests were run.

Requirements

  • FR-001: The cmd_test function in cli/tri-mcp/src/main.rs MUST invoke t27c test-report <spec> instead of t27c test <spec>.
  • FR-002: The tri_test tool MUST interpret the test-report output using the verdict logic from specs/tri/test/report.t27 (via the generated gen/rust/tri/test/report.rs module included in tri-mcp).
  • FR-003: When the verdict is green (built, at least one test ran, none failed), the tool MUST return isError: false.
  • FR-004: When the verdict indicates FAIL, BLOCKED, no tests, or an unbelievable report, the tool MUST return isError: true with the verdict's why text, totals, and failing test names.
  • FR-005: A test MUST exist in cli/tri-mcp/tests/ that drives the MCP server over stdio and verifies the tri_test tool behavior for both passing and failing specs.
  • FR-006: cli/tri-mcp MUST be listed in the workspace members in Cargo.toml so that cargo test -p tri-mcp runs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions