tri-mcp: the tri_test tool only lists a spec's tests and reports success whatever they did #8140
Description
What is wrong
cli/tri-mcp/src/main.rs cmd_test (the MCP tool tri_test, "Runs tests for a .t27 spec file using t27c.") runs t27c test <spec>. t27c test only LISTS the tests; it does not run them. The tool then reports "success": true from that exit code, so an agent asking tri-mcp whether a spec's tests pass is told yes even when one FAILs.
This is the same defect #7369 fixes for the tri test CLI command; #7369 records it for tri-mcp and leaves it unchanged ("cli/tri-mcp/src/main.rs cmd_test (around line 725) runs the same vacuous t27c test; noted, not changed here").
What it should do
tri_test runs t27c test-report <spec> and takes its verdict from the same rules as tri test: specs/tri/test/report.t27, through its t27c gen-rust output gen/rust/tri/test/report.rs (added in #7400). tri-mcp includes that generated module; it gets only I/O glue, no parsing or verdict logic of its own.
- green (built, at least one test ran, none failed) ->
isError: false - a FAIL, BLOCKED, no test, or a report that cannot be believed ->
isError: true, with the verdict's words fromreport::why, the totals and the failing names
Boundary
cli/tri-mcp/src/main.rs(cmd_test, thetri_testtool entry)cli/tri-mcp/tests/(a test that drives the server over stdio)Cargo.toml(cli/tri-mcpinto workspacemembers, socargo test -p tri-mcpexists; see cli/tri-mcp is in neither workspace members nor exclude -- the same defect Cargo.toml already documents for ffi #5395)specs/tools/mcp/tri-mcp.t27(the tool card quotes the tool's description)
Done when
tri_teston a spec whose tests all pass returnsisError: false; on a copy with one planted failing test it returnsisError: trueand names the failing test- a stand-in t27c that only lists tests (the old behaviour) is not read as a pass
cargo test -p tri-mcpandcargo test -p tripass
Stacked on #7400.
User Scenarios
- Given a .t27 spec file whose tests all pass, When an agent calls the
tri_testtool with that spec, Then the tool returnsisError: false. - Given a .t27 spec file that has at least one failing test, When an agent calls the
tri_testtool with that spec, Then the tool returnsisError: trueand includes the failing test name and the verdict'swhytext. - Given a .t27 spec file that has no tests, When an agent calls the
tri_testtool with that spec, Then the tool returnsisError: truewith a message indicating no tests were run.
Requirements
- FR-001: The
cmd_testfunction incli/tri-mcp/src/main.rsMUST invoket27c test-report <spec>instead oft27c test <spec>. - FR-002: The
tri_testtool MUST interpret the test-report output using the verdict logic fromspecs/tri/test/report.t27(via the generatedgen/rust/tri/test/report.rsmodule included in tri-mcp). - FR-003: When the verdict is green (built, at least one test ran, none failed), the tool MUST return
isError: false. - FR-004: When the verdict indicates FAIL, BLOCKED, no tests, or an unbelievable report, the tool MUST return
isError: truewith the verdict'swhytext, totals, and failing test names. - FR-005: A test MUST exist in
cli/tri-mcp/tests/that drives the MCP server over stdio and verifies thetri_testtool behavior for both passing and failing specs. - FR-006:
cli/tri-mcpMUST be listed in the workspacemembersinCargo.tomlso thatcargo test -p tri-mcpruns.
Activity