Skip to content

tri test: refuse vacuous passes -- report.t27 reads the "vacuous passes N of M" line (Closes #10497) - #10507

Merged
gHashTag merged 1 commit into
masterfrom
claude/tri-test-vacuous-10497
Oct 18, 2026
Merged

tri test: refuse vacuous passes -- report.t27 reads the "vacuous passes N of M" line (Closes #10497)#10507
gHashTag merged 1 commit into
masterfrom
claude/tri-test-vacuous-10497

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Closes #10497

tri test <spec> passed a spec whose every test executed 0 runtime asserts, because specs/tri/test/report.t27 classified t27c's vacuous passes N of M line as LINE_OTHER and cmd_test never looked at it. The rule now lives in report.t27 (FR-002) and tri test fails when N > 0 (FR-001).

What changed

  • specs/tri/test/report.t27
    • New line kinds: LINE_VACUOUS (vacuous passes N of M), LINE_VACUOUS_NAME ( 0 <name> VACUOUS), LINE_VACUOUS_UNKNOWN (vacuous passes unknown). Parsers vacuous_passes (N) and vacuous_of (M).
    • New verdicts: V_VACUOUS (N > 0) and V_UNCOUNTED (unknown: the counting build did not compile, so a vacuous pass cannot be ruled out). Only V_GREEN passes.
    • Fail closed, like the existing rules: M must equal the report's pass total and N <= M, otherwise V_TOTALS_DISAGREE; N must equal the number of VACUOUS names, otherwise V_NAMES_DISAGREE; VACUOUS names with no summary line (a report cut short) are refused.
    • A report with no vacuous line at all (a t27c older than t27c test-report: count executed asserts and flag vacuous passes (sieve C10, T730) #6509, which printed no per-test counts) keeps the reading it always had. This is said in the spec and pinned by a test.
    • A FAIL is still read before a vacuous pass, so a mutant that fails a test stays V_FAILED and tri mutate plant keeps naming it.
    • 9 new tests (13 -> 22), including the negative control only_the_vacuous_lines_refuse_a_run. The old line_kind(" vacuous passes 0 of 3") == LINE_OTHER assert now expects LINE_VACUOUS.
  • gen/rust/tri/test/report.rs: regenerated with t27c gen-rust (not hand-edited).
  • .trinity/seals/test_TriTestReport.json: resealed.
  • cli/tri/src/main.rs: glue only. report_summary appends , vacuous: <names> to the one greppable line. 3 lines changed, 3 deleted: 0 net hand-written Rust added. cmd_test itself needed no change: the generated verdict/passes/why carry the new verdicts through, and the report body it already prints holds the VACUOUS lines.

Measured results

./target/release/t27c test-report specs/tri/test/report.t27:

  tests       22
  pass        22
  FAIL        0
  vacuous passes  0 of 22  (passed with 0 runtime asserts executed)

Success criteria:

command exit result
(a) tri test specs/claw/model_route.t27 0 tests: 6, pass: 6, fail: 0, vacuous passes 0 of 6
(b) tri test on a copy of model_route.t27 whose test an_unknown_role_is_a_cheap_sub_task has its two asserts replaced by var unused : u32 = 1; 1 0 an_unknown_role_is_a_cheap_sub_task VACUOUS, vacuous passes 1 of 6, tests: 6, pass: 6, fail: 0, vacuous: an_unknown_role_is_a_cheap_sub_task, then Error: ... a test passed vacuously: it executed 0 runtime asserts, so its pass certifies nothing (T730); the VACUOUS lines name it
(c) t27c test-report specs/tri/test/report.t27 0 22/22, 0 vacuous

Also: tri test specs/tri/test/report.t27 exits 0 (22/22). A 3-test probe with two reached-no-assert tests exits 1 and prints vacuous: vacuous_one, vacuous_two.

Seal: t27c seal specs/tri/test/report.t27 --verify -> spec_hash, gen_hash_zig, gen_hash_verilog, gen_hash_c, gen_hash_rust all MATCH.

Negative controls with tri mutate plant on report.t27 itself (each KILLED by exactly the expected tests, spec sha256 unchanged after the run):

  • if (n > 0) { return V_VACUOUS; } with n > 9: killed by a_vacuous_pass_is_refused_and_named, every_vacuous_test_is_named, only_the_vacuous_lines_refuse_a_run.
  • count(text, LINE_VACUOUS_UNKNOWN) > 0 with > 9: killed by an_uncounted_report_is_refused.
  • count(text, LINE_VACUOUS_NAME) > 0 with > 9 (cut-short report): killed by only_the_vacuous_lines_refuse_a_run.

cargo test -p tri: 875 passed, 11 failed, all 11 in fpga::tests::m2l_scratch_*, piddir::tests::* and mutate::tests::* and caused by the agent sandbox (no process-table access: Operation not permitted, reaped child still reads as running; 6 mutate tests also failed on a ~/.cache/zig PermissionDenied and pass once ZIG_GLOBAL_CACHE_DIR is writable). the_summary_line_names_what_the_report_names passes. CI should be the judge of the 5 process-table ones.

Not done, on purpose

  • The issue's Boundary mentions "unless the spec opts out with a named reason". Neither the Requirements nor the Success Criteria ask for it and no spec in the sample needs it, so no opt-out exists yet: every vacuous pass fails. An opt-out would need a marker the spec carries and t27c reports; say if you want it as a follow-up.
  • vacuous passes unknown is refused (V_UNCOUNTED) rather than passed: the instrument failed, so the claim is unproved. Of 8 specs sampled (plant, gft_relu, test_report_exit, receipt, hot, lanes, ledger_union, own_language) none printed unknown.
  • No Rust unit test was added for the glue line: the foreign-line budget forbids net added lines, and the line is covered by the criteria runs above.

Possible CI note: the diff edits an existing hand-written Rust file (cli/tri/src/main.rs), within the 40/80 budget and net 0. Since 2026-10-18 that no longer needs owner-approved-foreign; if a gate asks for the label anyway, it is the owner's to apply.

🤖 Generated with Claude Code

@claude
tri test: refuse vacuous passes -- report.t27 reads the "vacuous pass…
52a0d54 
…es N of M" line (Closes #10497)

`tri test <spec>` ran `t27c test-report` and passed any report whose tests all
passed and at least one ran. It never read the line
`vacuous passes N of M  (passed with 0 runtime asserts executed)`, and
report.t27 classified it LINE_OTHER, so a spec whose every test executed 0
runtime asserts passed `tri test`.

report.t27 now reads it (the rule is .t27; the Rust is glue):
- LINE_VACUOUS (`vacuous passes N of M`), LINE_VACUOUS_NAME (`0  <name>   VACUOUS`),
  LINE_VACUOUS_UNKNOWN (`vacuous passes  unknown`), and parsers vacuous_passes (N)
  and vacuous_of (M).
- V_VACUOUS: N > 0. The VACUOUS lines name the tests.
- V_UNCOUNTED: `unknown`, the counting build did not compile, so a vacuous pass
  cannot be ruled out.
- Fail closed: M must be the report's pass total, N <= M, and N must be the number
  of VACUOUS names; VACUOUS names with no summary line (a cut report) are refused.
- A report with no vacuous line at all (a t27c older than #6509) keeps the reading
  it always had. A FAIL is still read before a vacuous pass, so `tri mutate plant`
  keeps naming the killed tests.
- 9 new tests (22 in all, 0 vacuous), including a negative control; the old
  LINE_OTHER test for the vacuous line now expects LINE_VACUOUS.

Glue: report_summary in cli/tri/src/main.rs appends `, vacuous: <names>` to the
one greppable line (3 lines changed, 0 net). gen/rust/tri/test/report.rs is
regenerated with `t27c gen-rust`; the seal is saved and verifies MATCH.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@gHashTag
gHashTag enabled auto-merge (squash) October 18, 2026 15:37
@gHashTag gHashTag mentioned this pull request Oct 18, 2026
Open
16 of 21 tasks
@github-actions

github-actions Bot commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-18 15:38:17 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 49
PRs with All Checks Green 1
READY 0
FAILING 49
PENDING 0
NO CHECKS YET 0

These columns do not partition: 0 + 49 + 0 + 0 = 49, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fad12492937a != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@gHashTag
gHashTag merged commit 96e69c4 into master Oct 18, 2026
38 of 41 checks passed
This was referenced Oct 18, 2026
Merged
Closed
Merged
gHashTag added a commit that referenced this pull request Oct 18, 2026
@gHashTag @claude
skills: tri-pipeline fast path (t27b first, frontier --only seal reus…
fb19d98 
…e) and a t27o skill (Closes #10541) (#10542)

* skills: tri-pipeline fast path (t27b first, frontier --only seal reuse) and a t27o skill (Closes #10541)

tri-pipeline: t27b test for the inner loop (rust_std.t27 0.89 s against
94 s for t27c test-report), t27c frontier --only for unchanged specs
(REUSE = the sealed verdict stands; 5 specs in about 5 s), test-report
once as the final gate, symlinked release binaries instead of rebuilds,
t27b exit codes, the sandbox ZIG cache trap, git add -f for gen files;
drop the stale "tri test does not yet fail on vacuous" (#10507 landed).

t27o: where its truth lives, making a refusal class convertible and
measuring it, --check, the loop tick for #10522, tonight's traps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* skills: the net-zero hand-line rule of own-language, and $TMPDIR over the scratchpad (Closes #10541)

#10538 failed the required own-language check with +9/-0 hand lines in
cli/tri/src/main.rs: check_budget refuses a diff whose counted foreign
lines gain more than they lose, on top of the 40/80 cap. tri-pipeline
says so under PRs; the t27o skill adds the numstat one-liner. The
scratchpad was wiped on a session resume (worktrees, cargo target,
measurements); $TMPDIR survived, so loop worktrees live there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

tri test: a spec whose every test passed vacuously still passes -- report.t27 must read the "vacuous passes N of M" line

2 participants

@gHashTag @claude