Repository navigation
t27b: per-test reference differential -- mismatch 0 today compares the JIT with an interpreter over the same IR #6441
Description
Problem
"mismatch 0" on the t27b lab is not a comparison with the reference path.
- In
cli/t27b/src/main.rs(the--checkblock), MISMATCH means the JIT disagrees witheval.rs::Interp. Both read the same IR fromlower.rs. Iflower.rsmistranslates t27 semantics (wrap vs trap, promotion, float rounding, layout), both sides compute the same wrong thing and agree. - The only comparison with the reference is per file.
blockers.rs::parse_test_reportkeeps only thetestsandFAILtotals oft27c test-report, even though t27c already prints each test's outcome (bootstrap/src/test_report.rs,Report.outcomes). - steward.t27
is_alarm(fail, fail)is false, so the 7 files where both fail (lab run 03362c1) count as agreement without checking that the same test failed. Example:specs/port/trinity/src/tri/gen_vector2d.t27: t27b failsVec2_lengthwith assert_eq left 4627730092099942521 vs right 4627730092099895296 (one f64 ulp). The reference says only "1 of 4 tests fail". Stop::FuelandStop::Depthare also counted as agreement whatever the JIT did.
Change
parse_test_reportreturns the per-test outcome list. The reference cache stores it (new row format; old rows are read as file-level only).t27b corpus --referencecompares test by test. A test whose t27b verdict differs from the reference verdict isref_mismatch, a new verdict counted insummary.ref_mismatchand treated by the doctor asLAB-MISMATCH.- steward.t27
is_alarmjudges per test when per-test data exists. - Interpreter Fuel/Depth stops report
unknown, not agreement.
Done when
- The lab JSON carries
ref_mismatchper file and in the summary, and the 7 fail/fail files show which tests disagree, if any. - steward.t27 has tests for the per-test
is_alarm. - New subcommand
tri t27b diff <spec>prints both verdict lists side by side.
🤖 Generated with Claude Code
Activity
gHashTag commentedon Oct 5, 2026
First measured
reference_disagreeon master, from brancht27b-refdiff-20261005T1500(de50cbd = master 36ad787 plus this change). Run by hand on the Railway t27b lab container at 2026-10-05 15:39-15:47Z:qemu-aarch64 t27b corpus specs --blockers --reference t27c --jobs 12 --timeout-ms 60000. Corpus exit 0.reference_disagree0 files, 0 tests, of 668 files compared test by test. That is 2466 tests per side, and both sides have the same test names.jit_interp_mismatch(oldmismatch) 0.gen_vector2d.t27failsVec2_lengthon both, so the file-level agreement was right this time. The others aretiming_tb,build_verify,gf16_dot4,check-first-party-doc-language,gen_w392andsearch/pattern.specs/ui/viewport.t27. t27b failsrail_capacity_follows_the_css_budget, but the reference isblocked, so there is no reference verdict to compare it with.No disagreement was found, so no root-cause issues were filed.
The branch is pushed and has no PR. The owner's rule from 2026-10-05 ~15:45Z forbids hand-written non-t27 code in new PRs. The branch contains the following:
is_alarm_testsandlanes_stopinspecs/tri/t27b/steward.t27.gen/c/tri/t27b/steward.cwas regenerated, andtri t27b gen-checkon the t27c lab says SAME 904af7fc1f22.cli/t27b/src/{blockers,main}.rsand tests) and Python (lab.py,t27b.py,t27b_rules.py, CI tests).Refs #6063
feat(t27b): per-test reference verdicts, reference_disagree apart fro…
theory(sieve): re-read the t27c and t27b cells after T741's fix issue…