Repository navigation
duplicate-bodies: share tree helpers across eight gen_w bench ports #9633
Description
Title: duplicate-bodies: the eight gen_w bench ports copy indent, build_level, flat_index and is_ascii; write them once
Summary
Master (cd8a656) fails python3 tools/dupe_scan.py with exit 1 (12 violations; master CI run 38757535577 prints the same list).
Four of the groups come from eight specs/port/scripts/gen_w*.t27 bench ports that each carry the same helper block: indent, build_level, flat_index, is_ascii
(gen_w698 spells indent as a table; its outputs are the same). AGENTS.md "DRY" forbids new duplicate bodies, and the ledger only moves down.
The same fix was already made for gen_w310..gen_w322 (specs/port/scripts/gen_w_blocks.t27).
This issue covers only the gen_w part. The trinity_block STEP1/STEP2 groups are tracked in #9631 and are not touched here; I am not taking #9631.
The rust_convert and ratchet work is tracked in #9632 and #9600 and is not touched here.
Boundary
Exactly these eleven paths and no others:
specs/port/scripts/gen_w_tree.t27
specs/port/scripts/gen_w698.t27
specs/port/scripts/gen_w709.t27
specs/port/scripts/gen_w713.t27
specs/port/scripts/gen_w716.t27
specs/port/scripts/gen_w721.t27
specs/port/scripts/gen_w728.t27
specs/port/scripts/gen_w729.t27
specs/port/scripts/gen_w732.t27
tools/duplicate_bodies_baseline.txt
docs/reports/suite_expectations.json
gen_w_tree.t27 is new. The baseline change is two removed lines (flat_index 2, is_ascii 2) and nothing else.
The suite expectations change is a CONDITIONAL CANDIDATE, not a proven improvement. It removes exactly one entry, specs/port/scripts/gen_w698.t27 phase gen-verilog, and lowers max_entries from 83 to 82.
What was observed: the corpus ratchet of the first commit reported an unexpected pass (job 116524835401 of run 38764724462 on PR #9636: ledger 83 of 83, UNEXPECTED FAILURES 0, UNEXPECTED PASSES 1:
specs/port/scripts/gen_w698.t27 [gen-verilog], "fixed -- remove from ledger"). The entry recorded W469 (a table of strings in indent, issue #1959), and the table is gone.
What it does NOT prove: see "Blocked" below. gen-verilog does not carry imported function bodies, so the spec generates with its helpers silently missing. The entry may be removed for real only after #8769 is fixed
and the helper definitions are shown in the generated Verilog. No other entry, discard cap or expiry changes.
No gen/ file, no seal, no workflow, no Rust, no foreign-exceptions entry, no port file of #9622 or #9630.
User Scenarios
Scenario 1: a maintainer runs the duplicate gate after the change
- Given master plus this change (and ci: remove copied helpers from STEP1/STEP2 t27 ports #9631 not yet merged)
- When
python3 tools/dupe_scan.pyruns - Then no line names
build_level,indent,flat_indexoris_ascii; the STEP1/STEP2 groups of ci: remove copied helpers from STEP1/STEP2 t27 ports #9631 may still be listed, so the full scanner is still red until ci: remove copied helpers from STEP1/STEP2 t27 ports #9631 is integrated.
Scenario 2: a gen_w port still behaves the same
- Given any one of the eight ports
- When
t27c test-reportruns on it - Then it reports the same 13 tests as on master, all passing, with 0 vacuous, and the helper functions resolve through
use.
Scenario 3: the shared helpers have their own tests
- Given specs/port/scripts/gen_w_tree.t27
- When it is tested by the reference and natively
- Then 7 of 7 tests pass in both, with runtime asserts.
Scenario 5: the Verilog output keeps the helpers (blocked by #8769)
- Given gen_w698 and a base port such as gen_w709, both using or defining the helpers
- When
t27c gen-verilogruns on each - Then both outputs define
function indent,elem_x,elem_y,build_levelandbuild_tree. On the base gen_w709 they exist; on gen_w698 after the move touseonlyfunction generateremains, so this scenario FAILS today.
Scenario 4: the full gate after #9631
- Given this change and the ci: remove copied helpers from STEP1/STEP2 t27 ports #9631 change both integrated on master
- When
python3 tools/dupe_scan.pyruns - Then it exits 0 with no new and no grown group.
Requirements
- FR-001: The gen_w tree helpers (
TreeBuild,indent,elem_x,elem_y,build_level,build_tree,flat_index,is_ascii) MUST exist once, in specs/port/scripts/gen_w_tree.t27, with tests. - FR-002: Each of the eight ports MUST drop its local copies of those items and MUST
use port::scripts::gen_w_tree;. - FR-003: Each port's header constants,
generate()and every existing test MUST be left unchanged. - FR-004: The behavior of every helper MUST be unchanged;
indentMUST give the same string for every level, including levels at or below 0 and above 7. - FR-005: tools/duplicate_bodies_baseline.txt MUST only move down: exactly the two lines
flat_index 2andis_ascii 2are removed. - FR-006: The change MUST add no foreign code and MUST NOT edit a generated file by hand.
- FR-007: The change MUST NOT touch the STEP1/STEP2 specs, rust_convert.t27, the t27b ledger or any workflow.
- FR-008: docs/reports/suite_expectations.json MAY lose exactly the entry for specs/port/scripts/gen_w698.t27 (gen-verilog) with
max_entriesgoing from 83 to 82 only as a conditional candidate; it MUST NOT be treated as justified until gen-verilog: a function brought in byuseis called but never defined, and gen-verilog exits 0 #8769 is fixed and FR-009 holds. No other entry, cap or expiry MUST change and no broad bless MUST be run. - FR-009: The Verilog generated for every touched port MUST still define every helper function it defined before the move (
indent,elem_x,elem_y,build_level,build_tree, andflat_index/is_asciiwhere the backend emitted them). An exit code 0 of any gate MUST NOT be read as proof of this.
Success Criteria
Run from the repository root with the repository's t27c.
./target/release/t27c test-report specs/port/scripts/gen_w_tree.t27 --specs-dir specsexits 0 and reports 7 of 7 tests../target/release/t27c test-report specs/port/scripts/gen_w698.t27 --specs-dir specsexits 0 and reports 13 of 13 tests with 0 vacuous; the same command for gen_w709, gen_w713, gen_w716, gen_w721, gen_w728, gen_w729 and gen_w732 (verified together as 104 of 104 on the prepared branch)../target/release/t27b test specs/port/scripts/gen_w_tree.t27 --checkexits 0 (7 of 7, 33 asserts on the prepared branch). The eight ports stayblocked ConstDeclnatively, as they were in the master artifact; this issue does not claim they are native green.python3 tools/dupe_scan.py --self-testexits 0.! python3 tools/dupe_scan.py 2>&1 | grep -E 'build_level|indent|flat_index|is_ascii'exits 0 (none of the four names remains).git diff --numstat origin/master -- tools/duplicate_bodies_baseline.txtshows 0 added lines and 2 deleted lines.git diff --numstat origin/master -- docs/reports/suite_expectations.jsonshows 1 added line and 8 deleted lines (themax_entriesline and the seven-line gen_w698 entry).- Conditional on gen-verilog: a function brought in by
useis called but never defined, and gen-verilog exits 0 #8769: the gen-verilog output of specs/port/scripts/gen_w698.t27 containsfunction indent,function elem_x,function elem_y,function build_levelandfunction build_tree(checked on the generated text, as for the base gen_w709, whose output has them). - Conditional on that:
grep -n '"max_entries": 82' docs/reports/suite_expectations.jsonexits 0 and the corpus ratchet job reports UNEXPECTED PASSES 0. These two are not proof of FR-009. - After ci: remove copied helpers from STEP1/STEP2 t27 ports #9631 is integrated:
python3 tools/dupe_scan.pyexits 0.
Blocked: #8769 (gen-verilog drops imported helper functions)
Found on the prepared change: gen-verilog imports enum and struct types from a used spec but not functions. Evidence (Codex, same base t27c):
- /tmp/t27-ci-gen-w709-base.v, gen-verilog of the unchanged base gen_w709: defines
function indent,elem_x,elem_y,build_level,build_treeand the rest. - /tmp/t27-ci-gen-w698-after.v, gen-verilog of gen_w698 after its helpers moved behind
use: onlyfunction generate; every helper definition is gone, and the test names appear only as comments.
So for the Verilog backend this move loses code, even though the reference (Zig) and Rust targets andt27c test-reportpass. The suite ratchet'sUNEXPECTED PASSfor gen_w698 is a symptom of that loss, not evidence that W469 is fixed;
the corpus ratchet check does not catch missing definitions (clean ratchet, UNEXPECTED FAIL 0 and PASS 0 after the expectation is removed). Exit 0 does not prove preservation.
The root is the existing issue gen-verilog: a function brought in byuseis called but never defined, and gen-verilog exits 0 #8769. Until it is fixed this change MUST NOT merge, even with every check green. Other lanes' work on gen-verilog: a function brought in byuseis called but never defined, and gen-verilog exits 0 #8769 is not touched here.
Not green until
The full duplicate-bodies check stays red on any branch that has only this change, because the STEP1/STEP2 groups (#9631) remain. PRs #9622 and #9630 cannot be called green from this issue alone.
Dmitrii Fedorov (@dmitrii-f-t27)
Activity
test(corpus): retire fixed gen_w698 Verilog expectation (Refs gHashTa…
ci(dupe): turn duplicate-bodies green -- share gen_w tree helpers, re…