research(tnf): cost-sweep refit verdicts outrun their data (PIPELINE confound, 1-dof fit) #898

What

Run 37266625622 is the first cost sweep in which every arm placed and routed (5 of 5, after #896 / #897). Its refit.md draws four conclusions that are stronger than the data behind them. The numbers are fine; the verdict text and the comparisons are not. research/arxiv_tnf/refit_cost_model.py and fpga/openxc7-synth/gen/gen_tnf_cost_sweep.py are research generators, so every fix below is the owner's call. This issue records the defects so the paper does not pick up the verdicts as they stand.

1. The two rows past M = 25 are a different circuit

gen_tnf_cost_sweep.py:97 sets .PIPELINE(1) on gf_adder_param when M >= 25, and PIPELINE=0 otherwise.

• Fmax. With PIPELINE=0 the adder runs from input pads to the output register, which is not a register-to-register path, so nextpnr's clock Fmax never times it. For e4m9 the reported path is 3 cells of out_valid control logic. With PIPELINE=1 the reported path is the stage-2 datapath (50 cells for e3m33). The first three Fmax values are not adder measurements, and the column cannot be compared across the M = 25 boundary.
• LUT. The rows where Q2 looks for flattening are exactly the two pipelined ones. Whether the pipeline split changes the LUT count is not measured. It is a confound, not a known effect.

2. The Q2 fit has one residual degree of freedom, and one coefficient is a single row

n = 5, p = 4. Only e3m33 has E_t = 3, so the E_t column acts as an indicator for that one row: its residual is exactly 0 in both the quadratic and the linear fit (checked by refitting the five rows from refit.md with the same lstsq). In practice the quadratic is a three-parameter curve through four E_t = 4 points. R^2 = 0.9997 and "better by adjusted R^2: quadratic" come out of a fit with 1 residual degree of freedom, so they carry no evidential weight.

Leave-one-out with the same model: dropping e4m29 predicts 1822 LUT against the measured 1634 (−188, 11.5 %). Dropping e3m33 makes the design matrix rank-deficient (rank 3).

The script already refuses to fit the M <= 25 subset at n = 3. The all-rows fit has no equivalent guard.

3. The residual verdict has no minimum n and mixes regions

"The published coefficients do not reproduce the new rows within 15 % … superseded for this region" fired on 5 rows. The worst error (141.3 %) is e3m33, at M = 33. The paper's own sentence says the published model is fitted on M <= 25 and "is not extrapolated", so a residual at M = 29 and M = 33 tests a use the paper already rules out. Inside the published range the misfits are still large: e4m9 +299 (+56 %), e4m11 +177 (+29 %), e4m21 −511 (−44 %). Which flow produced the published rows is not stated in the paragraph I read (tnf_paper.tex around line 6434), and I did not trace it. If it was a different flow, these residuals compare flows rather than models.

4. Q1's wording claims a cause

Q1 is answered: e4m9 536 LUT, e4m11 610 LUT, +74, monotone in this flow. e4m9 gives the same LUT, FF and Fmax in runs 37265666318 and 37266625622; its nextpnr logs differ only in wall-clock fields. That is one seed, so it shows determinism, not spread. The script then prints "the earlier inconsistency was a flow artefact. The paper must be corrected". That names a cause the sweep did not test. It also prints "Both arms did not route" whenever either arm is missing from the routed rows, including when the arm was not run at all.

Proposed, for the owner to choose

• Register the inputs of the PIPELINE=0 arms, or report Fmax only for PIPELINE=1 arms, so the column measures one thing.
• Add a pipeline column to the refit rows and fit within one value of it.
• Guard the all-rows fit on residual degrees of freedom, and give the residual verdict the same minimum-n guard the M <= 25 fit already has.
• Q1: separate "not run" from "did not route", and say "monotone in this flow" instead of naming a cause.
• Several seeds per arm before any coefficient goes into the paper.

Not measured

• Seed-to-seed spread: every arm used heap/router1/seed1.
• Whether PIPELINE=1 changes the LUT count at a fixed M. A matched pair (one M run both ways) would settle it.
• The flow behind the published rows.
• Anything on the board.

🤖 Generated with Claude Code