Skip to content

nightly: re-baseline differential from the scheduled run on main at 9a26f5b5 (#1302, #1304) - #1367

Merged
wshlavacek merged 2 commits into
mainfrom
nightly/rebaseline-differential-1302
Oct 28, 2026
Merged

wshlavacek merged 2 commits into
mainfrom
nightly/rebaseline-differential-1302

Conversation

@wshlavacek

Copy link
Copy Markdown
Collaborator

Re-baselines parity_checks/nightly/baselines/differential.json. Every Path-agreement run since 2026-10-27 19:00 UTC fails on one alert, BIOMD0000000338|klu_vs_dense PASS → EXCEPTION. That includes main's own scheduled night, 40712357505, and the differential PRs #1315-#1323. The change comes from main, not from any of those PRs.

The file is replaced byte for byte by the verdicts.json of scheduled run 40712357505 on main at 9a26f5b5. Re-extracting that run's report.json with verdicts.py extract gives the same bytes.

  • All 13,530 cases are on both sides, with none added or dropped.
  • Three cases change outcome, and no others do:
case before after why
BIOMD0000000338 klu_vs_dense PASS EXCEPTION #1302: the dense jacobian="fd" leg fails on Linux (below)
BIOMD0000000385 ajac_vs_fdjac TIMEOUT PASS #1321: per-check deadlines and the 1500 s cap
BIOMD0000000385 klu_vs_dense TIMEOUT PASS #1321: per-check deadlines and the 1500 s cap
  • The tally moves only there: PASS 11,225 → 11,226, EXCEPTION 7 → 8, TIMEOUT 20 → 18. DIFF 8, SKIP 2,217 and UNSUPPORTED 53 are unchanged.
  • verdicts.py diff of that night against this file gives no alerts.

BIOMD0000000338: what changed it

It changed at a commit boundary, not with the runner. The case reads PASS on every run before 2026-10-27 18:31 UTC and EXCEPTION on every run from 19:00 onward. Runs on each side used AMD EPYC 7763 and 9V45 runners, and two Xeon runs after 19:00 fail too:

run when (UTC) CPU klu_vs_dense
40521487203 (scheduled, main b543f7b) 10-27 15:23 EPYC 9V45 PASS
40537573193 10-27 18:16 EPYC 7763 PASS
40539428325 10-27 18:31 EPYC 9V74 PASS
40565116208 10-27 20:11 EPYC 9V74 EXCEPTION
40586213480 10-27 22:39 Xeon EXCEPTION
40712357505 (scheduled, main 9a26f5b) 10-28 16:24 EPYC 7763 EXCEPTION

The first failing run is #1302's. The first failing run was created at 19:00 UTC, when the only bngsim change new to main was #1302 (merged 17:31 UTC). #1321's harness change, a timeout= on every run, is not it: on macOS the run is bitwise identical with and without a timeout, for KLU, dense and the default solver.

The failing leg is the dense one.

It does not reproduce on macOS arm64. At 9a26f5b the dense leg completes, and Fg at t = 2 is 0.734, which agrees with libRoadRunner. The KLU leg ends at 311: #1304's branch split. With BNGSIM_JAC_DEBUG=1, neither leg's quotient has an entry the jump test mends.

The model is #1304's. KLU and dense take different branches on it past a step-size collapse, so a bit-level change in either quotient can move a leg. The new failure is a loud one: an exception, not an answer. Recording it here stops it alerting every night, and #1304 carries the finding.

A baseline-only change: the nightly's sweeps are not re-run for it.

…a26f5b (#1302, #1304)

verdicts.json of scheduled run 40712357505 on main at 9a26f5b, byte for
byte. Three cases move, and no others:

- BIOMD0000000338|klu_vs_dense PASS -> EXCEPTION. Since #1302, the
  forced-dense jacobian="fd" leg fails at t = 0.72 (CV_CONV_FAILURE) on
  every Linux runner; the KLU leg completes (ajac_vs_fdjac, the same KLU
  run bit for bit, passes). The model is #1304's.
- BIOMD0000000385|ajac_vs_fdjac and |klu_vs_dense TIMEOUT -> PASS. The old
  file is #1291's run, where the pool killed every row at the 900 s cap;
  with #1321's per-check deadlines and the 1500 s cap these two finish.
wshlavacek added a commit to Montana/bngsim that referenced this pull request Oct 28, 2026
)

verdicts.json of this PR's Path-agreement run 40721476951, on 18c677c
merged with main at 9a26f5b. Every row main's checks give is the same as
in main's scheduled night 40712357505 (lanl#1367 re-baselines main from it),
plus this PR's 1,353 jacobian_values rows. Against the old file:

- BIOMD0000000338|klu_vs_dense PASS -> EXCEPTION, from main (lanl#1302, lanl#1304);
- BIOMD0000000385|ajac_vs_fdjac and |klu_vs_dense TIMEOUT -> PASS (lanl#1321);
- MODEL1112050001|jacobian_values SKIP -> PASS (its base ran this time);
- MODEL1006230049|jacobian_values TIMEOUT -> SKIP, no analytic Jacobian;
- the seven models Model.jacobian() refuses stay SKIP (18c677c).
#1354 re-baselined BIOMD0000000338|klu_vs_dense to the EXCEPTION row of run 40712357505; this branch already carries that run's verdicts.json whole, so its file is kept.
@wshlavacek
wshlavacek merged commit fc5b5f6 into main Oct 28, 2026
14 checks passed
@wshlavacek
wshlavacek deleted the nightly/rebaseline-differential-1302 branch October 28, 2026 21:39
wshlavacek added a commit to Montana/bngsim that referenced this pull request Oct 28, 2026
main's differential.json (lanl#1367, lanl#1354) is run 40712357505 at 9a26f5b; this branch's is run 40721476951 at the same main. All 13,530 rows main has carry the same outcome here, and the 1,353 jacobian_values rows are this PR's own, so this branch's file is kept.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant