Skip to content

differential: compiled SSA propensities against the interpreter (#1310) - #1322

Open
Montana wants to merge 11 commits into
lanl:mainfrom
Montana:fix/1310-ssa-codegen
Open

Montana wants to merge 11 commits into
lanl:mainfrom
Montana:fix/1310-ssa-codegen

Conversation

@Montana

@Montana Montana commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1310.

Stacked on #1321. The first nightly here failed Path-agreement on TIMEOUTs only, at the cap on MODEL1006230049, 0053 and 0077, which any added check tipped over. #1321 bounds each check by the model's cap, keeps a killed worker's finished rows, and raises the job's cap to 1500 s. It is merged in here; until it merges, this PR also shows its commits.

Builds on #1291 (merged). It adds two rows to CHECKS, as #1315, #1316, #1318, #1319 and #1320 each add theirs, so whichever lands second keeps both. I'll resolve it.

What it adds

An SSA arm of the differential report: compiled SSA propensities against interpreted ones.

check path reference
ssa_cc_vs_interp the default Simulator, which compiles the propensities of an eligible model codegen=False
ssa_backend_as_asked ssa_diagnostics["propensity_backend"] what was asked: codegen=False interpreted, the default compiled (#1213's class)

codegen=True is refused for SSA, so the compiled side is the default Simulator. Each job runs n_rep replicates at seeds seed up to seed + n_rep − 1:

  • every seed identical: PASS;
  • an all-mass-action model, any seed differing: DIFF. Both sides compute the same doubles, so the event sequence must match draw for draw;
  • otherwise: judged in distribution, by differ.ensemble_verdict at ensemble_k_for(n_rep), as the SSA parity suites judge. That branch is for when eligibility widens to functional laws; today none is compiled.

Cost, which shaped the design. Compiled propensities serve exact SSA only on mass-action models without events, up to a size (GH #190). The Simulator decides that before any run, and builds no propensity library for a model it finds ineligible. So:

  • no library: both checks SKIP, and no replicate runs. On a first design that ran ensembles both ways, 31 of the first 98 models timed out at 600 s; even one replicate each way, 18 of 101 ran past the 900 s cap;
  • a library the engine still declines: one replicate each way decides both checks;
  • otherwise: the full ensembles.

Job gains method, n_rep and seed; checks_for(job) gives an SSA job the SSA checks only, and an ODE job none of them. The runner reads rr_parity/ssa_jobs.json (--ssa-jobs, '' leaves the arm out), each job at its own window and seed, capped at --max-reps (20).

     "backend_as_asked",
+    # The SSA arm (#1310), on SSA jobs only.
+    "ssa_cc_vs_interp",
+    "ssa_backend_as_asked",
 )

Over the corpus

rr_ssa's 268 jobs on disk, local run at the nightly's settings (--workers 4 --timeout 900): 34 s in all (median 0.12 s per model, worst 2 s).

PASS SKIP DIFF EXCEPTION TIMEOUT
ssa_cc_vs_interp 23 245 0 0 0
ssa_backend_as_asked 23 245 0 0 0

The 23 are the models the default compiles: every seed of 20 is identical draw for draw, and the default ran cc. No finding.

Tests (python/tests/test_differential_shapes.py, appended)

  • an SSA job runs the SSA checks and an ODE job none;
  • compiled and interpreted mass action agree draw for draw (a dimerization, 10 seeds);
  • negative control: one rate constant 1% off on the compiled side only reads DIFF, "mass-action". A change of 1e-9 moved no draw: it shifts each event time by as little, which crosses no sample;
  • functional laws are judged in distribution: the same ensemble at other seeds passes, and one with a propensity tripled does not;
  • codegen=False reporting compiled propensities reads DIFF;
  • a model with no propensity library runs no replicate;
  • a library the engine declines takes one replicate each way;
  • the runner reads rr_ssa's jobs as SSA jobs.

The compiled-side tests skip on a machine with no C compiler, which the skip audit declares.

Mutations, each caught (rerun on the final commit):

  • a differing seed read as a pass;
  • the mass-action rule off;
  • the ensemble verdict always passing;
  • codegen=False not passed to the interpreted side;
  • codegen=False reporting compiled propensities left unread;
  • the no-library skip off;
  • the one-replicate probe off.

Suites: parity_checks/tests and the shapes give 1,104 passed, 28 skipped, 1 xfailed. One local failure, test_jacobian_characterization_gate, needs main's fill_dense_fd_jacobian_half_step (#1002), which my local build predates; CI builds fresh. ruff 0.14.0 is clean. Merged with lanl main.

Still to do

The baseline (parity_checks/nightly/baselines/differential.json) gains these 536 rows from this PR's own nightly run. I'll record them from its nightly-differential artifact.

…#1310)

An SSA job runs the same seeds once with the default Simulator (which
compiles the propensities of an eligible model) and once with
codegen=False. On an all-elementary model the two must agree draw for
draw; otherwise the ensembles must agree at ensemble_k_for(n_rep). A
second check holds ssa_diagnostics["propensity_backend"] to what was
asked for. differential_run.py reads rr_parity/ssa_jobs.json under
--ssa-jobs, capped at --max-reps replicates.
…ng (lanl#1310)

Most of rr_ssa's 268 models run interpreted whatever is asked, so both
SSA checks are decided by one replicate of each. On the overloaded first
corpus run 31 of the first 98 models timed out at 600 s with full
ensembles both ways; the nightly's job has a 180-minute cap.
…inished

The pool kills a model's worker at its cap (900 s on the nightly), and
every row of the model with it. MODEL1006230049, 0053 and 0077 take
716-751 s of that on main, so any check a pull request added to every
model turned all their rows TIMEOUT: lanl#1315, lanl#1318 and lanl#1319 each read
27-36 regressions, none of them a finding.

check_all now takes a deadline, 0.9 of the cap. Every run has until it
(run(timeout=)), no run or check starts past it, and a check that does
not finish is its own TIMEOUT, said why. Each verdict is written as it is
reached, so a worker the pool still kills (one compiling C past the
deadline, which no run's timeout bounds) leaves the runner every row it
finished; only the rest are TIMEOUT.
…ty library (lanl#1310)

One replicate each way was still too much: 18 of the first 101 rr_ssa
models took a single replicate past the 900 s cap. The Simulator decides
before any run whether a model is eligible for compiled propensities,
and builds no library where it is not. Without one there is nothing
compiled to tell apart, and lanl#1213's class (codegen=False ignored) needs a
default that compiles: both checks are SKIP, with no replicate run.
With lanl#1321, PRs stacked on this one still read 11-12 regressions: main's
own checks on MODEL1006230049 and 0053 did not fit the 810 s deadline of
a 900 s cap. Across five runs of the same checks those models took
689-894 s, so the cap flipped their rows with runner speed. 1500 s puts
the deadline at 1350, and costs only the few models that use it.
…degen

check_all keeps lanl#1321's deadline and its verdicts-as-reached; an SSA job
is routed to check_ssa inside it, so its verdicts are kept the same way.
…rash

With the 1500 s cap this PR's nightly read 2 new crashes, both
BIOMD0000000385: given the time to reach its codegen compile, 4.2 MB of C
ran past bngsim's 600 s compile limit, which codegen=True reports as a
RuntimeError. Those rows were TIMEOUT at 900 s, and a timeout they stay:
one anywhere in the cause chain, or in codegen=True's refusal, which
carries the compile's timeout in its text and not as its cause. The other
five rows of the model went TIMEOUT -> PASS.
… fix/1310-ssa-codegen

# Conflicts:
#	python/tests/test_differential_shapes.py
…ax-reps 0 (lanl#1310)

Job's n_rep defaults to 0. With no replicates both ensembles were empty:
ssa_cc_vs_interp read PASS, 'all 0 seeds identical', and
ssa_backend_as_asked read DIFF, 'codegen=False ran []'. Nothing was
compared, so both are SKIP, said why, and --max-reps below 1 is an error.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

differential: SSA with compiled propensities against interpreted ones, seed for seed

2 participants