Repository navigation
Conversation
…#1310) An SSA job runs the same seeds once with the default Simulator (which compiles the propensities of an eligible model) and once with codegen=False. On an all-elementary model the two must agree draw for draw; otherwise the ensembles must agree at ensemble_k_for(n_rep). A second check holds ssa_diagnostics["propensity_backend"] to what was asked for. differential_run.py reads rr_parity/ssa_jobs.json under --ssa-jobs, capped at --max-reps replicates.
…ng (lanl#1310) Most of rr_ssa's 268 models run interpreted whatever is asked, so both SSA checks are decided by one replicate of each. On the overloaded first corpus run 31 of the first 98 models timed out at 600 s with full ensembles both ways; the nightly's job has a 180-minute cap.
…inished The pool kills a model's worker at its cap (900 s on the nightly), and every row of the model with it. MODEL1006230049, 0053 and 0077 take 716-751 s of that on main, so any check a pull request added to every model turned all their rows TIMEOUT: lanl#1315, lanl#1318 and lanl#1319 each read 27-36 regressions, none of them a finding. check_all now takes a deadline, 0.9 of the cap. Every run has until it (run(timeout=)), no run or check starts past it, and a check that does not finish is its own TIMEOUT, said why. Each verdict is written as it is reached, so a worker the pool still kills (one compiling C past the deadline, which no run's timeout bounds) leaves the runner every row it finished; only the rest are TIMEOUT.
…ty library (lanl#1310) One replicate each way was still too much: 18 of the first 101 rr_ssa models took a single replicate past the 900 s cap. The Simulator decides before any run whether a model is eligible for compiled propensities, and builds no library where it is not. Without one there is nothing compiled to tell apart, and lanl#1213's class (codegen=False ignored) needs a default that compiles: both checks are SKIP, with no replicate run.
With lanl#1321, PRs stacked on this one still read 11-12 regressions: main's own checks on MODEL1006230049 and 0053 did not fit the 810 s deadline of a 900 s cap. Across five runs of the same checks those models took 689-894 s, so the cap flipped their rows with runner speed. 1500 s puts the deadline at 1350, and costs only the few models that use it.
…degen check_all keeps lanl#1321's deadline and its verdicts-as-reached; an SSA job is routed to check_ssa inside it, so its verdicts are kept the same way.
…rash With the 1500 s cap this PR's nightly read 2 new crashes, both BIOMD0000000385: given the time to reach its codegen compile, 4.2 MB of C ran past bngsim's 600 s compile limit, which codegen=True reports as a RuntimeError. Those rows were TIMEOUT at 900 s, and a timeout they stay: one anywhere in the cause chain, or in codegen=True's refusal, which carries the compile's timeout in its text and not as its cause. The other five rows of the model went TIMEOUT -> PASS.
… fix/1310-ssa-codegen # Conflicts: # python/tests/test_differential_shapes.py
…ax-reps 0 (lanl#1310) Job's n_rep defaults to 0. With no replicates both ensembles were empty: ssa_cc_vs_interp read PASS, 'all 0 seeds identical', and ssa_backend_as_asked read DIFF, 'codegen=False ran []'. Nothing was compared, so both are SKIP, said why, and --max-reps below 1 is an error.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1310.
What it adds
An SSA arm of the differential report: compiled SSA propensities against interpreted ones.
ssa_cc_vs_interpcodegen=Falsessa_backend_as_askedssa_diagnostics["propensity_backend"]codegen=Falseinterpreted, the default compiled (#1213's class)codegen=Trueis refused for SSA, so the compiled side is the default Simulator. Each job runsn_repreplicates at seedsseedup toseed + n_rep − 1:differ.ensemble_verdictatensemble_k_for(n_rep), as the SSA parity suites judge. That branch is for when eligibility widens to functional laws; today none is compiled.Cost, which shaped the design. Compiled propensities serve exact SSA only on mass-action models without events, up to a size (GH #190). The Simulator decides that before any run, and builds no propensity library for a model it finds ineligible. So:
Jobgainsmethod,n_repandseed;checks_for(job)gives an SSA job the SSA checks only, and an ODE job none of them. The runner readsrr_parity/ssa_jobs.json(--ssa-jobs,''leaves the arm out), each job at its own window and seed, capped at--max-reps(20).Over the corpus
rr_ssa's 268 jobs on disk, local run at the nightly's settings (
--workers 4 --timeout 900): 34 s in all (median 0.12 s per model, worst 2 s).ssa_cc_vs_interpssa_backend_as_askedThe 23 are the models the default compiles: every seed of 20 is identical draw for draw, and the default ran
cc. No finding.Tests (
python/tests/test_differential_shapes.py, appended)codegen=Falsereporting compiled propensities reads DIFF;The compiled-side tests skip on a machine with no C compiler, which the skip audit declares.
Mutations, each caught (rerun on the final commit):
codegen=Falsenot passed to the interpreted side;codegen=Falsereporting compiled propensities left unread;Suites:
parity_checks/testsand the shapes give 1,104 passed, 28 skipped, 1 xfailed. One local failure,test_jacobian_characterization_gate, needs main'sfill_dense_fd_jacobian_half_step(#1002), which my local build predates; CI builds fresh. ruff 0.14.0 is clean. Merged with lanlmain.Still to do
The baseline (
parity_checks/nightly/baselines/differential.json) gains these 536 rows from this PR's own nightly run. I'll record them from itsnightly-differentialartifact.