[Montana]

Follow-up to #1291 (#1257).

Why

The pool kills a model's worker at its cap (--timeout 900 on the nightly), and every row of the model goes with it. On main, MODEL1006230049, 0053 and 0077 already take 716–751 s of that 900 (run 40440170341). So any check a pull request adds to every model tips them over, and every check they passed before reads TIMEOUT. The baseline gate counts each of those as a regression:

PR	Path-agreement regressions	all of them	models
#1315	27	TIMEOUT, "cap of 900 s"	MODEL1006230049, 0053, 0077
#1318	36	TIMEOUT, "cap of 900 s"	the same, plus BIOMD0000000810, BIOMD0000000876
#1319	33	TIMEOUT, "cap of 900 s"	the same three, plus BIOMD0000000810

None is a finding. #1315's own baseline run even had these three passing, so they are also flaky at the edge, depending on runner speed.

What

A deadline per model. check_all(job, locale, checks, deadline) takes 0.9 of the cap (CAP_SHARE):

• every run has until then (run(timeout=), and the sparsity rule's one-interval runs);
• the locale child process gets the smaller of its 600 s and what is left;
• no run and no check starts once the deadline has passed. backend_as_asked builds and compiles Simulators without running them, so no run's timeout bounds it; on MODEL1006230049 it started past a 180 s deadline and took the worker to 203 s.

A check that doesn't finish is its own TIMEOUT, "not finished within the model's cap", and the model keeps every other row. A base that doesn't finish makes every row TIMEOUT rather than no_base, and a sensitivity run that times out is TIMEOUT rather than a refusal.

What a kill leaves. A constructor can still run past the deadline: deriving the sensitivity RHS takes 20 s, compiling C up to 50 s on these models. So each verdict is written as it is reached (on_verdict, then _keep, to a per-model file in a temp directory). When the pool kills a worker, to_results keeps every row it finished, and only the rest are TIMEOUT.

-            verdicts = D.check_all(job, spec["locale"], tuple(spec["checks"]))
+            deadline = time.monotonic() + spec["timeout"] * CAP_SHARE - (time.perf_counter() - t0)
+            verdicts = D.check_all(
+                job, spec["locale"], tuple(spec["checks"]), deadline,
+                lambda c, v: _keep(spec, c, v),
+            )

Without a deadline (any caller but the runner), nothing changes.

The cap itself (second commit)

With this PR merged into them, #1315, #1318, #1319 and #1320 still read 11–12 regressions. Main's own checks on MODEL1006230049 and 0053 did not fit the 810 s deadline: across five CI runs of the same checks those models took 689–894 s, so a 900 s cap flipped their rows with runner speed. The job's cap is now 1500 s (deadline 1350). It costs only the few models that use it: this PR's own job took 51 of its 180 minutes. Models that timed out at 900 s may now finish, which reads as improved, not as an alert.

A compile past bngsim's own limit (third commit)

With the 1500 s cap, this PR's nightly read 2 new crashes, both on BIOMD0000000385. Given the time to reach its codegen compile, its 4.2 MB of C ran past bngsim's own 600 s compile limit. codegen=True reports that as a RuntimeError, and the harness read it as EXCEPTION. Those rows were TIMEOUT at 900 s, and a timeout they stay.

_timed_out now reads a timeout:

• anywhere in the exception's cause chain (the sensitivity build chains the compile's TimeoutExpired);
• or in bngsim's fixed "Codegen compilation timed out" text, which codegen=True's refusal carries without chaining its cause.

The row's note says the compile timed out, not the model's cap. A compile that fails is still EXCEPTION. The model's other five rows went TIMEOUT → PASS ("improved", not an alert).

Tests: the in-text and the chained case are both TIMEOUT, and a compile failure stays EXCEPTION. Mutations, each caught: the chain walk off; the text match off.

Verified

End to end through the runner on the three models, at a deliberately tight --timeout 200 (3 workers, 194 s):

model	before	now
MODEL1006230049	9 TIMEOUT	5 PASS, 1 SKIP; grid, sparsity_covers_jacobian, backend_as_asked TIMEOUT
MODEL1006230053	9 TIMEOUT	5 PASS; cc_vs_mir, grid, sparsity_covers_jacobian, backend_as_asked TIMEOUT
MODEL1006230077	9 TIMEOUT	5 PASS; the same 4 TIMEOUT

At the nightly's 900 s their checks finish (716–751 s) inside the 810 s deadline as before, so main's verdicts are unchanged. A check a pull request adds then runs in what is left, and is TIMEOUT on its own row, which is a new case and not a regression.

Tests (python/tests/test_differential_shapes.py):

• a run past the deadline does not start;
• a cap spent after the base times out each check alone (no exception, no pass), and clears the deadline;
• no check starts past the deadline (a planted check_backend is never called);
• a base past the deadline is TIMEOUT on every row;
• no deadline bounds nothing;
• the runner's worker hands each model its cap;
• a killed model keeps the rows it finished;
• each verdict is heard as it is reached.

Mutations, each caught:

• the run unbounded;
• the timeout not mapped;
• a sensitivity timeout read as a refusal;
• check_all setting no deadline;
• the deadline not cleared;
• _states unbounded;
• a base timeout read as no_base;
• the worker passing no deadline;
• a check starting past the deadline;
• kept rows ignored;
• nothing kept;
• verdicts not heard.

Suites: parity_checks/tests and the shapes give 1,103 passed, 28 skipped, 1 xfailed. One local failure, test_jacobian_characterization_gate, needs main's fill_dense_fd_jacobian_half_step (#1002), which my local build predates; CI builds fresh. ruff 0.14.0 is clean.

Once this lands I'll merge it into #1315, #1318 and #1319, and order each one's new checks after main's, so they run in what is left of the cap.