fix: park EP scripts as NEEDS_FIX no_run for release testing - #82
Merged
Conversation
EP is not release-ready. The per-factor searches in the EP scripts are truncated (autofit_workspace_test graphical/ep.py caps maxcall/maxiter at 1000), so each search projects a posterior with an effective sample size of 1 - a single sample carries all the weight and the weighted variance of every parameter is exactly 0. EP feeds that degenerate projection back as the next cycle's prior, which is then either a delta function or, after the cavity division cancels two near-identical natural-parameter vectors, a prior whose mean has diverged (observed: shared centre = 6.4e7 for data spanning ~100 pixels). Either way the next search draws initial points that all yield an identical log likelihood, and PyAutoFit raises InitializerException. This is intermittent - it depends on where the truncated search happens to stop - and it broke the 2026-08-03 nightly release integrate (PyAutoHeart run 30788224561) after passing the five previous runs. Damping does not fix it: q* arrives already collapsed, so an EP updater with delta < 1 still ingests a zero-variance projection. The existing check_sigma_collapse diagnostic also misses it, because the collapse is a single-step drop rather than the monotone shrinkage it looks for. Park every EP script in the release matrix rather than papering over the instability in one script's search settings. Tagged NEEDS_FIX so each mega-run surfaces them until the underlying EP projection is fixed (PyAutoFit #1332 F10). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
EP is not release-ready. The 2026-08-03 nightly release integrate failed on
scripts/graphical/ep.py(PyAutoBrain run 30787947298 -> PyAutoHeart run30788224561), reported as "1 failed: autofit_test graphical/ep.py".
This parks the EP scripts rather than papering over the instability in one
script's search settings.
Root cause (reproduced locally under the release profile)
The cloud message "The
log_likelihood_functionis always returningnanvalues" is misleading — it is one of three causes listed in a single
InitializerException. The real trigger is the "all initial samples have thesame figure of merit" branch. No NaN is involved.
ep.pytruncates its per-factor search (maxcall=1000, maxiter=1000).Probing one such search directly gives an effective sample size of 1 — a
single sample carries all the weight, so the weighted std of every
parameter is exactly
0. An uncapped search on the same data gives ESS≈ 1420 and healthy stds ≈ 0.13.
the next cycle's prior. Instrumenting the EP update shows
q*arrivingalready collapsed (
q*_sigma=1.5e-06against a healthycavity_sigma=30), so the collapse enters at the search→message projection,not the cavity division.
likelihood, and PyAutoFit raises
InitializerException.Both failure faces were captured: sigma collapsing to delta functions, and the
shared
centrediverging to 6.4e7 (data spans ~100 pixels) so the Gaussianfalls entirely outside the data — giving a figure-of-merit spread of exactly
0.0.It is intermittent (it depends on where the truncated search happens to stop),
which is why
graphicalpassed the five previous nightly runs. The EP sourcehas not changed in weeks — this is a long-standing flake that finally bit, not
a regression.
Two things that do not fix it
delta=0.5a prior still collapsed in a single step —q*is already degenerate before damping applies.check_sigma_collapse. It looks for monotone shrinkage or std < 1e-8;this is a one-step drop to ~1e-7, so it reports (none) even mid-collapse.
See PyAutoFit #1332 F10.
Change
Four
NEEDS_FIX-tagged entries inconfig/build/no_run.yaml, so every mega-runsurfaces them with a warning banner until EP is fixed. Sibling PRs park the
remaining two EP scripts in the release matrix (
autofit_workspacefeatures/expectation_propagation.py,autolens_workspaceguides/modeling/advanced/expectation_propagation.py). HowToFit's two EPtutorials are not in the release matrix and need no entry.
Verification
.py-anchored and checked against the runner's ownshould_skip: exactly 4 scripts newly skipped here, no over-match, noinert entries. (An unanchored
graphical/epwould have silently swallowed allfour via substring match — anchoring keeps each reason attached to its script.)
find_needs_fix_skipswith correct dates and reasons.run_python.py autofit_test scripts/graphical --env-config config/build/profile_release.yaml→ 4 skipped with reasons, 3 passed.
🤖 Generated with Claude Code