Skip to content

fix: park EP scripts as NEEDS_FIX no_run for release testing - #82

Merged
Jammy2211 merged 1 commit into
mainfrom
fix/ep-no-run-release
Aug 3, 2026
Merged

fix: park EP scripts as NEEDS_FIX no_run for release testing#82
Jammy2211 merged 1 commit into
mainfrom
fix/ep-no-run-release

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Why

EP is not release-ready. The 2026-08-03 nightly release integrate failed on
scripts/graphical/ep.py (PyAutoBrain run 30787947298 -> PyAutoHeart run
30788224561), reported as "1 failed: autofit_test graphical/ep.py".

This parks the EP scripts rather than papering over the instability in one
script's search settings.

Root cause (reproduced locally under the release profile)

The cloud message "The log_likelihood_function is always returning nan
values"
is misleading — it is one of three causes listed in a single
InitializerException. The real trigger is the "all initial samples have the
same figure of merit"
branch. No NaN is involved.

  1. ep.py truncates its per-factor search (maxcall=1000, maxiter=1000).
    Probing one such search directly gives an effective sample size of 1 — a
    single sample carries all the weight, so the weighted std of every
    parameter is exactly 0. An uncapped search on the same data gives ESS
    ≈ 1420 and healthy stds ≈ 0.13.
  2. EP projects that degenerate posterior into a message and feeds it back as
    the next cycle's prior. Instrumenting the EP update shows q* arriving
    already collapsed (q*_sigma=1.5e-06 against a healthy
    cavity_sigma=30), so the collapse enters at the search→message projection,
    not the cavity division.
  3. The next search then draws initial points that all yield an identical log
    likelihood, and PyAutoFit raises InitializerException.

Both failure faces were captured: sigma collapsing to delta functions, and the
shared centre diverging to 6.4e7 (data spans ~100 pixels) so the Gaussian
falls entirely outside the data — giving a figure-of-merit spread of exactly
0.0.

It is intermittent (it depends on where the truncated search happens to stop),
which is why graphical passed the five previous nightly runs. The EP source
has not changed in weeks — this is a long-standing flake that finally bit, not
a regression.

Two things that do not fix it

  • Damping. At delta=0.5 a prior still collapsed in a single step —
    q* is already degenerate before damping applies.
  • check_sigma_collapse. It looks for monotone shrinkage or std < 1e-8;
    this is a one-step drop to ~1e-7, so it reports (none) even mid-collapse.

See PyAutoFit #1332 F10.

Change

Four NEEDS_FIX-tagged entries in config/build/no_run.yaml, so every mega-run
surfaces them with a warning banner until EP is fixed. Sibling PRs park the
remaining two EP scripts in the release matrix (autofit_workspace
features/expectation_propagation.py, autolens_workspace
guides/modeling/advanced/expectation_propagation.py). HowToFit's two EP
tutorials are not in the release matrix and need no entry.

Verification

  • Patterns are .py-anchored and checked against the runner's own
    should_skip: exactly 4 scripts newly skipped here, no over-match, no
    inert entries. (An unanchored graphical/ep would have silently swallowed all
    four via substring match — anchoring keeps each reason attached to its script.)
  • All entries parse via find_needs_fix_skips with correct dates and reasons.
  • The exact CI command for the failing shard now exits 0:
    run_python.py autofit_test scripts/graphical --env-config config/build/profile_release.yaml
    → 4 skipped with reasons, 3 passed.

🤖 Generated with Claude Code

EP is not release-ready. The per-factor searches in the EP scripts are
truncated (autofit_workspace_test graphical/ep.py caps maxcall/maxiter at
1000), so each search projects a posterior with an effective sample size of
1 - a single sample carries all the weight and the weighted variance of
every parameter is exactly 0. EP feeds that degenerate projection back as
the next cycle's prior, which is then either a delta function or, after the
cavity division cancels two near-identical natural-parameter vectors, a
prior whose mean has diverged (observed: shared centre = 6.4e7 for data
spanning ~100 pixels). Either way the next search draws initial points that
all yield an identical log likelihood, and PyAutoFit raises
InitializerException.

This is intermittent - it depends on where the truncated search happens to
stop - and it broke the 2026-08-03 nightly release integrate
(PyAutoHeart run 30788224561) after passing the five previous runs.

Damping does not fix it: q* arrives already collapsed, so an EP updater
with delta < 1 still ingests a zero-variance projection. The existing
check_sigma_collapse diagnostic also misses it, because the collapse is a
single-step drop rather than the monotone shrinkage it looks for.

Park every EP script in the release matrix rather than papering over the
instability in one script's search settings. Tagged NEEDS_FIX so each
mega-run surfaces them until the underlying EP projection is fixed
(PyAutoFit #1332 F10).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Jammy2211
Jammy2211 merged commit 1a39a7b into main Aug 3, 2026
3 checks passed
@Jammy2211
Jammy2211 deleted the fix/ep-no-run-release branch August 25, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant