feat: cache per-galaxy result images for SLaM resume fast-path (#502) - #504
Merged
Conversation
galaxy_name_image_dict_via_result_from rebuilds the result's maximum log likelihood fit (fresh JIT compile + inversion for pixelized fits) on every call — on a resumed SLaM pipeline this dominates resume overhead (autolens_profiling#70: ~55s of a 148s resume, more in production). The raw per-galaxy image dict is now cached into the upstream result's own files/ folder on first computation (galaxy_images_model.fits / galaxy_images_snr.fits, FITS layout mirroring the aggregator's adapt_images artifact) and loaded on every later call. The cache file is also appended into the search's .zip — a resumed search's paths.restore() wipes the output dir and re-extracts the zip, so a files/-only cache would be destroyed by the next resume. Staleness is structurally guarded: model/search changes produce a new identifier and a fresh output dir. Results with no on-disk output (NullPaths) always compute. The minimum-percent clip is applied after load/compute identically, so returned values are unchanged. Validated via autolens_profiling pipeline_resume (with the PyAutoLens positions leg): per-stage inter-stage resume cost 151s/130s/128s -> 0.3s/ 0.0s/0.1s. Issue: PyAutoGalaxy#502. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Companion PyAutoLens PR: PyAutoLabs/PyAutoLens#619 (merge this PyAutoGalaxy PR first — the PyAutoLens leg imports _append_to_search_zip from it). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the PyAutoGalaxy leg of the SLaM resume fast-path (#502, judgment from autolens_profiling#70):
galaxy_name_image_dict_via_result_fromnow caches the raw per-galaxy image dictionary into the upstream result's ownfiles/folder on first computation and loads it on every later call, instead of rebuilding the result's maximum log likelihood fit (fresh JIT compile + inversion) each time.Key mechanics:
galaxy_images_model.fits/galaxy_images_snr.fits, FITS layout mirroring the aggregator'sadapt_imagesartifact (mask HDU + per-galaxyEXTNAMEHDUs)..zip(_append_to_search_zip) — a resumed search'spaths.restore()deletes the output dir and re-extracts the zip, so afiles/-only cache would be destroyed by the next resume (found empirically during validation).NullPathsresults always compute; the minimum-percent clip applies after load/compute identically.Validated with the PyAutoLens leg via the
autolens_profilingpipeline_resumeinstant recipe: per-stage inter-stage resume cost 151s / 130s / 128s → 0.3s / 0.0s / 0.1s.API Changes
None — no public API changes. Behaviour of
galaxy_name_image_dict_via_result_fromis identical on first computation; later calls return equal values from disk. New side effect: two FITS cache files written into a result'sfiles/folder (and its zip) when the result has on-disk output.Details
Added
autogalaxy/analysis/adapt_images/adapt_images.py: private helpers_galaxy_images_cache_path,_galaxy_image_dict_from_cache,_galaxy_image_dict_to_cache,_append_to_search_zip(the last is imported by the PyAutoLens leg).Changed Behaviour
galaxy_name_image_dict_via_result_from: cache-aside on the result'spaths; compute-and-write on first arrival, load thereafter.Migration
Test Plan
test_autogalaxy/analysis/test_adapt_images.py— 3 new tests: FITS round-trip equality, poisoned-result cache hit (recompute raises), pathless result always computes.test_autogalaxy/suite green in the task worktree.autolens_profilingpipeline_resume(numbers above).Companion PR: PyAutoLens positions leg (same branch name, link added after creation).
🤖 Generated with Claude Code