feat: Record + Z-Stack acquisition mode

[hongquanli]

Summary

Adds a new Record + Z-Stack acquisition mode (its own tab, next to Wellplate Multipoint, gated by ENABLE_RECORDING). It walks selected wells × a per-well FOV grid × Nt time points and, at each FOV, runs two phases in sequence:

1. Recording — a continuous single-channel video: T = round(fps × duration) frames at one Z plane, saved as a per-FOV Zarr (T, 1, 1, Y, X).
2. Z-stack — a multi-channel stack over a Z range, saved as OME-Zarr (Nt, C, NZ, Y, X) per FOV.

Both phases are positioned in Z by offsets relative to a reference plane (laser reflection AF when enabled+referenced, otherwise the current Z). Each phase is independently enable-able.

Architecture

• MultiPointWorkerBase — behavior-preserving refactor lifting the shared capture mechanics (channel apply, single-frame capture, frame callback + SaveZarrJob dispatch, backpressure, abort/progress) out of MultiPointWorker. After merging master, the acquisition-watchdog hooks (_abort_due_to_error, per-image run-state beats) live on the base so both workers classify error-aborts identically.
• StreamingCapture (control/core/streaming_capture.py) — the recording primitive: a continuous frame source → bounded queue (count- and byte-capped) → writer thread → direct ZarrWriter. Generic frame_source/frame_router/stop_condition seam, shaped so a future hardware-sequenced source drops in.
• RecordZStackWorker / RecordZStackController — thin t×well×FOV loop reusing the inherited mechanics; recording via StreamingCapture, z-stack via the existing SaveZarrJob path. Timepoints are paced on the absolute t0 + k·dt grid with grid-preserving skip (mirrors MultiPointWorker); each experiment dir gets the acquisition_channels.yaml snapshot and .done marker like every multipoint acquisition.
• RecordZStackMultiPointWidget + gui_hcs tab wiring. Per-well FOV grid configurable inline and rebuilt on tab entry/well clicks; channels editable inline with Copy-from-Live; channel combos refresh on objective/profile/config changes. XY/Time controls mirror WellplateMultiPointWidget's tabbed row (see below).
• Camera frame-rate hint — AbstractCamera.set_frame_rate(fps) (no-op default; ToupTek PRECISE_FRAMERATE; simulated camera honors it). The recording sizes its dataset, pacing, and time_increment_s from the achievable rate the camera reports, with software downsampling as the portable safety net.

Widget UI polish

• XY/Time tabbed row, mirroring WellplateMultiPointWidget (checkbox + combo in a frame that highlights orange/green when active), skipping Z since the widget's own Z-Stack phase group already covers z-stacking:
  • combobox_xy_mode now offers two real modes: Select Wells (today's tile-a-grid-over-selected-wells behavior, default) and Current Position (single FOV at the live stage position, bypassing well selection entirely — validate() no longer requires a well selection in this mode). Unchecking XY forces Current Position and disables the combo; re-checking restores the previous mode.
  • checkbox_time wraps Nt/dt: unchecked forces a single timepoint and hides the controls, restoring the previous Nt/dt on re-check.
  • checkbox_laser_af moved into this row as a plain checkbox, matching how WellplateMultiPointWidget places its own Laser AF toggle.
• Channel-add seeds from configured settings: the z-stack "+ Add" button and the recording-channel combo (initial row and on selection change) now seed exposure/gain/illumination from the channel's own config via liveController.get_channels(), instead of a hardcoded 50 ms / 0 gain / 50%.
• Layout: removed the channel tables' fixed 530px width — it fought against whatever width the main window happened to lock in for this tab (centralWidget.setFixedWidth(minimumSizeHint()) in gui_hcs.py sizes the whole app once at startup from whichever tab is active then) — in favor of a responsive fill plus tightened field widths, so the panel no longer needs a horizontal scrollbar. The Z-Stack channel table now spans the group's full width to match the Recording table, and z-stack channel names get a tooltip with the full name since the Channel column truncates.
• ~20 new tests; caught in review and fixed with a reproducing test first: refresh_channel_list() was silently resetting the user's manually-edited recording exposure/gain/illumination back to the channel's base config (repopulating the combo transiently fires currentIndexChanged); a checkbox toggle could rebuild the scan region twice; dark-theme text contrast on the new Time tab; duplicated channel-lookup logic consolidated into a shared _find_channel() helper.

Data-integrity guarantees (recording pipeline)

These came out of the review rounds below and are worth stating as contract:

• acquisition_complete: true is only stamped when every expected frame was written. Write errors, backpressure drops, and under-delivery (camera stall) all seal the store incomplete with counts (write_errors, dropped_frames, captured_frames/expected_frames); user aborts additionally stamp aborted: true.
• The recording dataset is sized from one real processed frame (crop/rotation/ROI applied), not get_resolution() — sensor-vs-delivered mismatches previously produced silently blank recordings on real cameras.
• Frames map to the time slot nearest their arrival time: delivery jitter can't halve the capture rate, and a post-stall burst leaves the stall as fill holes instead of compressing the time axis.
• The acquisition fails fast (aborts, store sealed incomplete) on systematic recording failure: write errors, sustained drops, or a wedged writer — instead of grinding through hours of blank FOVs. A wedged writer's captured backlog is flushed once the stall clears, never discarded.
• Hardware state is restored after every run: trigger mode, channel configuration (exposure/gain/illumination), live view, and Z.

Review & hardening

Three review rounds (multi-agent adversarial review with independent verification, plus hand-verification when API limits interrupted round 2) — 40 confirmed findings, all fixed, each with a test that failed first:

• Round 1 (17): critical hardware-path bugs simulation couldn't surface — blank recordings from get_resolution() sizing, error-swallowing drain, finalize deadlock, unresponsive abort, trigger/channel state corruption after runs, fps-gate jitter halving capture rate, count-only queue bound (~13 GB RAM risk), per-FOV z coords dropped, dt pacing/metadata mismatch, missing cleanup/bookkeeping.
• Round 2 (10 + 13 after full verification): completeness-attribute lies (drops/under-delivery), wedged-finalize data discard and false-positive fail-fast, well-selector lost after acquisitions, inert well clicks on the tab, silent recording-channel swaps, monotonic pacing, seal-attribute honesty, snapshot dedupe, per-FOV double camera reconfiguration.
• Merge with master ported the acquisition-watchdog (feat: acquisition watchdog — Slack alert on prematurely-ended acquisitions #565) and z-offset logging (feat(worker): log actual z after each per-channel z-offset move #573) into the refactored base class.

Testing

• ~120 new tests: StreamingCapture/RecordingWriter/RecordingRouter (error/OOB/abort/wedge/drop/jitter/burst paths), worker simulation (shapes, state restore, pacing, fail-fast, bookkeeping), widget (validation, Copy-from-Live, refresh, Start/Stop handoff, FOV-grid wiring), gui-level tab/selector behavior, camera fps.
• Runtime-verified end-to-end by driving the real GUI in simulation (screenshots + on-disk Zarr shape/attr verification): happy path, validation rejects, mid-recording abort, Nt=2 t-indexing, phase-solo runs, z-offset stage return, back-to-back acquisitions.
• CI green (full install-and-test suite) after resolving the master conflict — note the PR had been CONFLICTING, which silently skips pull_request workflows, so earlier pushes only ran lint.
• black clean (CI 25.12.0).

Hardware testing (pending — the remaining merge gate)

See the test-plan comment: instrument checklist + a camera-only bench mode (real ToupTek camera, everything else simulated via the [SIMULATION] INI section). Highest-value bench test: the PRECISE_FRAMERATE fps hint, which has never executed against real hardware.

Deferred follow-ups (documented, out of scope here)

• GUI-visible error dialog when the fail-fast aborts an acquisition (currently loud in the log only; needs an error signal through the Qt controller).
• closeEvent acquisition guard: app exit during a run now waits 30 s for the worker to unwind, but a still-running worker can race teardown beyond that; a proper "acquisition in progress — abort and exit?" prompt is the full fix.
• NTP-step resilience for the recording router's absolute-slot anchor (wall-clock frame timestamps; a mid-recording clock step mispaces until re-anchored).
• From the original plan: shared pre-warmed-JobRunner mixin, and replacing the duck-typed record-tab stubs (display_progress_bar, emit_selected_channels) with a protocol.

🤖 Generated with Claude Code

[hongquanli]

Hardware test plan

Simulation testing (GUI driven end-to-end, Zarr outputs verified — happy path, validation, Copy-from-Live, mid-recording abort, Nt=2 t-indexing, recording-only / z-stack-only, z-offset with stage return) is green as of 9b5be65. The items below are the paths that only run on real hardware and should be checked on an instrument before this ships.

Setup

1. Add enable_recording = True to the [GENERAL] section of configuration_*.ini and restart.
2. The Record + Z-Stack tab appears next to Simple Recording. Switching to it should keep the well selector visible and log no traceback (fixed in 9b5be65 — worth re-confirming on the instrument build).
3. Put a real sample (or fluorescent slide) on stage, focus, and select 1–2 wells.

Tests

• 1. Recording illumination (validates the critical fix). Recording phase only, BF channel, 10 fps × 5 s. Open the recording Zarr — frames must show the sample, not darkness. Repeat with a fluorescence channel. This exercises the illumination-energize fix that simulation can't prove (the sim camera synthesizes images regardless of illumination).
• 2. fps hint (ToupTek PRECISE_FRAMERATE). 10 fps × 10 s → expect exactly T=100 frames, _squid.time_increment_s = 0.1, wall time ≈ 10 s. Then request an impossible rate (e.g. 50 ms exposure @ 30 fps) — the portable safety net should downsample in software; check the log for dropped/queue-full warnings and confirm the Zarr still seals with acquisition_complete: true.
• 3. Hardware-trigger live interplay. Set live to Hardware Trigger, start live, stop live, then run an acquisition with both phases. The z-stack must not come back dark (the trigger-mode fix), and after the acquisition live view must restore and work in the previously selected mode.
• 4. Laser AF reference path. Initialize laser AF and set a reference, tick Laser AF in the tab, run with a recording z-offset (e.g. +10 µm) and a z-stack range. Both phases should be positioned relative to the reference plane — compare focus between the recording frames and the z-stack center plane. Also confirm Start is rejected with a clear message when Laser AF is ticked without a reference.
• 5. Abort mid-recording. 60 s recording, hit Stop after ~10 s. UI must unlock immediately, the recording Zarr must carry _squid.aborted: true / acquisition_complete: false, stage and illumination must return to a sane state, and a follow-up acquisition must work without restarting the app.
• 6. Sustained throughput / backpressure. Realistic load: full-resolution camera, ~30 fps × 60 s recording × 2 wells × a small FOV grid, plus a multi-channel z-stack. Watch the RAM indicator, check the log for queue full / drop warnings from the recording writer, and verify every per-FOV Zarr has the full expected frame count.
• 7. Multi-timepoint on hardware. Nt=3, dt ≥ per-timepoint duration. Expect recording/t0..t2/ trees and a z-stack Zarr (3, C, NZ, Y, X) with all three t-slices populated (verified in simulation; re-check with real stage settle/AF timing).

Inspecting outputs

Per-FOV shapes: recording (T, 1, 1, Y, X), z-stack (Nt, C, NZ, Y, X). Quick check without opening napari:

import json, glob
for p in glob.glob("<experiment_dir>/**/zarr.json", recursive=True):
    m = json.load(open(p))
    if m.get("node_type") == "array":
        print(p, m["shape"], json.load(open(p)).get("attributes", {}).get("_squid", {}))

Log signals worth watching: squid.control.core.zarr_writer warnings (abort path is intentionally loud), recording-writer drop warnings, and [MCU] >>> illumination commands around phase transitions.

---

Camera-only bench (no assembled scope)

If you only have the ToupTek camera on a dev machine, Squid's per-component simulation covers most of the list above. In configuration_*.ini:

[SIMULATION]
simulate_camera = false           ; real ToupTek camera
simulate_microcontroller = true   ; simulated MCU, stage, illumination
simulate_spinning_disk = true
simulate_filter_wheel = true
simulate_objective_changer = true
simulate_laser_af_camera = true

Then launch without --simulation (the flag forces everything simulated; without it the per-component settings apply — see _should_simulate in microscope.py). Stage moves become instant no-ops; every frame comes off the real sensor.

What this bench covers, in value order:

• fps hint (PRECISE_FRAMERATE) — the new camera_toupcam.py code has never executed against real hardware (the simulated camera has its own set_frame_rate). 10 fps × 10 s → exactly T=100 frames, wall time ≈ 10 s, _squid.time_increment_s = 0.1.
• StreamingCapture under real frame timing — real callback thread, real jitter, full-res frames into the bounded queue. Feasible rate: no drop warnings, exact count. Then request an infeasible rate (e.g. 100 ms exposure @ 30 fps) and observe: does the run stretch until T frames arrive, or do drops get logged loudly?
• Real pixels through the save path — point the camera at the room, wave a hand during recording, verify frame-to-frame differences in the Zarr are non-zero (a live temporal stream, not repeated buffers).
• Abort/recovery with a real device — Stop mid-recording → aborted: true seal → Start Live again streams normally without replug/restart.
• Multi-well/FOV fan-out at real data rates — 2 wells × 4 FOVs; simulated stage walks instantly, per-FOV Zarr routing carries real frames.

Still instrument-only after this bench: illumination energize (dark-frame check), hardware-trigger interplay, laser AF, real stage-settle timing (tests 1, 3, 4 above).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds a new Record + Z-Stack acquisition mode to the HCS GUI, enabling per-FOV sequential acquisition of (1) a continuous recording phase saved as per-FOV Zarr and (2) a multi-channel Z-stack saved as OME-Zarr, with shared acquisition mechanics refactored into a reusable worker base.

Changes:

• Introduces a recording pipeline (StreamingCapture) with bounded backpressure and integrity sealing for incomplete captures.
• Adds RecordZStackController/RecordZStackWorker plus GUI wiring for a new “Record + Z-Stack” tab.
• Refactors shared capture/job-dispatch mechanics into MultiPointWorkerBase and adds a best-effort camera FPS hint API (set_frame_rate) including Toupcam support.

Reviewed changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
software/control/acquisition_yaml_loader.py	Extends YAML parsing to support widget_type: record_zstack.
software/control/camera_toupcam.py	Adds Toupcam set_frame_rate() support and continuous-mode max FPS calculation.
software/control/core/acquisition_setup.py	Introduces shared helpers for pixel size calculation and experiment directory creation.
software/control/core/multi_point_controller.py	Uses shared experiment-dir creation helper.
software/control/core/multi_point_worker.py	Refactors shared acquisition mechanics into MultiPointWorkerBase.
software/control/core/record_zstack_controller.py	Adds controller for Record + Z-Stack acquisition setup/threading and YAML snapshotting.
software/control/core/record_zstack_worker.py	Adds worker implementing per-FOV recording + z-stack sequencing.
software/control/core/streaming_capture.py	Adds recording primitives (frame source/router/stop condition/writer) with bounded queues.
software/control/core/zarr_writer.py	Improves event-loop handling and expands abort sealing to distinguish incomplete vs aborted runs.
software/control/gui_hcs.py	Wires new Record + Z-Stack tab/widget/controller into the HCS GUI and well-selector behavior.
software/squid/abc.py	Adds AbstractCamera.set_frame_rate() API (default no-op returning max achievable).
software/squid/camera/utils.py	Implements set_frame_rate() and continuous pacing behavior in the simulated camera.
software/tests/control/test_HighContentScreeningGui.py	Adds GUI tests ensuring well-selector behavior on the new tab.
software/tests/control/test_acquisition_yaml_loader.py	Adds YAML loader tests for record_zstack acquisitions.
software/tests/core/init.py	Test package init (structure/support for new core tests).
software/tests/core/test_record_zstack_worker.py	Adds worker/controller smoke + behavior tests for Record + Z-Stack acquisition.
software/tests/core/test_streaming_capture.py	Adds extensive unit tests for the recording pipeline and its failure modes.
software/tests/test_acquisition_yaml_drop_mixin.py	Adds tests for YAML drop mixin generalizations supporting the 3rd widget type.
software/tests/test_record_zstack_camera_fps.py	Adds tests for camera FPS hinting/clamping (sim + Toupcam math).

Comments suppressed due to low confidence (1)

software/control/acquisition_yaml_loader.py:133

• parse_acquisition_yaml() is annotated/documented to return AcquisitionYAMLData, but it now returns RecordZStackYAMLData when widget_type == "record_zstack". This makes the public API misleading and can break type-aware tooling (and any downstream code that relies on AcquisitionYAMLData-specific fields without checking widget_type first).

def parse_acquisition_yaml(file_path: str) -> AcquisitionYAMLData:
    """Parse acquisition YAML file and return structured data.

    Args:
        file_path: Path to the acquisition.yaml file

    Returns:
        AcquisitionYAMLData with parsed values

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

software/squid/camera/utils.py:312

• SimulatedCamera.set_frame_rate() clamps achievable fps using get_total_frame_time() (exposure + strobe), but the CONTINUOUS streaming loop paces frames using only exposure time. This can make the simulated camera deliver frames faster than the achievable fps it reports, which can mis-size recordings and invalidate tests that rely on achievable-fps behavior.

                period_s = self._exposure_time_ms / 1000.0
                if self._target_frame_period_s is not None:
                    period_s = max(period_s, self._target_frame_period_s)
                time_since = time.time() - last_frame_time
                if time_since >= period_s and self._acquisition_mode == CameraAcquisitionMode.CONTINUOUS:

[Alpaca233]

fixed in lastest master

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

software/squid/camera/utils.py:312

• The simulated camera’s streaming loop paces frames using exposure time only, but set_frame_rate() clamps/returns an achievable rate using get_total_frame_time() (exposure + strobe). This makes the simulated camera’s actual free-run period inconsistent with the frame-time contract used elsewhere (and with the achievable fps that callers compute).

            last_frame_time = time.time()
            while self._continue_streaming:
                period_s = self._exposure_time_ms / 1000.0
                if self._target_frame_period_s is not None:
                    period_s = max(period_s, self._target_frame_period_s)
                time_since = time.time() - last_frame_time
                if time_since >= period_s and self._acquisition_mode == CameraAcquisitionMode.CONTINUOUS: