Mask noise-only channels in cleaning

[thomaswilliamsastro]

This PR adds a function to test if channels in a cube are noise-only, and if so to mask them in the clean mask and skip through multiscale/singlescale cleaning.

There are two checks that go on here: the first is that the sample-size-independent departure score (calculated from skew and kurtosis of the data) is above a sigma-threshold to a null test. The second is that the Jarque-Bera statistic is above a sigma-threshold. There is a little complication here that statistics need to be corrected for the effective number of independent measurements (number of beams), but this is all handled within the code.

By default, this check is off. There's one exposed parameter, channel_noise_sigma_threshold, for which a value of 3 seems reasonable. Lower values are more stringent (i.e. will say there's signal in the channel at lower significance).

Speed up is good! For an example 7m single field, processing time goes from 367s to 211s. For a 7m mosaic, it's 4.2h down to 2.4h. So about twice as fast.

The cubes are also basically identical. Plots attached for single field and mosaic, the blue show the median of the data and 1/99th percentiles, the black the same for the difference between the two cubes. Any deviations are well within the noise of the data.

Across these two cases, if we take the 99th percentile of the absolute differences of the cubes, the differences are (as a sigma value):

• Mosaic field: 0.39sigma
• Single field: 0.55sigma

this is well within the typical noise. Taking the maximum of the residuals, this rises to 3.21/1.64sigma. Given the number of effective samples in the data, a purely Gaussian field should have maximum outliers of 4.47/3.85sigma, so we're within this tolerance.

Note that this test becomes more stringent with number of effective samples (effectively number of pixels in the image). So these tests on 7m data should really be the lower limits on how strict the tests are.

Changes:

• Added functions to casaMaskingRoutines to calculate and mask noise-only channels
• Add task_mask_noise_channels function to handlerImagingChunked to mask noise channels, post-dirty beam but pre-multiscale
• Add skip_multiscale_if_mask_empty to handlerImagingChunked, which mimics the existing skip_singlescale_if_mask_empty. True by default
• Add task_mask_noise_channels function to handlerImaging to mask noise channels, post-dirty beam but pre-multiscale
• Update CHANGELOG.md

[codecov]

Codecov Report

❌ Patch coverage is 7.98122% with 196 lines in your changes missing coverage. Please review.
✅ Project coverage is 7.96%. Comparing base (6670982) to head (8b5c6da).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff            @@
##           master    #399      +/-   ##
=========================================
- Coverage    7.96%   7.96%   -0.01%     
=========================================
  Files          42      42              
  Lines       16615   16820     +205     
  Branches     3978    4017      +39     
=========================================
+ Hits         1323    1339      +16     
- Misses      15269   15458     +189     
  Partials       23      23              

Components	Coverage Δ
casa	5.03% <9.02%> (+0.10%)	⬆️
handler	6.31% <5.79%> (+<0.01%)	⬆️
spectral-cube	10.14% <ø> (ø)
utils	13.59% <ø> (ø)

[thomaswilliamsastro]

@low-sky I've added in some numbers here to try and quantify some worst-case outliers we see, and how they compare to what we'd expect just given a random noise field. The tl;dr here is that within the noise of the cube, these two methods are identical