Skip to content

Batching: Automate time period selection + combine trajectories - #274

Open
sjavis wants to merge 7 commits into
batching-registryfrom
batching-buffer-period
Open

sjavis wants to merge 7 commits into
batching-registryfrom
batching-buffer-period

Conversation

@sjavis

@sjavis sjavis commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator
  1. This combines the output files from each batch into one single netcdf file. Since each file starts from a trajectory index of zero, these get re-indexed, then the trajectories are padded to match in length, and combined them with cf.aggregate. Closes Combine batch outputs #247

    • The netcdf variable names are used to match variables in cf.aggregate because the default method fails due to the start and end flags both having the same standard_name.
    • It also fixes a mistake in the padding in to_netcdf which was causing issues with the concatenation.
  2. This adds the time period and batch interval as arguments to batching. It automatically subspaces the input files to the relevant time period and saves them to the batch directory. There are options to say a file doesn't vary in time or should be kept in memory for preprocessing. Closes Time-based selection of inputs for batching #192

  3. It also enables setting buffer periods with the buffer_period and start_buffer_period options to BatchingConfig. When combining the output files it will only keep the tracks that start within the main batch periods. Closes Combine batching iterations using buffer periods #193

    • start_buffer_period is set to a day by default because if it is in any of that period then it is assumed to be in the previous batch. But if buffer_period is not set it is not used.

This replaces the PRs #249 and #261 as there was a lot of overlap in the code changes.

This will also close #216 as batching will now handle missing output files itself.

@sjavis
sjavis force-pushed the batching-buffer-period branch from 958022e to 5cdaf09 Compare October 14, 2026 12:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant