Skip to content

Cache sequential native preparation reads on the xarray handle - #6

Merged
yuyi13 merged 1 commit into
mainfrom
codex/sequential-preprocess-20261017
Oct 17, 2026
Merged

yuyi13 merged 1 commit into
mainfrom
codex/sequential-preprocess-20261017

Conversation

@yuyi13

@yuyi13 yuyi13 commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Daily native preparation read multi-year source chunks one day at a time through a small default cache, repeatedly decompressing the same spatial chunk slab. Apply the bounded per-variable cache to the actual NetCDF handle used by xarray. Preserve CF decoding and the existing daily reprojection, close handles on exit, and leave global defaults unchanged.

Validation:

  • Gadi job 181482217 passed 392 tests, package builds, the offline paired workflow and the installed-wheel demo.
  • A real 31-day comparison preserved values and metadata exactly: 26.45 seconds with the original reader versus 1.32 seconds with the cache.
  • Preparing one native 167 × 167 component across all 5,844 days took 62.35 seconds; peak parent memory was 2.24 GiB.
  • Tests cover packed values, missing values, classic NetCDF, selected-variable caches, unchanged global defaults and handle closure on success and failure.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T17:34:22.241345Z 4e6693e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@yuyi13
yuyi13 merged commit 90c802e into main Oct 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant