Skip to content

Add cycle-aware NCEP observation windows - #1012

Closed
NickGeneva wants to merge 6 commits into
NVIDIA:mainfrom
NickGeneva:codex/ufs-cycle-window-utils
Closed

Add cycle-aware NCEP observation windows#1012
NickGeneva wants to merge 6 commits into
NVIDIA:mainfrom
NickGeneva:codex/ufs-cycle-window-utils

Conversation

@NickGeneva

@NickGeneva NickGeneva commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stacked on: #1011

  • Route UFS observation task planning through the shared NCEP cycle-window utility.
  • Add a cycle_aware option to UFS, GDAS, NNJA conventional, and NNJA satellite observation sources.
  • Default cycle_aware=True excludes future cycle files relative to the request upper tolerance bound; cycle_aware=False retains retrospective reads that include the next cycle file when needed.
  • Add task-planning tests for cycle-aware and cycle-unaware behavior.

Closes: #967

Validation

  • uv run ruff check earth2studio/data/utils_ncep.py earth2studio/data/ufs.py earth2studio/data/gdas.py earth2studio/data/nnja.py test/data/test_utils_ncep.py test/data/test_ufs.py test/data/test_gdas.py test/data/test_nnja.py
  • uv run mypy --ignore-missing-imports earth2studio/data/utils_ncep.py earth2studio/data/ufs.py earth2studio/data/gdas.py earth2studio/data/nnja.py
  • uv run pytest test/data/test_ufs.py -q -m 'not slow'
  • Direct observation_cycle_times assertions for cycle-aware and cycle-unaware windows

Note: local GDAS/NNJA test files are dependency-gated in this environment (data/pybufrkit) and were skipped by pytest.

@NickGeneva
NickGeneva marked this pull request as ready for review July 24, 2026 04:55
@NickGeneva
NickGeneva force-pushed the codex/ufs-cycle-window-utils branch from cf99578 to b7ac02d Compare July 24, 2026 05:00
@NickGeneva

NickGeneva commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

@aayushg55 I changed my mind 😆 and just make cycle_aware always an option and have it on by default since that represents the observations available in the operational sense.

lmk your thoughts

@NickGeneva
NickGeneva requested a review from aayushg55 July 24, 2026 05:03
@greptile-apps

greptile-apps Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a cycle_aware flag (default True) to observation_cycle_times, cycle_windows, and all four NCEP observation sources (GDAS, NNJA conv/sat, UFS conv/sat), and also routes the UFS task-planning loop through the shared cycle_windows utility.

  • Semantic model change: cycle_aware=True (new default) treats each cycle file as covering observations starting at the file timestamp ([T, T+cadence)), so only files with timestamps <= tmax are fetched. cycle_aware=False preserves the legacy "end-of-window" behaviour where the next cycle file is also fetched.
  • UFS refactor: UFSObsConv._create_tasks and UFSObsSat._create_tasks now delegate cycle enumeration to cycle_windows, matching the existing GDAS/NNJA pattern; the previous hand-rolled while day <= tmax loop is removed.
  • Tests: New parametrised tests cover both modes for all four sources; the existing NNJA satellite test is updated to reflect the new default semantics.

Confidence Score: 4/5

The logic changes are internally consistent and all test assertions verify against the new cycle-selection semantics.

The core utility function and all four data-source integrations are well-tested and logically consistent. Inserting cycle_aware before cache in several constructors is a silent positional-argument hazard, and cycle_aware=False with a negative lower tolerance now downloads one extra cycle file compared to pre-PR. Neither issue causes data corruption or incorrect observations in practice.

earth2studio/data/utils_ncep.py (the cycle_aware=False + negative-tolerance path at line 1089) and earth2studio/data/ufs.py (constructor parameter ordering at lines 74-80).

Important Files Changed

Filename Overview
earth2studio/data/utils_ncep.py Adds cycle_aware parameter to observation_cycle_times and cycle_windows; changes default cycle-selection logic from end-of-window to start-of-window. The tolerance_lower >= timedelta(0) guard in the cycle_aware=False path is a deliberate semantic refinement.
earth2studio/data/ufs.py Routes UFSObsConv and UFSObsSat task planning through the shared cycle_windows utility, matching the pattern already used by GDAS/NNJA. Adds cycle_aware parameter to _UFSObsBase.__init__ and UFSObsSat.__init__.
earth2studio/data/gdas.py Minimal change: threads cycle_aware through _create_tasks to cycle_windows. Logic is correct and follows the new shared pattern.
earth2studio/data/nnja.py Adds cycle_aware to NNJAObsConv and NNJAObsSat; threads it through cycle_windows in _create_tasks. All three call sites updated consistently.
test/data/test_utils_ncep.py Renames and expands the cycle-times test to cover both cycle_aware=True and cycle_aware=False for each existing case.
test/data/test_ufs.py New parametrized test covers both UFSObsConv and UFSObsSat for cycle_aware=True and False with a single requested time. No test exercises multiple requested times sharing the same 6-hour cycle.
test/data/test_gdas.py Cycle-aware assertions added inline to the existing task-creation test. Correct assertions for both aware and unaware modes.
test/data/test_nnja.py New standalone test for NNJAObsConv cycle_aware selection; existing NNJAObsSat cycle-window test updated for new default semantics and extended with explicit cycle_aware=False assertions.

Reviews (1): Last reviewed commit: "Update changelog for cycle-aware observa..." | Re-trigger Greptile

Comment thread earth2studio/data/utils_ncep.py Outdated
Comment thread earth2studio/data/ufs.py
Comment thread test/data/test_ufs.py
@NickGeneva
NickGeneva force-pushed the codex/ufs-cycle-window-utils branch from eeb5325 to 400a5a3 Compare July 24, 2026 05:26
@NickGeneva
NickGeneva force-pushed the codex/ufs-cycle-window-utils branch from 400a5a3 to 40bd633 Compare July 24, 2026 05:33
@aayushg55

aayushg55 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Thanks @NickGeneva . I like the idea of the cycle_aware flag. But I think this doesn't fully resolve the issue as is. See some examples of how tolerances map to files under each setting of the flag and the actual desired file. While it is mostly okay, in some cases it reads additional files or still causes some overlap.
Screenshot 2026-07-24 at 11 41 54 AM

Maybe a simpler way to do this is the below:

if cycle_aware:
    require_cadence_aligned(time)  # Must be 00/06/12/18Z
    files = [time]                 # Select exactly that analysis file
    observation_window = (
        time + tolerance_lower,
        time + tolerance_upper,
    )

Where we separate the file selection (force the requested time to be cycle aligned so that it removes all ambiguity) and the time range of obs returned within that file.

@NickGeneva NickGeneva closed this Aug 24, 2026
@NickGeneva
NickGeneva deleted the codex/ufs-cycle-window-utils branch September 4, 2026 21:20
@NickGeneva
NickGeneva restored the codex/ufs-cycle-window-utils branch September 4, 2026 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛[BUG]: NCEP observation sources: divergent time-tolerance semantics and cycle-window file selection

2 participants