Skip to content

chore: various speedups and QOL improvements - #25

Merged
jeffseif merged 7 commits into
mainfrom
jeffseif/various-fixups
Nov 2, 2026
Merged

jeffseif merged 7 commits into
mainfrom
jeffseif/various-fixups

Conversation

@jeffseif

@jeffseif jeffseif commented Nov 2, 2026

Copy link
Copy Markdown
Member

Background

Some of these were on my list from earlier and a couple were discovered during the global backfill for FDaP.

The changes

  1. a8c0a1c add sharding to jdluc.ingest so we can get simple, non-colliding concurrency across processes or VM's. No tests necessary
  2. 848db32 switch to GDAL's drivers for COG conversion. Verified that outputs are identical and observed a 1.4x speedup on a full FDaP tile
  3. 72ce2f3 guard against OOM's when ingesting at a high concurrency (either with threads or processes). Observed a large decrease in memory pressure with no slowdown during the FDaP backfill
  4. fec51c2 in harmonize, batch all bands for a dataset/tile into one VRT. Verified that outputs are identical and observed a 6x speedup on the LUC stack (with all 25 GACED30 bands). Harmonize was one of the heaviest stages during the v202610 backfill, so this is a well-received speedup
  5. 80f22e9 make --help commands more legible. jdluc.ingest was especially hard to read now that the number of datasets listed is ~30
  6. 38e8ca9 update docs to reflect these changes

Notes

  1. All timings were performed on an M4 laptop, so although we expect speedups to be universal the degree may differ
  2. Most of the effort here was spent ensuring that results matched before/after
  3. No cache bumps or backfills are necessary since all outputs are identical

@jeffseif
jeffseif requested review from blallen and danohu November 2, 2026 19:51
Comment thread jdluc/ingest.py Resolved
Comment thread jdluc/harmonize.py Resolved

@blallen blallen left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found two potential bugs with codex, up to you if you want to address them

@jeffseif
jeffseif merged commit 97a3621 into main Nov 2, 2026
2 checks passed
@jeffseif
jeffseif deleted the jeffseif/various-fixups branch November 2, 2026 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants