Add TNM elevation product and source-coverage support

[camante]

Summary

Add explicit USGS The National Map (TNM) elevation-product support for downstream DEM workflows.

The change adds TNM product discovery, authoritative source footprints, WESM project metadata for 1 m data, and the metadata needed for downstream resolution hierarchy and chronology decisions.

Existing TNM behavior remains available; the new provider behavior is opt-in.

What changed

• Add explicit TNM elevation products for 1 m and gridded National Elevation Dataset products.
• Preserve source-footprint coverage with each discovered source so downstream consumers can distinguish true coverage gaps from NoData inside a source.
• Preserve WESM project identity and chronology metadata for 1 m products.
• Propagate the common TNM metadata needed by downstream stream and provenance handling.
• Fail closed when authoritative source coverage required for hierarchy decisions is unavailable.
• Extend the existing Fetchez registry, stream initialization, and recipe machinery rather than adding a separate TNM workflow.

This PR intentionally stops at source discovery and metadata. Resolution precedence, project superseding, coastline policy, and final DEM stacking remain responsibilities of downstream consumers such as Globato.

Validation

Repository tests and pre-commit checks pass.

Targeted tests cover:

• TNM product discovery and aliases;
• WESM project identity and chronology metadata;
• source-footprint handling;
• raster-source coverage;
• metadata propagation through stream initialization;
• recipe/registry integration;
• fail-closed behavior when required source coverage is unavailable.

An existing Globato TNM workflow was also run as a no-regression check against the exact Fetchez candidate. The candidate matched the base output for that established workflow.

globato build \
  -R -118.65/-118.60/34.05/34.10 \
  -X 6:5 \
  -P epsg:4269+5703 \
  -E 0.1111111s \
  -O tnm_1m_palisades_smoke \
  -D '<RUN_OUTPUT>' \
  --shared-cache '<RUN_CACHE>' \
  tnm:datasets=2,weight=100

<RUN_OUTPUT> and <RUN_CACHE> are isolated per validation run.

Reproducibility

• Base: origin/main @ 2af18b88fb1c89c02763d4d069602e405cfd62d2
• Candidate: 867d7b778b15b5e80bd1f91a1bfc45804f378706
• Pre-commit: PASS
• Repository validation: PASS
• Existing-workflow integration check: PASS

The complete validation evidence, exact candidate patch, and resolved runtime commands are preserved in the associated dem-devel evidence bundle.

Supporting evidence

The attached evidence_bundle.zip contains the exact candidate patch, revision identities,
validation reports, resolved scientific commands, and supporting artifacts generated by
dem-devel for this reviewed candidate. It is provided for deeper review or reproduction;
the validation relevant to this PR is summarized above.
evidence_bundle.zip

---

Checklist

• PR title is descriptive
• PR body contains links to related and resolved issues (e.g. closes #1)
• If needed, CHANGELOG.md updated
• If needed, docs and/or README.md updated
• If needed, unit tests added
• All checks passing (comment pre-commit.ci autofix if pre-commit is failing)
• At least one approval

---

🔍 Docs preview: https://fetchez--385.org.readthedocs.build/en/385/

[matth-love]

Thanks for the great contributions! You have built some seriously powerful remote spatial parsing techniques here. I'm thinking it might be best though to break up this PR a bit into some separate atomic updates.

For some context, one of the things we're trying to do with the fetchez framework is enforce a "Lightweight Core, Heavy Extensions" modular architecture. The core fetchez framework and its modules/hooks should try to remain as generic and agnostic as possible; ideally (for modules) just scanning an API and returning fetchable endpoints, while domain-specific routing, heavy processing, and bespoke spatial intersections are handled by the Hook system or as a collection of modules and hooks in standalone extensions (like Globato) when necessary.

So, to get these features integrated without tightly coupling the core registry to USGS datasets and having processing code in the modules directory, I would suggest we break this up and create some cool new hooks, etc.:

1. The core.py HTTP Fixes (Immediate Merge)
   The HttpFile context manager updates for windowed/range reads, are fantastic. Could you pull the core.py updates into a standalone, atomic PR? Would love to merge that immediately, as it would immediately benefit all users and allow for some neat new hook features.

2. Refactoring modules/tnm_*.py into Generic Manifest Hooks
   The logic in tnm_raster.py, tnm_ned.py, and tnm_wesm.py is wonderful and shoulnd't be restricted just the tnm module. Since we're trying to build a modular generic system, hardcoding footprint generation at the end of the tnm module's run() method breaks the agnostic data-discovery pattern. Instead, these should be refactored into Manifest Hooks (hooks that run during the manifest stage (before they are donwloaded)).
   By making these into generic hooks, you keep tnm.py lightweight, and any module in the ecosystem serving COGs or archived shapefiles can use your code as generic hooks. Your Globato bundle would then just chain them in the normal fetchez fashion (e.g., fetchez run tnm --datasets 1_9as --hook remote_archive_footprint) to achieve the same outcome.

3. Registry and pipeline (Should be refactored)
   The updates to the registry and pipeline cli should probably be refactored into the module itself or be refactored as hooks (module arguments become cli/recipe arguments for the specific module automatically). Since these are domain-specific features, they should either be coupled with the specific module or split into hooks so we don't tie in the domain-logic just for the tnm module into the core registry.

How do you feel about possibly making these changes? The best next step would be to extract the core.py updates into their own PR. We can then port the tnm_*.py helpers into generic hooks that the tnm module (and any other module that wants to try) can use instead and think about ways to inject the registry/pipeline changes into the tnm module itself or as separate hooks. I think this would better follow the modular nature of fetchez while still getting the workflow you've provided here.

Would love to help with any of these suggestions, as these are some great features you've made.

[matth-love]

So, to follow my advise here, I removed the cull functionality from the dav module as well as the dedupe functionality from the tnm module and instead made a new hook called spatial_cull.py. This compares fetchez entries based on geometry and a key (default year) and culls overlapping items based on being sorted by the key (year by default). This replicates both the dedupe and cull logic from the tnm and dav modules and can now be used fetchez-wide by any module that supplies geometries to compare against. So something similar is what i'm suggesting for this PR and apologies for updating the tnm module here as you had touched that one quite a bit in this PR as well (though it's only removing the dedupe options)...see #390

[matth-love]

For example, the 'dedupe' in the tnm module was mainly to remove duplicated NED tiles since they just publish new ones alongside older ones; and now we can use the spatial_cull hook to do this:

$ fetchez run --global-hook list-only -R -124.5/-124.0/41.5/44.0 tnm --datasets 1_as --hook spatial_cull:sort_by="date" 
Executing dynamic pipeline...
[ INFO ] spatial_cull: [spatial_cull] Culled 20 redundant entries based on 'date'.
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1/TIFF/historical/n43w124/USGS_1_n43w124_20260818.tif
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1/TIFF/historical/n42w124/USGS_1_n42w124_20260601.tif
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1/TIFF/historical/n42w125/USGS_1_n42w125_20260601.tif
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1/TIFF/historical/n43w125/USGS_1_n43w125_20260601.tif
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1/TIFF/historical/n44w124/USGS_1_n44w124_20250804.tif
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1/TIFF/historical/n45w124/USGS_1_n45w124_20250804.tif
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1/TIFF/historical/n44w125/USGS_1_n44w125_20241001.tif
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1/TIFF/historical/n45w125/USGS_1_n45w125_20130911.tif

[camante]

thanks for the great feedback Matt! As you saw, I just did the HTTP PR. This is what I'm thinking about doing next... let me know if you have any thoughts!

1. Remote raster footprint hook: Read a remote raster’s outer extent through HttpFile and attach it to the manifest entry. Any module serving compatible rasters could use it.
2. Remote archive footprint hook: Read footprint features from remote archives, with options to select the layer and fields. Keep the archive-reading code reusable beyond TNM.
3. TNM discovery update: Keep tnm.py focused on finding products, URLs, dates, and WESM project identifiers. Use normal module arguments without TNM-specific registry or pipeline changes.
4. Globato hierarchy integration: Connect those pieces in glob-tnm, using higher-resolution source coverage to control fallback and preserving project provenance.