Skip to content

Read remote raster footprints with a manifest hook - #395

Merged
matth-love merged 4 commits into
mainfrom
feat/remote-raster-footprint
Sep 14, 2026
Merged

matth-love merged 4 commits into
mainfrom
feat/remote-raster-footprint

Conversation

@camante

@camante camante commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

This pulls the raster footprint code out of #385 into a reusable hook, as Matt suggested.

remote_raster_footprint reads enough of a remote raster to determine its location and outer boundary without downloading the whole file. It adds that outline to the Fetchez entry so later hooks can compare coverage. The outline includes any blank areas inside the raster.

Other modules serving compatible rasters can use this hook too. This keeps footprint reading separate from TNM discovery and the rules for choosing between overlapping sources.

This hook requires Rasterio (pip install rasterio).

Tests

All six new hook tests passed, along with the full repository suite: 131 passed and 2 skipped. The tests use real TIFFs served over local HTTP and cover different coordinate systems, rotated rasters, NoData-only rasters, missing CRS information, antimeridian rejection, and use with spatial_cull. Pre-commit checks also passed.

The Newport comparison produced identical DEM elevations and hillshade across all three runs. Source masks and spatial metadata still varied between unchanged-base runs, consistent with the Globato issue we discussed.

The dem-devel evidence bundle includes the test results, exact commits, and Newport comparison.
evidence_bundle.zip


🔍 Docs preview: https://fetchez--395.org.readthedocs.build/en/395/

@camante
camante requested a review from matth-love September 13, 2026 11:04

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as other pr regarding a new hook directory in the docs. You don't have to change anything; I can do this later.


with requests.Session() as session:
for _, entry in entries:
with HttpFile(entry["url"], session=session) as remote:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is where I'd probably make a small change. I recently merged #400 to add a new helper to core.py gdal_vsi_path. I think if you use that instead of HttpFile in this case it will perform better. I'd have to test, but I think using HttpFile here may inadvertently load the entire file into memory rather than what you expect, as right below this you're sending the resulting python file-object to rasterio. If instead you use the new gdal_vsi_path here you can open that directly in rasterio to get the result you want and it's also a little more explicit i think regarding the functionality here. Something you might want to add to the test, instead of assert requests you might want to assert that transferred bytes were substantially less than the raster size.

Here, you might consider something like:

with rasterio.Env(
    GDAL_DISABLE_READDIR_ON_OPEN="TRUE",
):
    with rasterio.open(gdal_vsi_path(entry["url"])) as source:

instead. Worth testing this though either way.

@camante
camante force-pushed the feat/remote-raster-footprint branch from a932a0d to 138480a Compare September 14, 2026 01:11
@camante

camante commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator Author

Updated based on your feedback @matth-love:

Rebased onto current main, including #400.
Replaced the HttpFile path with gdal_vsi_path.
Added GDAL_DISABLE_READDIR_ON_OPEN="TRUE" through rasterio.Env.
Strengthened the range-read test using a larger raster and a transferred-bytes threshold.
Updated the hook documentation.

Focused tests passed (11), and the full Fetchez suite and pre-commit checks passed. The dem-devel Newport comparison was inconclusive because the unchanged baseline differed between repeated runs; the evidence did not attribute that instability to this candidate.

@camante
camante marked this pull request as ready for review September 14, 2026 01:15
@matth-love
matth-love merged commit 770934a into main Sep 14, 2026
4 checks passed
@matth-love
matth-love deleted the feat/remote-raster-footprint branch September 14, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants