Read remote raster footprints with a manifest hook - #395
Conversation
There was a problem hiding this comment.
Same as other pr regarding a new hook directory in the docs. You don't have to change anything; I can do this later.
|
|
||
| with requests.Session() as session: | ||
| for _, entry in entries: | ||
| with HttpFile(entry["url"], session=session) as remote: |
There was a problem hiding this comment.
This is where I'd probably make a small change. I recently merged #400 to add a new helper to core.py gdal_vsi_path. I think if you use that instead of HttpFile in this case it will perform better. I'd have to test, but I think using HttpFile here may inadvertently load the entire file into memory rather than what you expect, as right below this you're sending the resulting python file-object to rasterio. If instead you use the new gdal_vsi_path here you can open that directly in rasterio to get the result you want and it's also a little more explicit i think regarding the functionality here. Something you might want to add to the test, instead of assert requests you might want to assert that transferred bytes were substantially less than the raster size.
Here, you might consider something like:
with rasterio.Env(
GDAL_DISABLE_READDIR_ON_OPEN="TRUE",
):
with rasterio.open(gdal_vsi_path(entry["url"])) as source:instead. Worth testing this though either way.
a932a0d to
138480a
Compare
|
Updated based on your feedback @matth-love: Rebased onto current main, including #400. Focused tests passed (11), and the full Fetchez suite and pre-commit checks passed. The dem-devel Newport comparison was inconclusive because the unchanged baseline differed between repeated runs; the evidence did not attribute that instability to this candidate. |
Summary
This pulls the raster footprint code out of #385 into a reusable hook, as Matt suggested.
remote_raster_footprintreads enough of a remote raster to determine its location and outer boundary without downloading the whole file. It adds that outline to the Fetchez entry so later hooks can compare coverage. The outline includes any blank areas inside the raster.Other modules serving compatible rasters can use this hook too. This keeps footprint reading separate from TNM discovery and the rules for choosing between overlapping sources.
This hook requires Rasterio (
pip install rasterio).Tests
All six new hook tests passed, along with the full repository suite: 131 passed and 2 skipped. The tests use real TIFFs served over local HTTP and cover different coordinate systems, rotated rasters, NoData-only rasters, missing CRS information, antimeridian rejection, and use with
spatial_cull. Pre-commit checks also passed.The Newport comparison produced identical DEM elevations and hillshade across all three runs. Source masks and spatial metadata still varied between unchanged-base runs, consistent with the Globato issue we discussed.
The dem-devel evidence bundle includes the test results, exact commits, and Newport comparison.
evidence_bundle.zip
🔍 Docs preview: https://fetchez--395.org.readthedocs.build/en/395/