Add explicit TNM elevation product discovery - #397
Conversation
There was a problem hiding this comment.
I love this; adding docs for each module is a new goal. Then we can edit the module table to link to the specific module doc page!
| if products is not None: | ||
| if datasets is not None: | ||
| raise ValueError("Use either products or datasets, not both") | ||
| selected = products.split("/") if isinstance(products, str) else products |
There was a problem hiding this comment.
products is initialized above as Optional[str] = None but here we're supporting the option of it being a list else products list. So it might be nice to do something like products: Sequence[str] | str | None = None as the initialization so that we can input a list of values instead of just the string "/" format. Even better would be to use the existing fetchez.utils.parse_arg_to_list function here:
from fetchez.utils import parse_arg_to_list
self.products = set([x.lower() for x in parse_arg_to_list(products, str)])and that would handle a list of string as well as the / separated string. and cast the outputs as expected. e.g.:
>>> set([x.lower() for x in parse_arg_to_list("5m/1m/5m", str)])
['5m', '1m']
>>> set([x.lower() for x in parse_arg_to_list(["5m", "1m", "5m"], str)])
['5m', '1m']Though up to you really, just make sure the initialization of products can support lists as well; this will likely get flagged when we make mypy stricter.
7d16022 to
d822a54
Compare
|
@matth-love Updated products parsing to use the existing parse_arg_to_list utility. It now supports slash-separated strings and sequences while preserving requested order and ordered de-duplication. Focused TNM tests passed (33), the full Fetchez suite passed (184 passed, 2 skipped), and all pre-commit checks passed. |
2 participants
Summary
This separates the TNM discovery changes from #385, following Matt's suggestion to keep footprint reading and source selection in hooks and Globato.
The new
productsoption lets you ask for specific elevation products, such as1m/1_3as. Each returned file is labeled with its product, alongside the source ID, dates, metadata links, and project directory when available. Files with the same name from different URLs get separate download paths.If a product request fails or returns incomplete results, discovery stops and removes the entries collected during that run. This helps prevent a failed request from looking like an area has no higher-resolution data. Existing
datasetsqueries remain available.This PR does not read footprints or choose between overlapping sources. WESM project matching and the resolution hierarchy come next in Globato. No new dependencies are needed.
Tests
The repository tests passed, along with type checks and package checks. Pre-commit checks also passed.
The tests cover product selection, source metadata, files with matching names, pagination, and failed or incomplete requests. They use saved API-style responses rather than live USGS requests.
Source masks and spatial metadata still varied between unchanged-base runs, consistent with the Globato issue we discussed.
Newport exercises the existing workflow; the repository tests exercise the new product option. The dem-devel evidence contains the results and tested commits.
evidence_bundle.zip
🔍 Docs preview: https://fetchez--397.org.readthedocs.build/en/397/