[benjaminleighton]

Improves on and replaces fork pull request #15

pybomwater/bom_water.py:60 - BomWater now accepts rate-limit, timeout, retry, and sample-size settings. - Uses a persistent requests.Session with a versioned User-Agent. - Adds request throttling, timeouts, exponential backoff, Retry-After support, and retries for HTTP 429/5xx. - Cache lookup now prefers the user cache, then falls back to bundled package cache files. - Cache initialization downloads only whichever files are missing. - Chunk-size estimation now samples short time windows across the requested range, extrapolates their counts, and falls back to the previous feature-sampling approach on errors. - values_count() now counts measurements across all observation blocks rather than only the first.

pybomwater/cache/waterML_GetCapabilities.json

New untracked, valid JSON capabilities cache, approximately 195 KB.
Intended to allow startup without downloading capabilities immediately.
pyproject.toml:8 and setup.py:41 - Package cache/*.json files in both wheels and source distributions.

Test changes
test/test_core.py:113

Adds tests for bounded sampling windows, long-range sampling, counting multiple observation blocks, and sample-based chunk sizing.
pytest.ini - Registers a live_bom pytest marker.

test/conftest.py

Marks known live BoM integration tests and skips them by default.
They run only when PYBOMWATER_RUN_LIVE=1.
README.md:35 - Documents how to run the live integration tests.

Tests are included for progressive load testing up to 20 years across 40 gauges and optionally running notebooks

[benjaminleighton]

@fre171csiro as discussed this is a local branch now and implements progressively load testing and notebook tests (all of which are passing)

[benjaminleighton]

A deterministic comparison uses 40 stations, daily Water Course Discharge,
and a ten-year period from 2015-01-01 to 2025-01-01. The final response contains
146,120 observations. The figures below are response-body bytes from the
loopback SOS simulator; HTTP headers are not included.

Phase	origin/main requests / bytes	Feature branch requests / bytes
Feature lookup	1 / 2,025	1 / 2,025
Chunk-size estimation	2 / 818,888	5 / 65,840
Final observation download	1 / 16,369,648	1 / 16,369,648
Warm-query total	4 / 17,190,561	7 / 16,437,513

The warm query therefore downloads 753,048 fewer bytes, a 4.38% reduction.
The final dataset is identical on both main and this branch; the saving comes from chunk
estimation. origin/main downloads 7,306 preliminary observations by sampling
two stations over the complete history. This feature branch downloads 400 by
sampling all 40 stations in five distributed two-day windows. This is a 94.5%
reduction in preliminary observations and a 92.0% reduction in preliminary
response bytes. It replaces two large sampling requests with five small ones,
so it reduces transferred data and individual response size but increases the
number of HTTP transactions.

Cold construction has a much larger effect. On an empty user cache,
origin/main makes two additional BoM requests for capabilities and all station
metadata; this feature branch makes zero because it falls back to packaged
metadata. The repository snapshot contains 124,564 stations: stations.json
is 59,588,688 bytes and waterML_GetCapabilities.json is 195,098 bytes. These
files total approximately 59.8 MB after installation and compress to about
3.2 MB for distribution. The processed cache size is not the exact size of the
source SOS XML, so it should be treated as an indication of scale rather than a
measured wire-byte saving. It nevertheless shows why avoiding a repeated
runtime cache build removes substantially more BoM load than the 4.38% warm
query saving alone suggests.

This feature branch also changes how transient load is handled. In the fixed
failure experiment, every third physical request returns HTTP 503:

Revision	Logical successes	Logical failures	Physical requests
origin/main	7	3	10
Feature branch	10	0	14

Production defaults space requests by at least one second and retry connection
errors, timeouts, HTTP 429, 500, 502, 503, and 504 up to three times. Numeric
Retry-After is honoured; otherwise retries use exponential backoff with
jitter. The test uses a zero interval and Retry-After: 0 to remain fast, so it
demonstrates recovery and request counts rather than production timing.

Fallback behaviour further limits unnecessary API access:

• cache resolution prefers an existing user cache, then packaged metadata, and
  downloads only files that are still missing;
• if distributed time-window sampling cannot be used, chunk estimation falls
  back to individual-feature sampling over the requested period; and
• exhausted retries raise an HTTP error containing the response status and a
  bounded portion of the response body.

Overall, thisfeature branch should place less byte and peak-response load on
the BoM API, especially for first use and long historical queries, while making
more small HTTP requests during estimation and more physical attempts when a
transient failure is successfully recovered.

[fre171csiro]

I have checked out your branch locally and successfully ran the all tests, also checked out the notebook repo and updated the dependencies to use your pybomwater branch. Ran all notebooks with success! Nothing that I think you have done already. Thank you for this Ben, great to have your contribution and I am sure it will improve the usability, smoothing out the issues with requests to the BoM api