Retry a failed run report without waiting for another file (watcher 1.2.2)

[wasimxyz]

Closes #245. Supersedes #246.

Problem

When the watcher's first report of a run to Data Hub fails because of a short connection problem, the run stays unreported until another file in that run finishes writing, or until the watcher restarts. A single-file run, or a run whose last file just finished, can wait for days. In automatic mode the file is not uploaded either.

What changed

The first commit is the fix from #246 by @RowletCC, with their authorship kept. It retries a failed report on the monitor's existing once-per-second loop, so no new file or restart is needed. The later commits tidy and extend it.

• A failed report is retried after 5 seconds, then after waits that double up to 5 minutes. It keeps retrying every 5 minutes until the report goes through, so an outage of any length clears on its own.
• Connection errors, 5xx responses, and any other error the client raises (for example a connection that drops mid-response, or a proxy's HTML page instead of JSON) are retried, on the first attempt as well as on retries. A 4xx needs a person to fix the cause and is never retried. An error after the server accepted the report, such as a local database write failing, is not a failed report and is handled as before.
• run_report_failed is recorded once for a row of retried failures, even if more files in the run finish during the outage, so a long outage doesn't fill the 500-event queue. A 4xx is still recorded every time. The heartbeat's error count still counts every attempt.
• A new file during an outage is still sent right away, and a failure there keeps the run's backoff instead of restarting it.
• A failed update to an existing run is retried the same way. In automatic mode that update also decides whether the new file gets uploaded.
• Removed the 429 and Retry-After handling from Retry transient watcher run reports without new file events #246. Data Hub never sends either to the watcher.
• ApiError.is_transient holds the "no response or 5xx" rule, and the uploader now uses it too.
• The retry delays are named in constants.py, the monitor hook is called on_tick, and the tests live in test_run_detector.py and test_monitor_events.py.

Replaying a report is safe: the server inserts a run only when its instrument and run ID are new, sends notifications only for new runs, and skips files it already has.

Release

• Watcher 1.2.2, with uv.lock, the CLI catalog, and the docs bundle regenerated.
• Added a changelog entry, a release table row, a troubleshooting section, and developer docs for the retry behavior.
• The changelog file is dated today. Rename it if this merges on a later day.
• Tagging watcher-v1.2.2 and setting Latest version in Settings → Watchers happen after this reaches production.

How I tested it

• All 507 watcher unit tests pass, and make check passes, which includes docs-links.
• I broke these behaviors on purpose and a test failed for each: reporting a 4xx during retries, catching only ApiError on the first attempt, and restarting the backoff when a new file arrives.
• Before the last commit, I ran a watcher against the local dev server through a small proxy and a throwaway instrument. I stopped the proxy, dropped a one-file run into the watched folder, and watched the retries at 5 and 10 seconds. I restarted the proxy before the third retry, and the run appeared on its own with no restart and no second file. The server's event log held exactly one run_report_failed event. I deleted the throwaway rows afterward. The last commit doesn't change that path.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
data-hub	Ready	Preview	Oct 16, 2026 9:24pm UTC
data-hub-docs	Ready	Preview	Oct 16, 2026 9:24pm UTC