Skip to content

fix: Retry the test harness download and surface curl errors - #411

Merged
kinyoklion merged 1 commit into
v2from
devin/1786560023-downloader-retry
Aug 12, 2026
Merged

fix: Retry the test harness download and surface curl errors#411
kinyoklion merged 1 commit into
v2from
devin/1786560023-downloader-retry

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Aug 12, 2026

Copy link
Copy Markdown
Member

Requirements

  • I have added test coverage for new or changed functionality
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions

Related issues

Intermittent Download failed failures in SDK contract-test jobs, e.g. launchdarkly/python-server-sdk#484 where 4 of 5 matrix jobs died at the same second while the rest passed on identical code.

Describe the solution you've provided

downloader/run.sh resolves the release version through the GitHub API (authenticated when GITHUB_TOKEN is set), but then downloads the release asset with a single un-retried curl --fail -s, and -s hides the reason for a failure:

-curl --fail -s -L -o "${TEMP_DIR}/archive.${EXTENSION}" "${DOWNLOAD_URL}" || (echo "Download failed" >&2; exit 1)
+curl --fail -sS -L --retry 5 --retry-delay 2 \
+  -o "${TEMP_DIR}/archive.${EXTENSION}" "${DOWNLOAD_URL}" \
+  || { echo "Download failed" >&2; exit 1; }

--retry covers transient 5xx/408/429 responses (consistent with several matrix jobs failing at once while others succeed), and -sS means the next failure reports the actual curl error instead of a bare Download failed.

Describe alternatives you've considered

  • --retry-all-errors: rejected, it requires curl 7.71+ and this script also runs on older macOS runners.
  • Sending the token on the asset download: the asset URL redirects to an unauthenticated object store, so an Authorization header there is at best useless and can break the redirect.

Additional context

The contract-tests GitHub action reads this script from the v2 branch by default, so this also covers repos running the v3 harness through that action. main has the same line and should get the same change.

Link to Devin session: https://app.devin.ai/sessions/bfe54128e2804a96bb100e6120e9a3ef
Requested by: @kinyoklion


Note

Overview
Contract-test jobs intermittently failed with a bare Download failed when fetching the sdk-test-harness release asset from GitHub.

The download curl in downloader/run.sh now uses --retry 5 --retry-delay 2 for transient HTTP failures (aligned with matrix jobs failing together while others pass) and -sS instead of -s so failures include the actual curl error message.

Reviewed by Cursor Bugbot for commit 2c23020. Bugbot is set up for automated code reviews on this repo. Configure here.

Co-Authored-By: rlamb@launchdarkly.com <4955475+kinyoklion@users.noreply.github.com>
@kinyoklion kinyoklion self-assigned this Aug 12, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot added the devin-pr PRs created by Devin label Aug 12, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

@cursor review

@kinyoklion
kinyoklion marked this pull request as ready for review August 12, 2026 20:27
@kinyoklion
kinyoklion requested a review from a team as a code owner August 12, 2026 20:27
@devin-ai-integration

Copy link
Copy Markdown
Contributor

More evidence for this: I pushed six SDK PRs within a few seconds to validate v2.39.0 / v3.2.0-alpha.6 end-to-end, and nearly every contract-test job failed on Download failed between 21:55 and 21:58 UTC — python-server-sdk#484, go-server-sdk#430, js-core#1849, php-server-sdk#265, cpp-sdks#594 — for both the v2.39.0 and v3.0.0-alpha.6 assets. Re-downloading the same URLs from another host right after returns HTTP 200, so the asset endpoint is throttling concurrent downloads and one un-retried curl is enough to kill the job.

The jobs that did get the binary confirm the new test works end-to-end, e.g. go with sdk-test-harness v2.39.0:

[hooks/evaluation/provides the environment ID]
[hooks/evaluation/provides the environment ID/default data source]
[hooks/evaluation/provides the environment ID/polling]
All tests passed

v3 counterpart of this change: #413.

@kinyoklion
kinyoklion merged commit 2a7533b into v2 Aug 12, 2026
8 checks passed
@kinyoklion
kinyoklion deleted the devin/1786560023-downloader-retry branch August 12, 2026 22:11
kinyoklion added a commit that referenced this pull request Aug 13, 2026
**Requirements**

- [ ] I have added test coverage for new or changed functionality
- [x] I have followed the repository's pull request submission
guidelines
- [x] I have validated my changes against all supported platform
versions

**Related issues**

Same change as #411, on the v3 branch.

**Describe the solution you've provided**

The release-asset download is a single un-retried `curl --fail -s`, and
`-s` hides the reason for a failure:

```diff
-curl --fail -s -L -o "${TEMP_DIR}/archive.${EXTENSION}" "${DOWNLOAD_URL}" || (echo "Download failed" >&2; exit 1)
+curl --fail -sS -L --retry 5 --retry-delay 2 \
+  -o "${TEMP_DIR}/archive.${EXTENSION}" "${DOWNLOAD_URL}" \
+  || { echo "Download failed" >&2; exit 1; }
```

**Describe alternatives you've considered**

- `--retry-all-errors`: rejected, it requires curl 7.71+ and this script
also runs on older macOS runners.
- Sending the token on the asset download: the asset URL redirects to an
unauthenticated object store, so an `Authorization` header there is at
best useless and can break the redirect.

**Additional context**

Fresh evidence: six SDK PRs were pushed within a few seconds to validate
v2.39.0 / v3.2.0-alpha.6 end-to-end, and nearly every contract-test job
died on `Download failed` between 21:55 and 21:58 UTC
(python-server-sdk, go-server-sdk, js-core, php-server-sdk, cpp-sdks),
for both the `v2.39.0` and `v3.0.0-alpha.6` assets. Downloading the same
URLs afterwards succeeds with HTTP 200, so the asset endpoint was
throttling concurrent downloads.


Link to Devin session:
https://app.devin.ai/sessions/bfe54128e2804a96bb100e6120e9a3ef
Requested by: @kinyoklion

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Makes CI contract-test jobs more resilient when pulling
**sdk-test-harness** release archives via `downloader/run.sh`.
> 
> The release download `curl` now uses **`-sS`** so failures still print
curl diagnostics, and **`--retry 5 --retry-delay 2`** to ride out brief
GitHub/asset throttling (the motivation described in the PR). The
failure path is unchanged in behavior: still exits with **Download
failed**.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
c7e7f09. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
kinyoklion added a commit that referenced this pull request Aug 13, 2026
**Requirements**

- [ ] I have added test coverage for new or changed functionality
- [x] I have followed the repository's [pull request submission
guidelines](../blob/master/CONTRIBUTING.md#submitting-pull-requests)
- [x] I have validated my changes against all supported platform
versions

**Related issues**

Follows up on the CI investigation in #411 (v2), where contract tests
failed at the harness download step rather than at version resolution.

**Describe the solution you've provided**

Previously `GITHUB_TOKEN` was only used for the releases-listing request
in `resolve_version`, so any run that passed a full version string (as
CI usually does) made no authenticated requests at all, and the asset
download was always anonymous. The
`github.com/.../releases/download/...` path is not a REST endpoint:
GitHub publishes no rate limit for it and returns no `x-ratelimit-*`
headers, and passing a token there is useless because `curl -L` drops
`Authorization` on the cross-host redirect to the signed asset host.

This change routes the download through the documented release-assets
API when a token is present:

- `github_curl` centralizes auth (`Authorization: Bearer`) plus `-sS
--retry 5 --retry-delay 2`, replacing the previous `eval`-built command
string that interpolated the token into a shell command.
- `resolve_asset_url` looks up the asset for the current OS/arch via
`GET /releases/tags/<version>` and downloads it from `GET
/releases/assets/<id>` with `Accept: application/octet-stream`. That
lands in the documented authenticated rate limit bucket instead of an
undocumented one.
- Without a token — or if the asset lookup fails — it falls back to the
previous anonymous `releases/download` URL, so external consumers are
unaffected.
- `curl -s` became `-sS` so a failed download prints the actual curl
error/status instead of a bare "Download failed".

Docs referenced: [rate limits for the REST
API](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api),
[release assets
endpoints](https://docs.github.com/en/rest/releases/assets).

**Describe alternatives you've considered**

- Sending the token to the existing `releases/download` URL — verified
that this has no effect on the actual byte transfer, since the header is
dropped at the redirect.
- Retry-only (as merged for v2 in #411) — that surfaces the error and
rides out transient failures, but leaves the download in the
unauthenticated bucket.

**How to test it**

```sh
export VERSION=v3.0.0-alpha.6 PARAMS="-help"
GITHUB_TOKEN=<token> sh downloader/run.sh   # downloads from .../releases/assets/<id>
sh downloader/run.sh                        # falls back to .../releases/download/...
GITHUB_TOKEN=bogus sh downloader/run.sh     # prints the 401, then falls back
```

Also exercised the partial-version path (`VERSION=v2`), the
cached-binary path, and an unmatched version.

**Additional context**

No UI or screenshots apply; this is a shell script used by CI in the SDK
repos. Note that the contract-tests GitHub Action currently fetches
`run.sh` from the `v2` branch, so this only takes effect in CI once the
follow-up ports land on `v2` (and `v3`), which will be opened after this
is accepted.


Link to Devin session:
https://app.devin.ai/sessions/98e5024773d947509ad2575ae80bf31b
Requested by: @kinyoklion

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **`downloader/run.sh`** now uses **`GITHUB_TOKEN`** for **release
asset downloads** as well as version resolution, so CI contract tests
that pass a full version string still hit GitHub’s authenticated rate
limits instead of anonymous download URLs.
> 
> A shared **`github_curl`** helper sends **`Authorization: Bearer`**,
adds **`--retry`**, and replaces the old **`eval`**-built curl for
listing releases. **`resolve_version`** uses it and returns with
**`return`** instead of **`exit`** when the version is already complete.
> 
> When a token is set, **`resolve_asset_url`** resolves the OS/arch
tarball via **`GET /releases/tags/<version>`** and downloads from **`GET
/releases/assets/<id>`** with **`Accept: application/octet-stream`**,
avoiding the **`releases/download`** redirect that strips
**`Authorization`**. If there is no token or asset lookup fails,
behavior falls back to the public **`releases/download`** URL (and
clears auth on bogus-token fallback). Downloads use **`curl -sS`** so
failures surface curl errors.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
f3a35c2. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
tanderson-ld pushed a commit that referenced this pull request Aug 19, 2026
🤖 I have created a release *beep* *boop*
---


##
[2.40.0](v2.39.0...v2.40.0)
(2026-08-19)


### Features

* add RETRY-conformance contract tests for FDv1 streaming and polling
([#404](#404))
([ee4d85f](ee4d85f))


### Bug Fixes

* Retry the test harness download and surface curl errors
([#411](#411))
([2a7533b](2a7533b))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Release 2.40.0** bumps the package version in
`.release-please-manifest.json`, `main.go` (`versionString`), and
documents the release in `CHANGELOG.md`.
> 
> The changelog for this tag records two already-merged changes:
**RETRY-conformance** contract tests for FDv1 **streaming** and
**polling** (capability-gated, including long-running retry-regime
checks), and a **downloader** fix that **retries** harness downloads and
**surfaces curl errors** for easier CI debugging.
> 
> There is no additional functional code in this PR diff beyond the
version and changelog updates.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
b96a76c. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
kinyoklion added a commit that referenced this pull request Aug 25, 2026
**Requirements**

- [ ] I have added test coverage for new or changed functionality
- [x] I have followed the repository's [pull request submission
guidelines](../blob/master/CONTRIBUTING.md#submitting-pull-requests)
- [x] I have validated my changes against all supported platform
versions

**Related issues**

Port of #415 (merged to `main`) and #416 (`v2`) to `v3`. The
contract-tests action fetches `run.sh` from the `v2` branch today, so
this port keeps `v3` from regressing when it becomes the branch CI uses.

**Describe the solution you've provided**

Previously `GITHUB_TOKEN` was only used for the releases-listing request
in `resolve_version`, so any run that passed a full version string (as
CI usually does) made no authenticated requests at all, and the asset
download was always anonymous. The
`github.com/.../releases/download/...` path is not a REST endpoint:
GitHub publishes no rate limit for it and returns no `x-ratelimit-*`
headers, and passing a token there is useless because `curl -L` drops
`Authorization` on the cross-host redirect to the signed asset host.

This change routes the download through the documented release-assets
API when a token is present:

- `github_curl` centralizes auth (`Authorization: Bearer`) plus `-sS
--retry 5 --retry-delay 2`, replacing the previous `eval`-built command
string that interpolated the token into a shell command.
- `resolve_asset_url` looks up the asset for the current OS/arch via
`GET /releases/tags/<version>` and downloads it from `GET
/releases/assets/<id>` with `Accept: application/octet-stream`. That
lands in the documented authenticated rate limit bucket instead of an
undocumented one.
- Without a token — or if the asset lookup fails — the token is cleared
and it falls back to the previous anonymous `releases/download` URL, so
external consumers are unaffected.

Docs referenced: [rate limits for the REST
API](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api),
[release assets
endpoints](https://docs.github.com/en/rest/releases/assets).

**Describe alternatives you've considered**

- Sending the token to the existing `releases/download` URL — verified
that this has no effect on the actual byte transfer, since the header is
dropped at the redirect.
- Retry-only (#411 on `v2`) — rides out transient failures, but leaves
the download in the unauthenticated bucket.

**How to test it**

```sh
export VERSION=v3 PARAMS="-help"
GITHUB_TOKEN=<token> sh downloader/run.sh   # downloads from .../releases/assets/<id>
sh downloader/run.sh                        # falls back to .../releases/download/...
GITHUB_TOKEN=bogus sh downloader/run.sh     # prints the 401, then falls back anonymously
```

Also exercised full and partial version strings, the cached-binary path,
and an unmatched version.

**Additional context**

Shell-script-only change used by CI, so no UI screenshots apply. This
also brings the retry and error-surfacing behavior of #411 to `v3`,
which never received it. The Windows `zip` path is unchanged apart from
going through `github_curl`.


Link to Devin session:
https://app.devin.ai/sessions/98e5024773d947509ad2575ae80bf31b
Requested by: @kinyoklion

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **CI downloader** (`downloader/run.sh`) now uses `GITHUB_TOKEN` for
**release asset downloads**, not only when resolving partial version
tags from the releases list.
> 
> A shared **`github_curl`** helper sends **`Authorization: Bearer`**
with retries and replaces the old `eval`-built curl that only
authenticated the releases list—and did not help downloads when CI
passed a full version string.
> 
> When a token is set, **`resolve_asset_url`** fetches the OS/arch
archive via the **releases assets REST API** (`Accept:
application/octet-stream`), avoiding the public `releases/download` URL
where **`curl -L` drops auth on redirect**. If asset lookup fails or no
token is provided, behavior falls back to the anonymous download URL
(token cleared on fallback so a bad token does not break public
downloads).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
7a13940. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devin-pr PRs created by Devin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants