SRE-946: Report why a mise download failed in .config/mise/install.sh - #9212
Conversation
The install script gave no usable diagnostic when a Vercel build failed in it:
`-f` discarded GitHub's response body, so a 4xx surfaced as a bare
`curl: (22)` with no status or URL, and the `curl | grep | sha256sum -c -`
pipeline reported a non-checksum body as a checksum parse failure.
Route both downloads through a `fetch` helper that drops `-f`, records
`%{http_code}` and the response headers, and on failure prints the URL,
curl's exit code with a label, the HTTP status, any rate-limit headers and
the first 2000 bytes of the body. Add retries and timeouts so a transient
drop is retried rather than failing the build, and send an `Authorization`
header when `GITHUB_TOKEN`/`MISE_GITHUB_TOKEN` is set, since curl does not
read those variables itself.
The checksum entry is now grepped into a file before verification, so a
truncated or error-page response is reported as such. Verification itself
is unchanged.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9212 +/- ##
=======================================
Coverage 59.63% 59.63%
=======================================
Files 1420 1421 +1
Lines 138767 138770 +3
Branches 6555 6555
=======================================
+ Hits 82753 82755 +2
- Misses 54950 54951 +1
Partials 1064 1064 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 15.38%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | bit_matrix/dense/iter_row[64] |
140.8 ns | 170 ns | -17.16% |
| ❌ | bit_matrix/dense/iter_row[200] |
185.8 ns | 215 ns | -13.57% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/sre-946-mise-install-error-output (2e363a5) with main (7409d46)
`--retry-delay 2` does not bound anything on the failure this change exists for. curl honours a server `Retry-After` in preference to `--retry-delay`, and GitHub answers a rate limit with `Retry-After: 60`, so the five retries stalled each download for five minutes — silently, because `-s` suppresses curl's "Will retry" notices and `-S` only re-enables hard errors. `--max-time 300` caps a single attempt, not the series. Measured against a stub returning 429 with `Retry-After: 60`: six attempts at t=0,60,120,180,240,300s, 300.3s wall per `fetch`. Add `--retry-max-time 60` so the series gives up after a minute (measured: 60.0s, two attempts). `--dump-header` appends across retries and redirect hops, so a real 429 printed the same five-line `x-ratelimit-*` block once per attempt — six repetitions of the detail the report is meant to make readable. Replace the `grep` with an awk pass that resets on each status line and prints only the headers that followed the last one.
`-S` already makes curl print its own diagnostic for a transport failure, so the label table restated it: a connection failure emitted both `curl: (7) Failed to connect to ... Couldn't connect to server` and `curl exit code: 7 (could not connect to host)`. On the HTTP-error path curl exits 0 and the label was the vacuous "no transport error". Keep the bare numeric exit code, which curl does not always make obvious, and drop the table.
The `nick-fields/retry` wrapper on `Install Rust` never protected the download it was added for. `idiomatic_version_file_enable_tools = ["rust"]` makes `mise install` read `rust-toolchain.toml`, and mise's `core:rust` plugin installs the toolchain by shelling out to `rustup` instead of fetching anything itself, so the toolchain and its components are downloaded during the earlier, unwrapped `jdx/mise-action` step. Attempt 1 of `Package (@rust/hash-codec)` on this branch died exactly there: a 63s TCP connect timeout to `static.rust-lang.org` fetching `rustfmt-nightly-x86_64-unknown-linux-gnu`, after which `Install Rust` was skipped and never got to retry. mise's own `http_retries` cannot cover this — it applies to mise's HTTP client, and rustup does its own downloading in a subprocess. Setting `install: false` and running `mise install` under the wrapper does not work either: the action saves its tool cache only inside the branch that runs the install, so that would leave the 543 MB cache permanently unwritten on a cache miss. Instead let the action install and cache as before, mark it non-fatal, and retry `mise install` in a following step. The retry is skipped on the happy path, and its condition tests `!= 'success'` so an unresolved `outcome` retries rather than swallowing the failure.
`continue-on-error` on `jdx/mise-action` absorbs every failure mode, not only the rustup connect timeout it was added for. When the action fails during its own setup, `mise` is never on `PATH` and the retry step spends five attempts and ~4 minutes of `retry_wait_seconds` on `mise: command not found` — less legible than the failure it replaced. Guard the retry with an explicit check on the same condition.
The reasoning behind `--retry-max-time`, the `--dump-header` awk pass, the cross-host redirect and the `continue-on-error` retry is all in the PR body; restating it in the files put them well above the comment density of the surrounding repo.
PR SummaryCursor Bugbot is generating a summary for commit 2e363a5. Configure here. |
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |
Requested by Tim Diekmann · Slack thread
🌟 What is the purpose of this PR?
Before. When a Vercel build for
apps/hash-frontendorapps/petrinaut-websitefailed inside.config/mise/install.sh, the log said one of two things and nothing more:No URL, no HTTP status code, no response body, no rate-limit headers — nothing to tell you which of the two downloads broke or why.
After. The same failure prints what actually happened:
and a checksum file that isn't a checksum file is named as such rather than being reported as a parse error:
On top of that, transient failures are now retried instead of failing the build outright, and the fetches authenticate when a GitHub token is present.
How. Both downloads go through one
fetchhelper. It deliberately does not pass--fail, because-fis what threw GitHub's response body away in the first place; instead it records--write-out '%{http_code}'and--dump-header, checks the status itself, and on failure prints the URL, curl's exit code, the HTTP status, the final response'sretry-after/x-ratelimit-*headers and the first 2000 bytes of the body. It adds--retry 5 --retry-all-errors --retry-delay 2 --retry-max-time 60 --connect-timeout 10 --max-time 300, and sendsAuthorization: Bearer …whenMISE_GITHUB_TOKENorGITHUB_TOKENis non-empty — curl does not read those variables on its own, so nothing was authenticating them before. Unauthenticated installs keep working unchanged. The checksum step greps its entry into a file before runningsha256sum -con it, so a truncated or error-page response is diagnosed as a bad download rather than as a checksum parse failure.This is not a confirmed root-cause fix. Rate limiting fits the evidence (unauthenticated fetches, a 6.6h burst of failures at 25% vs a 1.3% baseline, HTTP-error exit codes) but was never confirmed: the Vercel build logs are unreadable with the credentials available, and no rate-limit headers were ever captured — precisely because the script discarded them. The change is to make the next failure legible, so the cause can be read off the build log rather than inferred.
🔗 Related links
.config/mise/install.sh🚫 Blocked by
🔍 What does this change?
Mostly
.config/mise/install.sh—MISE_VERSION, the tar/extract step and everything else in that file are untouched. One related CI fix in.github/actions/install-tools/action.ymlrides along; it is the last bullet below.fetchhelper for both release-asset downloads. Keeps-sS -L, drops-f, adds--write-out '%{http_code}',--dump-headerand--output, and reports failure itself so the body survives to be printed.--retry 5 --retry-all-errors --retry-delay 2 --retry-max-time 60 --connect-timeout 10 --max-time 300. curl already treats 408/429/5xx as transient, so a rate-limited or flapping response is retried;--retry-all-errorsextends that to transport failures such as exit 56.--retry-max-time 60is what actually bounds the wait — see Known issues. There was no existing precedent in the repo for different values (the only other retry logic,.github/actions/install-tools/install-rust.sh, wrapsrustuprather than curl).--dump-headerappends across retries and redirect hops, so grepping the file printed the samex-ratelimit-*block once per attempt — six repetitions on a real 429, which defeats the readability this PR is for. An awk pass resets on eachHTTP/status line and prints only what followed the last one.-Salready makes curl print its own message for a transport failure, so the label restated it; on the HTTP-error path curl exits 0 and the label was the vacuous "no transport error". Dropped.Authorization: Bearer $MISE_GITHUB_TOKEN, falling back to$GITHUB_TOKEN, sent only when one of them is non-empty. Nothing is said or done when both are unset.MISE_GITHUB_TOKENtakes precedence to match mise's own resolution order.github.com302s release assets torelease-assets.githubusercontent.com; plain-LdropsAuthorizationon a cross-host redirect, and only--location-trustedwould forward it. Verified empirically against the curl in use rather than taken on trust — see the test note below.grep -E "^[0-9a-f]{64} (\./)?${tarball}$" SHASUMS256.txt > "${tarball}.sha256"first; if nothing matches, an explicit error names the tarball and shows the head of what was actually received. Thensha256sum -cruns against that file. Verification semantics are unchanged — the checksum is still enforced and a mismatch still fails.(\./)?is deliberate future-proofing, and it does not weaken anything. The realSHASUMS256.txtfor v2026.7.14 uses<sha> ./mise-v2026.7.14-linux-x64.tar.gz, so the old\./-requiring pattern is correct today; mise's own installer greps far more loosely and does not depend on the prefix, so the old pattern was tighter than upstream's actual contract and would break silently if mise ever switched fromsha256sum ./*tosha256sum *. The new pattern also anchors both ends and requires a 64-hex-digit hash, which the old substring match did not.tr -c '[:print:]\n\t' '.'before printing, so a half-downloaded tarball cannot spray control characters into the build log.mise installis now retried in.github/actions/install-tools/action.yml. Unrelated to the Vercel script, but the same class of bug and it broke this PR's own CI. Thenick-fields/retrywrapper sits onInstall Rust, yet the rustup download it protects is triggered earlier:idiomatic_version_file_enable_tools = ["rust"]makesmise installreadrust-toolchain.toml, and mise'score:rustplugin installs the toolchain by shelling out torustuprather than fetching anything itself. Attempt 1 ofPackage (@rust/hash-codec)died exactly there — a 63s TCP connect timeout tostatic.rust-lang.orgfetchingrustfmtinside the unwrappedjdx/mise-actionstep, after whichInstall Rustwas skipped and never got to retry. mise's ownhttp_retriescannot cover it (that setting applies to mise's HTTP client; rustup downloads in a subprocess), andinstall: falseplus a wrappedmise installis not viable either — the action saves its ~543 MB tool cache only inside the branch that runs the install, so that would leave the cache permanently unwritten on a miss. Instead the action step is marked non-fatal and a following step retriesmise install, skipped on the happy path. A guard step in between fails immediately ifmiseis not onPATHat all —continue-on-errorabsorbs a setup failure just as readily as the rustup one, and retryingmise installfive times without amisebinary would only printcommand not found.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
Note:
.config/mise/**is a global turbo input, so this PR busts the remote cache for its own CI run. That is expected and does not persist past merge.GITHUB_TOKENin the Vercel project environments is not part of this PR — it is a dashboard change, not a repo change. Until that happens the Vercel builds stay unauthenticated; the script handles that fine, it just does not get the higher rate limit. Worth doing separately if the improved output confirms rate limiting.--retry-max-time 60, not by--retry-delay. An earlier revision of this PR claimed--retry-delay 2gave a bounded worst case of ~10s of waiting. That was wrong: curl honours a serverRetry-Afterin preference to--retry-delay, and GitHub answers a rate limit withRetry-After: 60— precisely the scenario these retries exist for. Measured against a stub returning 429 withRetry-After: 60, the original flags made six attempts at t=0, 60, 120, 180, 240, 300s — 300.3s perfetch, roughly ten minutes across the two downloads, and silent throughout.--max-time 300does not help; it caps a single attempt, not the series. With--retry-max-time 60the same stub now gives up after 60.0s and two attempts. When the server dictates no delay the fixed 2s still applies and all five retries fit inside the window, so transient blips are unaffected.-sis kept deliberately, so curl's progress meter cannot end up in build logs — but it also suppresses curl'sWarning: Transient problem … Will retry in N secondsnotices, and-Sonly re-enables hard errors. Swapping-sfor--no-progress-meter(curl ≥ 7.67) would surface the retry notices without the progress bar. Not done here, to keep this change to the diagnostics it is about; worth considering as a follow-up.continue-on-erroron themise installstep trades fail-fast for retries. A genuinely broken mise config no longer fails on the first attempt; it fails after five, with 60s between them. The guard step only catches a missingmisebinary, not a config that makes every attempt fail identically. Left as is for consistency: the retry count and wait match the existingInstall Rustwrapper in the same action.--retry-all-errorsrequires curl ≥ 7.71. Vercel's Amazon Linux 2023 build image ships curl 8.x, so this is fine there, but it is a constraint if the script is ever run somewhere much older.🐾 Next steps
vercel-install.shfailure and read the printed status/headers. If it is a 429/403 withx-ratelimit-remaining: 0, addGITHUB_TOKENto both Vercel projects and this is closed out..config/mise,.lefthook.ymlorlint.yml). Adding one would be a reasonable small follow-up, given how much build-critical logic lives in.shfiles.🛡 What tests cover this?
None automated — the repo has no test harness for shell scripts. Verified manually; see below.
❓ How to test this?
bash -npasses, andshellcheck 0.11.0reports zero findings at default severity (-o allproduces only twoSC2312infos, both on pre-existing lines).Exercised against a local stand-in for the GitHub release endpoint:
curl exit code: 0,http status: 404and the JSON body. Exit 1.Retry-After: 60andx-ratelimit-*→ one copy of the five-line rate-limit block (six before the awk change), the JSON body, and the whole series bounded at 60.0s. Exit 1.SHASUMS256.txtbody is an HTML error page →error: no checksum entry for mise-v…tar.gz in …, followed by the received HTML. Exit 1.curl exit code: 7andhttp status: 000, alongside curl's own(7) Failed to connect … Couldn't connect to server. Exit 1.--fail../mise-v2026.7.14-linux-x64.tar.gz: FAILED+WARNING: 1 computed checksum did NOT match, exit 1. Verification is genuinely still enforced.SHASUMS256.txt, both pulled from GitHub) → checksumOK, extract succeeds,mise --versionprints2026.7.14 linux-x64. Run both with and without a token; theAuthorizationheader was observed present in the first case and absent in the second.jdx/miseproduced exactly the intended output —http status: 403plus the proxy's explanatory JSON body. Onmainthat same failure would have printedcurl: (22)and nothing else.All of the above were re-run after the
--retry-max-time, awk and exit-label changes; the checksum path still givesOKfor a valid tarball andFAILEDwith a non-zero exit for a tampered one.Also confirmed with the real
SHASUMS256.txt: the new pattern matches exactly one line for bothlinux-x64andlinux-arm64, identical to the old pattern's behaviour.And the cross-host token claim, tested rather than assumed (curl 8.5.0, redirect from
localhostto127.0.0.1):curl -L -H 'Authorization: …'curl --location-trusted -H 'Authorization: …'Bearer SECRET