Repository navigation
ci(rust-cache): lift mbx's download budget so restores stop re-uploading #8631
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
9a64f70
chore(quest): claim quest/m0/ci/mbx-failed-puts.md
kixelated 1681aa7
ci(rust-cache): lift mbx's download budget so restores stop re-uploading
kixelated 2500b38
quest: record the budget and upstream decisions for mbx-failed-puts
kixelated 059db51
Merge remote-tracking branch 'origin/main' into quest/m0/ci/mbx-faile…
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,91 +1,79 @@ | ||
| # [M] The Rust cache stops paying for uploads R2 refuses | ||
| # [S] The Rust cache stops paying for uploads R2 refuses | ||
|
|
||
| ## Goal | ||
|
|
||
| `mbx-moq-dev` takes measurably fewer Class A requests per scheduled night, | ||
| mostly by failing fewer conditional puts with HTTP 412. Each hosted Rust-cache | ||
| job keeps its hit rate within its night-to-night spread. A root fix is | ||
| proposed upstream in jdx/mr-boxington so that mbx stops issuing puts it can | ||
| tell will fail. | ||
| job keeps or raises its hit rate. A root fix is proposed upstream in | ||
| jdx/mr-boxington so that mbx stops issuing puts it can tell will fail. | ||
|
|
||
| ## Plan | ||
|
|
||
| Measured 2026-11-13 in a read-only Cloudflare audit of Oct 10 to Nov 9. R2 | ||
| cost $38.35 (Class A $22.50, Class B $8.64, storage $7.21), almost all of it | ||
| `mbx-moq-dev` (998 GB, 501k objects). That bucket served 5.65M Class A | ||
| requests and 24.9M Class B. 4.80M of the Class A requests failed with HTTP | ||
| 412. mbx uploads each output with a conditional put (`If-None-Match`), and 85% | ||
| of those puts fail because the object already exists. R2 still bills each one | ||
| as Class A, about $21 a month. The audit window straddles the 30 → 14-day | ||
| expiry change of 2026-11-07, and a shorter expiry removes some 412s: an | ||
| expired object's put succeeds instead, as a paid upload. So size the | ||
| workaround and the upstream fix from nights after 2026-11-07 only, as | ||
| [mbx-r2-verify](/quest/m0/ci/mbx-r2-verify.md) splits its measurement. A | ||
| shorter expiry still leaves every repeat upload of an object that is alive. | ||
| Found 2026-11-13 in [#8631](https://github.com/moq-dev/moq/pull/8631), from | ||
| the bucket's per-object request analytics and each job's mbx summary for | ||
| scheduled Nightly run | ||
| [43250182795](https://github.com/moq-dev/moq/actions/runs/43250182795) | ||
| (2026-11-12, the first night with only scheduled writers). That night's | ||
| Nightly, CodSpeed, Interop and TS took 163,302 puts, 151,043 of them 412. | ||
| Nightly bursts on the earlier nights since the 14-day expiry ran 143k to 158k | ||
| 412s each. | ||
|
|
||
| The audit named the writers as the hosted jobs on `main`: Nightly, CodSpeed, | ||
| and `platform.yml` on push. Checked against `main` on 2026-11-13, | ||
| `platform.yml` runs on push only for `release`, where the jobs only read, | ||
| since `.github/actions/rust-cache` writes only on `refs/heads/main`. On `main` | ||
| it runs only inside Nightly. So every writer is scheduled or dispatched: | ||
| Nightly and the workflows it calls, plus CodSpeed, all at 07:00 UTC, then | ||
| `interop.yml` and `ts.yml` at 08:00. Jobs that share a runner namespace start | ||
| together, so the likely cause is several jobs compiling the same crates at | ||
| once, with each one uploading them. This is unconfirmed. Other possible causes | ||
| are mbx re-uploading outputs it restored or prefetched, or several keys | ||
| sharing one stored object. Find out which from mbx's upload path and from | ||
| hourly request counts before you pick the writer set, since the cause decides | ||
| the fix. | ||
| - 99% of the 412s hit objects stored before that night, so jobs racing to | ||
| create the same object is not the cause. | ||
| - mbx caps one build's remote downloads at its action store's size budget, | ||
| `gc.max_size`, which a hosted disk scales to 5 GiB on Linux and 10 GiB on | ||
| Windows. 39 of the 41 Rust-cache jobs reached it, mostly on prefetch: the | ||
| prediction manifest is keyed by the workspace's `Cargo.lock`, so every job | ||
| on a platform prefetches from one shared set. Past the cap a restore fails | ||
| as "cached action is missing blob", the unit compiles again, and mbx | ||
| re-publishes the identical result: every blob and the action result, each | ||
| a conditional put. All 128 blobs the doctest job reported missing were in | ||
| the bucket, and their only puts that night were 412s. | ||
| - The budget refused 0.6 to 11.75 GiB more per job. macOS, whose default | ||
| budget it never reached, logged no refusals and hit the most. | ||
|
|
||
| Decided 2026-11-13 (maintainer, /quest-plan): | ||
| Decided 2026-11-13 (maintainer, on #8631): ✅ lift the download budget only, | ||
| as #8631 does with `MBX_GC_MAX_SIZE=20GiB` on hosted runners, the most any | ||
| job asked for (rejected: lifting it and also cutting writers; cutting writers | ||
| only, as first planned that day). Fewer writers would leave the refused | ||
| restores in place and stop storing the read-only jobs' own units. Whether to | ||
| cut writers is judged again once the remaining 412s are measured (below). | ||
|
|
||
| - ✅ A workaround here: fewer jobs write the remote cache. The decision named | ||
| writing only from Nightly instead of every push to `main`. Since no push | ||
| writes now, choose whichever writer set cuts the 412s and measurably keeps | ||
| hit rates. Candidates are one writer per runner namespace and workload, | ||
| staggered schedules, or read-only CodSpeed, interop and ts jobs. | ||
| Recommended: add a write opt-in input to the rust-cache action, set it only | ||
| on the chosen jobs, and drop the `cache-write` Environment from the jobs | ||
| that only read, so fewer jobs hold the write key. Update the action's | ||
| "Hosted runs on `main` write" comment, `infra/README.md`'s "Hosted jobs on | ||
| `main` write what they compile", and the | ||
| [CI epic](/quest/m0/ci/README.md)'s record of who writes. | ||
| - ✅ The root fix belongs upstream in mbx: before the conditional put, check | ||
| whether the object exists with a HEAD (Class B), or skip outputs mbx already | ||
| knows are stored. jdx/mr-boxington is outside the maintainer's orgs. Its | ||
| CONTRIBUTING.md, read 2026-11-13, allows AI-assisted Discussions, Issues and | ||
| pull requests that say AI contributed. It sends proposed changes to | ||
| Discussions first and closes bug reports that came from reading the source | ||
| rather than from using mbx, so report from these measurements. Re-read it | ||
| and its AGENTS.md before drafting. If they now ban LLM-written posts, draft | ||
| nothing and say so in the PR. Otherwise draft the post (or the pull | ||
| request) in this quest's PR body and add a `Maintainer:` item under | ||
| Waiting to approve posting it. An agent could post it, but the maintainer | ||
| approves each post outside their orgs. | ||
| - Rejected: only shortening the expiry, which leaves the failed writes alone | ||
| (shortened to 7 days on 2026-11-12 for storage). | ||
| Decided 2026-11-13 (maintainer, on #8631): ✅ post the drafted report | ||
| upstream (rejected: not posting). Posted as | ||
| [jdx/mr-boxington#950](https://github.com/jdx/mr-boxington/discussions/950) | ||
| after its CONTRIBUTING.md and AGENTS.md, re-read that day, allowed | ||
| AI-assisted Discussions from real use. It asks mbx to name a budget refusal | ||
| as such, not to re-send objects of a result the remote already holds, and to | ||
| keep speculative prefetch from spending the budget the build's own restores | ||
| need. Add the after numbers there once measured. | ||
|
|
||
| Measure before and after over enough scheduled nights to separate the change | ||
| from the spread between nights: | ||
| Measure the first scheduled nights after #8631 against 2026-11-12. The | ||
| 7-day expiry was applied that same day, after the 14-day rule of 2026-11-07, | ||
| so the baseline night may predate its first deletions while every later | ||
| night runs under it; note that beside the comparison: | ||
|
|
||
| - Class A requests by action and response status, and 412s, per hour. Use | ||
| Cloudflare's GraphQL `r2OperationsAdaptiveGroups`, filtered on `bucketName`, | ||
| - Class A requests by action and response status, and 412s, per hour, from | ||
| Cloudflare's GraphQL `r2OperationsAdaptiveGroups` filtered on `bucketName`, | ||
| with the maintainer's wrangler login as | ||
| [mbx-r2-verify](/quest/m0/ci/mbx-r2-verify.md) does. | ||
| - Hits, misses and not looked up for each Rust-cache job, from its mbx cache | ||
| summary. | ||
| [mbx-r2-verify](/quest/m0/ci/mbx-r2-verify.md) does. Per-object counts | ||
| (`objectName`) need one-minute windows to stay under the 10,000-row limit. | ||
| - Each Rust-cache job's hits, misses and not looked up, and any "download | ||
| budget exceeded" or "cached action is missing blob" warnings, from its log. | ||
| - Disk: no hosted job fails for space with the larger store. | ||
|
|
||
| An expired object turns a 412 into a successful upload. Compare nights under | ||
| one lifecycle window, or note the | ||
| 2026-11-12 expiry change between them. | ||
|
|
||
| If the PR lands before upstream fixes this, rewrite this quest to what is | ||
| left: the posting approval and the upstream fix. Plan the mbx bump that adopts | ||
| the fix as its own quest. That quest re-measures, and if the workaround cost | ||
| hits, it restores the writers it dropped. | ||
| If 412s stay high once restores succeed, size the remaining sources (repeat | ||
| stores of units mbx did not look up, and blobs every job's session sends once) | ||
| and judge again whether cutting writers is worth the hits it would cost. Plan | ||
| the mbx bump that adopts an upstream fix as its own quest. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Waiting | ||
|
|
||
| - GitHub Actions: the first two scheduled Nightly runs after #8631 merges | ||
| supply the after numbers above. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - its 2026-11-26 bucket measurement and its Windows report to mbx overlap this one's numbers and upstream contact | ||
| - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - its Windows misses may share this cause, since Windows also reached its 10 GiB budget |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cap the download budget to the hosted runner disk.
MBX_GC_MAX_SIZE=20GiBis larger than the 14 GB SSD on standard GitHub-hosted runners. The previous budget scaled with disk size. A fixed 20 GiB value removes that limit for hosted jobs with R2 credentials. A restore larger than the free disk can then fill the disk during restore or build, and the job can fail.Measure peak free disk space on a hosted run with this value, using the largest workspace. Keep the value only if restored objects and build outputs fit in the free space. Otherwise, use a lower cap for hosted jobs.