diff --git a/.github/actions/rust-cache/action.yml b/.github/actions/rust-cache/action.yml index e966411006d..436560e1276 100644 --- a/.github/actions/rust-cache/action.yml +++ b/.github/actions/rust-cache/action.yml @@ -59,6 +59,13 @@ runs: echo "AWS_SECRET_ACCESS_KEY=$SECRET" echo "MBX_REMOTE_S3_ENDPOINT=https://dd618f5dbd5da77b8296f1613c301f5c.r2.cloudflarestorage.com" echo "MBX_REMOTE_S3_REGION=auto" + # The action store's size budget also caps what one build downloads, + # and a hosted disk scales it to 5 GiB on Linux and 10 on Windows. + # Prefetching predictions every job shares spends that early, so the + # job's own results then fail to restore, compile again, and re-upload + # objects the bucket already holds, each a refused but billed put. + # 20 GiB covers the most any job downloaded with the cap lifted. + echo "MBX_GC_MAX_SIZE=20GiB" if [ "$WRITE" = "true" ]; then echo "MBX_REMOTE_MODE=read-write" # mbx trusts only a push to a protected branch, which leaves out diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 74840485230..05329a8f14f 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -49,7 +49,8 @@ Decisions (maintainer, 2026-10-13): queue, and no separate warm rebuilds what landed. Since [#6040](https://github.com/moq-dev/moq/pull/6040) the queue runs on the self-hosted `moq-queue` runner with mbx's local store, so the R2 writers are - the hosted Nightly jobs on `main`. + the hosted scheduled jobs on `main`: Nightly and the workflows it calls, + CodSpeed, Interop and TS. - Self-hosted CI is dropped: hosted runners plus the R2 caches replace the spare-desktop runner, its cache server, and the cache release it waited on. Reopened 2026-10-19, conditionally, to compare runners once the pipeline diff --git a/quest/m0/ci/mbx-failed-puts.md b/quest/m0/ci/mbx-failed-puts.md index 1da1f34b548..48b413cb425 100644 --- a/quest/m0/ci/mbx-failed-puts.md +++ b/quest/m0/ci/mbx-failed-puts.md @@ -1,91 +1,79 @@ -# [M] The Rust cache stops paying for uploads R2 refuses +# [S] The Rust cache stops paying for uploads R2 refuses ## Goal `mbx-moq-dev` takes measurably fewer Class A requests per scheduled night, mostly by failing fewer conditional puts with HTTP 412. Each hosted Rust-cache -job keeps its hit rate within its night-to-night spread. A root fix is -proposed upstream in jdx/mr-boxington so that mbx stops issuing puts it can -tell will fail. +job keeps or raises its hit rate. A root fix is proposed upstream in +jdx/mr-boxington so that mbx stops issuing puts it can tell will fail. ## Plan -Measured 2026-11-13 in a read-only Cloudflare audit of Oct 10 to Nov 9. R2 -cost $38.35 (Class A $22.50, Class B $8.64, storage $7.21), almost all of it -`mbx-moq-dev` (998 GB, 501k objects). That bucket served 5.65M Class A -requests and 24.9M Class B. 4.80M of the Class A requests failed with HTTP -412. mbx uploads each output with a conditional put (`If-None-Match`), and 85% -of those puts fail because the object already exists. R2 still bills each one -as Class A, about $21 a month. The audit window straddles the 30 → 14-day -expiry change of 2026-11-07, and a shorter expiry removes some 412s: an -expired object's put succeeds instead, as a paid upload. So size the -workaround and the upstream fix from nights after 2026-11-07 only, as -[mbx-r2-verify](/quest/m0/ci/mbx-r2-verify.md) splits its measurement. A -shorter expiry still leaves every repeat upload of an object that is alive. +Found 2026-11-13 in [#8631](https://github.com/moq-dev/moq/pull/8631), from +the bucket's per-object request analytics and each job's mbx summary for +scheduled Nightly run +[43250182795](https://github.com/moq-dev/moq/actions/runs/43250182795) +(2026-11-12, the first night with only scheduled writers). That night's +Nightly, CodSpeed, Interop and TS took 163,302 puts, 151,043 of them 412. +Nightly bursts on the earlier nights since the 14-day expiry ran 143k to 158k +412s each. -The audit named the writers as the hosted jobs on `main`: Nightly, CodSpeed, -and `platform.yml` on push. Checked against `main` on 2026-11-13, -`platform.yml` runs on push only for `release`, where the jobs only read, -since `.github/actions/rust-cache` writes only on `refs/heads/main`. On `main` -it runs only inside Nightly. So every writer is scheduled or dispatched: -Nightly and the workflows it calls, plus CodSpeed, all at 07:00 UTC, then -`interop.yml` and `ts.yml` at 08:00. Jobs that share a runner namespace start -together, so the likely cause is several jobs compiling the same crates at -once, with each one uploading them. This is unconfirmed. Other possible causes -are mbx re-uploading outputs it restored or prefetched, or several keys -sharing one stored object. Find out which from mbx's upload path and from -hourly request counts before you pick the writer set, since the cause decides -the fix. +- 99% of the 412s hit objects stored before that night, so jobs racing to + create the same object is not the cause. +- mbx caps one build's remote downloads at its action store's size budget, + `gc.max_size`, which a hosted disk scales to 5 GiB on Linux and 10 GiB on + Windows. 39 of the 41 Rust-cache jobs reached it, mostly on prefetch: the + prediction manifest is keyed by the workspace's `Cargo.lock`, so every job + on a platform prefetches from one shared set. Past the cap a restore fails + as "cached action is missing blob", the unit compiles again, and mbx + re-publishes the identical result: every blob and the action result, each + a conditional put. All 128 blobs the doctest job reported missing were in + the bucket, and their only puts that night were 412s. +- The budget refused 0.6 to 11.75 GiB more per job. macOS, whose default + budget it never reached, logged no refusals and hit the most. -Decided 2026-11-13 (maintainer, /quest-plan): +Decided 2026-11-13 (maintainer, on #8631): ✅ lift the download budget only, +as #8631 does with `MBX_GC_MAX_SIZE=20GiB` on hosted runners, the most any +job asked for (rejected: lifting it and also cutting writers; cutting writers +only, as first planned that day). Fewer writers would leave the refused +restores in place and stop storing the read-only jobs' own units. Whether to +cut writers is judged again once the remaining 412s are measured (below). -- ✅ A workaround here: fewer jobs write the remote cache. The decision named - writing only from Nightly instead of every push to `main`. Since no push - writes now, choose whichever writer set cuts the 412s and measurably keeps - hit rates. Candidates are one writer per runner namespace and workload, - staggered schedules, or read-only CodSpeed, interop and ts jobs. - Recommended: add a write opt-in input to the rust-cache action, set it only - on the chosen jobs, and drop the `cache-write` Environment from the jobs - that only read, so fewer jobs hold the write key. Update the action's - "Hosted runs on `main` write" comment, `infra/README.md`'s "Hosted jobs on - `main` write what they compile", and the - [CI epic](/quest/m0/ci/README.md)'s record of who writes. -- ✅ The root fix belongs upstream in mbx: before the conditional put, check - whether the object exists with a HEAD (Class B), or skip outputs mbx already - knows are stored. jdx/mr-boxington is outside the maintainer's orgs. Its - CONTRIBUTING.md, read 2026-11-13, allows AI-assisted Discussions, Issues and - pull requests that say AI contributed. It sends proposed changes to - Discussions first and closes bug reports that came from reading the source - rather than from using mbx, so report from these measurements. Re-read it - and its AGENTS.md before drafting. If they now ban LLM-written posts, draft - nothing and say so in the PR. Otherwise draft the post (or the pull - request) in this quest's PR body and add a `Maintainer:` item under - Waiting to approve posting it. An agent could post it, but the maintainer - approves each post outside their orgs. -- Rejected: only shortening the expiry, which leaves the failed writes alone - (shortened to 7 days on 2026-11-12 for storage). +Decided 2026-11-13 (maintainer, on #8631): ✅ post the drafted report +upstream (rejected: not posting). Posted as +[jdx/mr-boxington#950](https://github.com/jdx/mr-boxington/discussions/950) +after its CONTRIBUTING.md and AGENTS.md, re-read that day, allowed +AI-assisted Discussions from real use. It asks mbx to name a budget refusal +as such, not to re-send objects of a result the remote already holds, and to +keep speculative prefetch from spending the budget the build's own restores +need. Add the after numbers there once measured. -Measure before and after over enough scheduled nights to separate the change -from the spread between nights: +Measure the first scheduled nights after #8631 against 2026-11-12. The +7-day expiry was applied that same day, after the 14-day rule of 2026-11-07, +so the baseline night may predate its first deletions while every later +night runs under it; note that beside the comparison: -- Class A requests by action and response status, and 412s, per hour. Use - Cloudflare's GraphQL `r2OperationsAdaptiveGroups`, filtered on `bucketName`, +- Class A requests by action and response status, and 412s, per hour, from + Cloudflare's GraphQL `r2OperationsAdaptiveGroups` filtered on `bucketName`, with the maintainer's wrangler login as - [mbx-r2-verify](/quest/m0/ci/mbx-r2-verify.md) does. -- Hits, misses and not looked up for each Rust-cache job, from its mbx cache - summary. + [mbx-r2-verify](/quest/m0/ci/mbx-r2-verify.md) does. Per-object counts + (`objectName`) need one-minute windows to stay under the 10,000-row limit. +- Each Rust-cache job's hits, misses and not looked up, and any "download + budget exceeded" or "cached action is missing blob" warnings, from its log. +- Disk: no hosted job fails for space with the larger store. -An expired object turns a 412 into a successful upload. Compare nights under -one lifecycle window, or note the -2026-11-12 expiry change between them. - -If the PR lands before upstream fixes this, rewrite this quest to what is -left: the posting approval and the upstream fix. Plan the mbx bump that adopts -the fix as its own quest. That quest re-measures, and if the workaround cost -hits, it restores the writers it dropped. +If 412s stay high once restores succeed, size the remaining sources (repeat +stores of units mbx did not look up, and blobs every job's session sends once) +and judge again whether cutting writers is worth the hits it would cost. Plan +the mbx bump that adopts an upstream fix as its own quest. Public API: none. Wire: none. +## Waiting + +- GitHub Actions: the first two scheduled Nightly runs after #8631 merges + supply the after numbers above. + ## Related -- [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - its 2026-11-26 bucket measurement and its Windows report to mbx overlap this one's numbers and upstream contact +- [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - its Windows misses may share this cause, since Windows also reached its 10 GiB budget