From d82f1cd2ac14b0d4856d11154e2fd464b352edee Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 23:55:25 -0700 Subject: [PATCH 1/4] chore(quest): claim quest/m0/ci/mbx-r2-verify.md Quest-Claim: ec346152fae3d9ba From 8d55783a21dd27c1fb53551b10c88929415706f8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 23:58:25 -0700 Subject: [PATCH 2/4] ci: checkpoint hosted R2 manifest verification failure Co-Authored-By: GPT-6 --- quest/m0/ci/mbx-r2-verify.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/quest/m0/ci/mbx-r2-verify.md b/quest/m0/ci/mbx-r2-verify.md index b257bd8a28d..f446efbfb4b 100644 --- a/quest/m0/ci/mbx-r2-verify.md +++ b/quest/m0/ci/mbx-r2-verify.md @@ -24,6 +24,41 @@ closes this: - Bucket size and request counts after a month, against the 30-day expiry in `infra/mbx/lifecycle.json`. Shorten or lengthen it from that. +## Plan + +Verification checkpoint, 2026-10-21 (Pacific): + +- `cache-write` admits `gh-readonly-queue/main/*`. Current queue jobs use the + self-hosted runner's local cache; hosted R2 measurements below come from + scheduled Nightly [39600361153](https://github.com/moq-dev/moq/actions/runs/39600361153). +- [Windows](https://github.com/moq-dev/moq/actions/runs/39600361153/job/119223136733) + and [macOS](https://github.com/moq-dev/moq/actions/runs/39600361153/job/119223136736) + both report a compatible R2 namespace and `configured read-write, effective + read-write`. This proves credentials and policy, not cache reuse. +- macOS repeatedly refuses task manifest + `a90076a403e4add7587d176ba0f95ec1b79b991c730c07b9b9602c4662a627f4`: + `remote cache action manifest declared 17204435 bytes, over the 16777216 + byte limit`. All eleven object-cache summaries report `0 B downloaded`; + they include local hits and remote failures. The platform job succeeds, + but this run does not establish remote reuse or a timing improvement. +- The pinned mbx 1.25.0 bounds remote JSON at 16 MiB in + [`mbx-cache-core/src/lib.rs`](https://github.com/jdx/mr-boxington/blob/v1.25.0/crates/mbx-cache-core/src/lib.rs#L88). + Its [prediction pruning](https://github.com/jdx/mr-boxington/blob/v1.25.0/crates/mbx-cache-core/src/agent/manifest.rs#L81) + bounds entry count, so a manifest can still exceed the serialized-byte + limit. Fix producer sizing upstream before treating these results as a + cache pass; preserve the reader's bound rather than raising it to fit one + observed payload. Confirm the saved manifest before attributing every + remote miss to this refusal. +- No open upstream issue or PR identified this failure on 2026-10-21. + [mbx 1.26.0](https://github.com/jdx/mr-boxington/releases/tag/v1.26.0) + does not list a manifest fix. Posting to `jdx/mr-boxington` needs maintainer + approval under this repository's external-post policy; no report was sent. + +Remaining verification: remote hits across consecutive hosted Nightlies, +release PR read-only policy, a comparable before/after build measurement, +and bucket/request/expiry measurements after a full month. Keep this quest +until those observations and the manifest failure are resolved. + ## Related - [Require the release gate](/quest/m0/ci/release-gates.md) - the macOS and Windows jobs this cache serves From 0dc40f2ee6d1d3e837ed1e0a61dc6584bb207e4f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 09:00:09 -0700 Subject: [PATCH 3/4] ci: track the upstream mbx manifest fix as a pin bump jdx/mr-boxington#792 is fixed by jdx/mr-boxington#798, which no release carries yet. Split the pin bump into its own waiting quest and rewrite the verification quest to the evidence and the remaining measurements. Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/README.md | 3 +- quest/m0/ci/mbx-bump.md | 36 +++++++++++++++++++ quest/m0/ci/mbx-r2-verify.md | 68 +++++++++++++++++------------------- 3 files changed, 70 insertions(+), 37 deletions(-) create mode 100644 quest/m0/ci/mbx-bump.md diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 3fee91ca6af..d8401a348f6 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -82,6 +82,7 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Require the release gate](/quest/m0/ci/release-gates.md) - condition: the next cut carries `release-gate.yml`; then release PRs require it - [Share local check capacity](/quest/m0/ci/local-gate.md) - gate expensive phases and bound internal fanout -- [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - condition: the maintainer bootstrap; then measure the hosted Nightly jobs' hits, restore time, and expiry +- [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the Rust cache's pin +- [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands - [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm CI and releases use `nix.moq.dev` and retire Cachix - [First shared FFI release](/quest/m0/ci/nightly-ffi-reuse.md) - condition: the first `moq-ffi-v*` tag that runs `Release FFI`, and a `Shipping sizes` run; verify the wrappers and size keys diff --git a/quest/m0/ci/mbx-bump.md b/quest/m0/ci/mbx-bump.md new file mode 100644 index 00000000000..7803c99c437 --- /dev/null +++ b/quest/m0/ci/mbx-bump.md @@ -0,0 +1,36 @@ +# [XS] Bump the Rust cache to the mbx manifest-size fix + +## Goal + +Condition: an mbx release contains +[jdx/mr-boxington#798](https://github.com/jdx/mr-boxington/pull/798), the fix +for [jdx/mr-boxington#792](https://github.com/jdx/mr-boxington/issues/792). +It merged after v1.27.0 was tagged; the open release PR +[jdx/mr-boxington#797](https://github.com/jdx/mr-boxington/pull/797) proposes +it as 1.27.1. A tag carries the fix when +`gh api repos/jdx/mr-boxington/compare/...9c8d3c7eee60e5fedb56fd4a0f07bd0caf8df249 --jq .status` +prints `behind` or `identical`. + +Then set `version:` in `.github/actions/rust-cache/action.yml` to that +release, past the 1.25.0 hold from +[#6515](https://github.com/moq-dev/moq/pull/6515), and replace the hold's +comment. The release also carries +[jdx/mr-boxington#789](https://github.com/jdx/mr-boxington/pull/789) (in +1.27.0), the suspected cause of 1.26's empty `.rmeta` files. After it lands, +the merge queue shows no `error[E0463]: can't find crate` and the next +scheduled Nightly's macOS job logs no new `over the 16777216 byte limit` +digest. + +## Plan + +#798 caps a task manifest at a 15 MiB serialized budget below the reader's +16 MiB bound, but does not repair manifests already oversized on R2. Those +stay unreadable and unwritable, since the conditional PUT fails, until the +30-day rule in `infra/mbx/lifecycle.json` deletes them. If task identities +carry over to the new version, expect refusals for the three known macOS +digests until about 2026-11-22 (`a90076a4...`, `5cb8fb79...`, `b0310ce4...`); +a refusal for any other digest means the fix did not hold. + +## Related + +- [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measures hosted reuse once this lands diff --git a/quest/m0/ci/mbx-r2-verify.md b/quest/m0/ci/mbx-r2-verify.md index 7111645d378..3da87ec9b70 100644 --- a/quest/m0/ci/mbx-r2-verify.md +++ b/quest/m0/ci/mbx-r2-verify.md @@ -2,11 +2,7 @@ ## Goal -Condition: the `mbx-moq-dev` bootstrap in `infra/README.md` is complete and -scheduled Nightly runs on `main` have built Rust since. Check it with -`gh run list --workflow nightly.yml --branch main`. - -Then confirm the cache earns its keep for the hosted jobs that still use R2 +Confirm the cache earns its keep for the hosted jobs that still use R2 (Nightly on `main`, release pull requests, and the macOS and Windows jobs), and record the numbers in the PR that closes this: @@ -22,6 +18,9 @@ and record the numbers in the PR that closes this: - Bucket size and request counts after a month, against the 30-day expiry in `infra/mbx/lifecycle.json`. Shorten or lengthen it from that. +Use scheduled Nightly runs on `main` only (`gh run list --workflow +nightly.yml --branch main`), never a dispatch on a branch. + ## Plan Decided 2026-10-22 (maintainer): narrowed to hosted jobs. Since @@ -29,37 +28,34 @@ Decided 2026-10-22 (maintainer): narrowed to hosted jobs. Since self-hosted `moq-queue` runner with mbx's local store, so it neither reads nor writes R2. -Verification checkpoint, 2026-10-21 (Pacific): - -- Hosted R2 measurements below come from scheduled Nightly - [39600361153](https://github.com/moq-dev/moq/actions/runs/39600361153). -- [Windows](https://github.com/moq-dev/moq/actions/runs/39600361153/job/119223136733) - and [macOS](https://github.com/moq-dev/moq/actions/runs/39600361153/job/119223136736) - both report a compatible R2 namespace and `configured read-write, effective - read-write`. This proves credentials and policy, not cache reuse. -- macOS repeatedly refuses task manifest - `a90076a403e4add7587d176ba0f95ec1b79b991c730c07b9b9602c4662a627f4`: - `remote cache action manifest declared 17204435 bytes, over the 16777216 - byte limit`. All eleven object-cache summaries report `0 B downloaded`; - they include local hits and remote failures. The platform job succeeds, - but this run does not establish remote reuse or a timing improvement. -- The pinned mbx 1.25.0 bounds remote JSON at 16 MiB in - [`mbx-cache-core/src/lib.rs`](https://github.com/jdx/mr-boxington/blob/v1.25.0/crates/mbx-cache-core/src/lib.rs#L88). - Its [prediction pruning](https://github.com/jdx/mr-boxington/blob/v1.25.0/crates/mbx-cache-core/src/agent/manifest.rs#L81) - bounds entry count, so a manifest can still exceed the serialized-byte - limit. Fix producer sizing upstream before treating these results as a - cache pass; preserve the reader's bound rather than raising it to fit one - observed payload. Confirm the saved manifest before attributing every - remote miss to this refusal. -- No open upstream issue or PR identified this failure on 2026-10-21. - [mbx 1.26.0](https://github.com/jdx/mr-boxington/releases/tag/v1.26.0) - does not list a manifest fix. Posting to `jdx/mr-boxington` needs maintainer - approval under this repository's external-post policy; no report was sent. - -Remaining verification: remote hits across consecutive hosted Nightlies, -release PR read-only policy, a comparable before/after build measurement, -and bucket/request/expiry measurements after a full month. Keep this quest -until those observations and the manifest failure are resolved. +Evidence so far, from the platform jobs of scheduled Nightlies +[39600361153](https://github.com/moq-dev/moq/actions/runs/39600361153) +(2026-10-21, mbx 1.25.0), +[39779729461](https://github.com/moq-dev/moq/actions/runs/39779729461) +(2026-10-22, 1.25.0) and +[39951870847](https://github.com/moq-dev/moq/actions/runs/39951870847) +(2026-10-23, 1.26.0): + +- The bootstrap is done: Windows and macOS report `configured read-write, + effective read-write`. That proves credentials and policy, not reuse. +- macOS refuses task manifests written past the reader's 16 MiB bound + (`a90076a4...` at 17204435 bytes, then `5cb8fb79...` and `b0310ce4...`), so + those tasks get no remote predictions. No macOS summary downloads more than + 15 KiB. Reported as + [jdx/mr-boxington#792](https://github.com/jdx/mr-boxington/issues/792) and + fixed by [jdx/mr-boxington#798](https://github.com/jdx/mr-boxington/pull/798), + which keeps the read bound and caps the writer instead. No release carries + it yet; [the bump](/quest/m0/ci/mbx-bump.md) takes it. +- Windows logs no refusal but still no reuse. Its first build reports 0 hits + and 1084 of 1091 compilations "not looked up" for want of a matching + prediction on 2026-10-22, after a 1.25.0 Nightly wrote the day before, and + 1091 of 1098 on 1.26.0. The manifest fix does not explain this. If it + persists after the bump, find why before timing builds. +- No release pull request has run with the R2 cache yet. + +## Required + +- [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the pin ## Related From 4e51aed8f259b1005f88842d57c66882408ba430 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 09:47:23 -0700 Subject: [PATCH 4/4] quest: count only scheduled Nightly runs as R2 cache evidence Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/mbx-r2-verify.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quest/m0/ci/mbx-r2-verify.md b/quest/m0/ci/mbx-r2-verify.md index 3da87ec9b70..27e4b3860c5 100644 --- a/quest/m0/ci/mbx-r2-verify.md +++ b/quest/m0/ci/mbx-r2-verify.md @@ -19,7 +19,7 @@ and record the numbers in the PR that closes this: `infra/mbx/lifecycle.json`. Shorten or lengthen it from that. Use scheduled Nightly runs on `main` only (`gh run list --workflow -nightly.yml --branch main`), never a dispatch on a branch. +nightly.yml --branch main --event schedule`), never a dispatch on a branch. ## Plan