From d3753edf00ecf819f46bb98e1356ff9faf51c9ee Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 02:22:44 -0800 Subject: [PATCH 1/3] chore(quest): claim quest/m0/backport-mkv-buffer.md Quest-Claim: c111042ca80c67ac From f201c8f9659786ac4a7d18c819b36e5be8fb1969 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 02:28:06 -0800 Subject: [PATCH 2/3] docs(quest): record release MKV buffering blocker Co-Authored-By: GPT-6.1 --- quest/m0/backport-mkv-buffer.md | 90 +++++++++++++++++++++------------ 1 file changed, 59 insertions(+), 31 deletions(-) diff --git a/quest/m0/backport-mkv-buffer.md b/quest/m0/backport-mkv-buffer.md index 0b76d9c765e..2d83dc65599 100644 --- a/quest/m0/backport-mkv-buffer.md +++ b/quest/m0/backport-mkv-buffer.md @@ -18,41 +18,69 @@ backport, m0: confirm with the 1 GiB repro on `release`, size bound only. Why (the recommendation's reasoning, from #7234's Impact): #7234 is a real memory-exhaustion fix, but the full change is a rewrite that also refuses inputs `release` accepts today (subtitle tracks, CMAF output, video `BlockGroup`s with fields after their `Block`). A size -bound fixes the exhaustion on `release` without changing what it imports. - -- **Confirm first**: port #7234's reproduction, - `ignored_payload_does_not_accumulate_under_a_large_metadata_master` (a - 1 GiB EBML header holding a nearly 1 GiB Void, fed in 1 KiB chunks), onto - `release` and show it grows memory there. If it doesn't, report that and - stop. -- **Where it buffers**: `release`'s - `rs/moq-mux/src/container/mkv/import.rs:139` builds ebml-iterable's - `TagDecoder` to collect the `Ebml`, `Info`, `Tracks`, `TrackEntry`, - `Audio`, `Video`, and `BlockGroup` masters whole, and `decode` (`:170`) - keeps every byte of an incomplete tag in `self.buffer` until it completes; - `SimpleBlock` and ignored binary leaves such as `Void` are held the same - way. -- **The bound**: refuse once the bytes retained for the element in progress - exceed a cap, checked in `decode` before appending more. `Segment` and - `Cluster` arrive as start and end events and are never held whole, so a - large file is unaffected. Do not set ebml-iterable's - `max_allowable_tag_size`: it applies to every sized tag including - `Segment`, which [moq-dev/moq#7234](https://github.com/moq-dev/moq/pull/7234) - (merged) ruled out. Pick the cap from the largest legitimate block `release` can - publish (its transport frame limit, if it has one), and name the choice in - the PR. The cap applies to every held element, so it must also clear the - largest legitimate `Tracks`/`TrackEntry` (CodecPrivate) and `Void`; if one - cap can't cover both blocks and metadata, use a cap per element kind, or - skip an ignored leaf such as `Void` without holding its bytes. -- Tests on `release`: the reproduction now fails with the error and - bounded memory, a single element just over its cap fails while one just - under imports, and an ordinary file larger than the cap still imports. -- Version bumps stay with a human, per `CONTRIBUTING.md`: this PR bumps - nothing. +bound was proposed to fix the exhaustion without changing supported inputs. +The investigation below disproves that premise for a blanket scratch cap; +implementation waits for a course decision rather than adding a refusal. + +Investigation 2026-11-03, on +[release 338cb9b3e8](https://github.com/moq-dev/moq/commit/338cb9b3e82f2236d8611e682190e946077efd7b): + +- **Reproduced**: ported + `ignored_payload_does_not_accumulate_under_a_large_metadata_master` from + [moq-dev/moq#7234](https://github.com/moq-dev/moq/pull/7234). A 1 GiB + declared EBML master containing a nearly 1 GiB Void, fed in 1 KiB chunks, + fails the bounded-retention assertion with `ignored bytes retained: 1030`. + A diagnostic continuation measures 32,774 bytes after 32 chunks and 65,542 + after 64. This declares 1 GiB; it does not allocate or feed 1 GiB. +- **Transport bound**: `moq_net::group::MAX_CACHE_BYTES` is 33,554,432 bytes + in `rs/moq-net/src/model/group.rs`. `create_frame` and `write_frame` + enforce it, including output container overhead. It bounds a published + frame, not an MKV master or ignored input. +- **Supported-input counterexample**: complete WebM files containing a VP8 + track and frame, with an EBML Void payload of 33,554,431 or 33,554,433 + bytes, both decode and finish successfully when fed in 1 KiB chunks. + Thus the transport limit cannot be used as a blanket element cap. + ebml-iterable 0.7.1 already defaults to a 4,000,000,000-byte sized-tag + limit; increasing a new cap to that ceiling does not solve the reproduced + memory exhaustion. Its setter also limits sized Segments and Clusters, + so lowering it would restrict total file and cluster sizes. +- **Two owners, not one scratch buffer**: `Import::decode` retains an + incomplete leaf in `self.buffer`. The decoder separately captures + completed children of `Ebml`, `Info`, `Tracks`, `TrackEntry`, `Audio`, + `Video`, and `BlockGroup` in its private `OpenTag.children` vectors until + the master completes. Feeding 64 complete 1 KiB Void children under the + same 1 GiB master leaves scratch length zero, with decoder position + 66,121. `TagDecoder::deliver` still pushes every child into the captured + master. A scratch-length check misses that memory, including per-child + allocation overhead. Removing the capture list alone still buffers leaves; + 0.7.1 offers no skip/stream hook. +- No production code, API, wire, dependencies, or versions changed. No + release implementation PR exists. The temporary diagnostic probes were + removed after running them; they must not become tests that require the + vulnerability to persist. The main bookkeeping attempt is + [moq-dev/moq#7718](https://github.com/moq-dev/moq/pull/7718). + +After the course decision, remaining work is the release fix, bounded +retention and accepted-input boundary/large-file coverage, release checks, +semver verification, and local `/quest-review`. Preserve CMAF, skipped track +roles, and video BlockGroups with trailing fields; do not blindly cherry-pick +the full breaking rewrite. Version bumps and publication remain out of scope. Public API: none; if refusing needs a new `mkv::Error` variant on `release`, report it, since the enum is published. Wire: none. +## Waiting + +- Maintainer: approve replacing the size-bound-only plan with a support-preserving + incremental parser backport (recommended), or defer the patch until a release + cut containing the main fix. The former must discard ignored leaves and + metadata children as they arrive and bound retained block/config data by + actual output ownership, while keeping release's accepted BlockGroup layouts + and CMAF output. A new blanket refusal conflicts with both supported-input + preservation and the repository's bounded-ownership rule. Clear this wait + when the chosen course is recorded here or on + [moq-dev/moq#7718](https://github.com/moq-dev/moq/pull/7718). + ## Related - [Backport the relay and auth hardening to release](/quest/m0/backport-release-hardening.md) - other security backports for the same patch release From d9dabe25d068a17c276bdc98e150faa294732bea Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 15:21:59 -0800 Subject: [PATCH 3/3] docs(quest): defer the release MKV fix to the next cut MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The maintainer deferred the release MKV buffering patch to the next main → release cut (2026-11-03) instead of a separate backport. The fix already lives on main as #7234, so the backport quest has no work left: delete it and record the decision in the cut quest that ships it. Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 1 - quest/m0/backport-mkv-buffer.md | 86 --------------------------------- quest/m2/release/cut.md | 7 +++ 3 files changed, 7 insertions(+), 87 deletions(-) delete mode 100644 quest/m0/backport-mkv-buffer.md diff --git a/quest/m0/README.md b/quest/m0/README.md index 25a135c7f0d..a974afc5517 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -36,7 +36,6 @@ since cutting segments from first-frame pts waits on nothing in m1. - [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in a batched merge gate as one `just ci` pass per batch, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last - [More tests under load](/quest/m0/test-flakes-2/README.md) - load-only and queue failures, one quest per flake, fixed at the cause, since each one fails a whole batch - [Backport the relay and auth hardening to release](/quest/m0/backport-release-hardening.md) - `release`'s relay refuses dot-segment paths, serves its HTTP routes with main's response hardening, and withdraws advertisements a peer starves of send credit; its auth client refuses redirects -- [Release's MKV importer bounds what it buffers](/quest/m0/backport-mkv-buffer.md) - on `release`, an MKV input that declares a huge element and trickles its bytes no longer grows the importer's memory without limit - [HLS export on a declared grid](/quest/m0/hls-media-grid.md) - the catalog's `hls` field and record tracks go away, and the export numbers segments on `media.join.interval` from media tracks - [quest init keeps moq's nix-wrapped worktree hooks](/quest/m0/quest-worktree-hooks.md) - quest pin bumps keep the hooks that run moq's pinned `quest` - [HLS lists contributions](/quest/m0/hls-contributions.md) - an HLS master lists each catalog contribution's renditions, and a contribution's media is subscribed only while players fetch it diff --git a/quest/m0/backport-mkv-buffer.md b/quest/m0/backport-mkv-buffer.md deleted file mode 100644 index 2d83dc65599..00000000000 --- a/quest/m0/backport-mkv-buffer.md +++ /dev/null @@ -1,86 +0,0 @@ -# [S] Release's MKV importer bounds what it buffers - -## Goal - -On `release`, an MKV input that declares a huge element and trickles its -bytes no longer grows the importer's memory without limit: once the bytes -held for one incomplete element pass a size bound, the import fails with an -error. Ordinary files of any total size still import, and the published API -and supported inputs are unchanged. - -## Plan - -Decided 2026-10-31 (user, /quest-plan of the follow-ups from the 2026-10-31 /quest-complete run), from [moq-dev/moq#7234](https://github.com/moq-dev/moq/pull/7234). - -Backport #7234's MKV unbounded-buffer fix to `release`? ✅ Trimmed -backport, m0: confirm with the 1 GiB repro on `release`, size bound only. - -Why (the recommendation's reasoning, from #7234's Impact): #7234 is a real memory-exhaustion fix, but the full change is a -rewrite that also refuses inputs `release` accepts today (subtitle tracks, -CMAF output, video `BlockGroup`s with fields after their `Block`). A size -bound was proposed to fix the exhaustion without changing supported inputs. -The investigation below disproves that premise for a blanket scratch cap; -implementation waits for a course decision rather than adding a refusal. - -Investigation 2026-11-03, on -[release 338cb9b3e8](https://github.com/moq-dev/moq/commit/338cb9b3e82f2236d8611e682190e946077efd7b): - -- **Reproduced**: ported - `ignored_payload_does_not_accumulate_under_a_large_metadata_master` from - [moq-dev/moq#7234](https://github.com/moq-dev/moq/pull/7234). A 1 GiB - declared EBML master containing a nearly 1 GiB Void, fed in 1 KiB chunks, - fails the bounded-retention assertion with `ignored bytes retained: 1030`. - A diagnostic continuation measures 32,774 bytes after 32 chunks and 65,542 - after 64. This declares 1 GiB; it does not allocate or feed 1 GiB. -- **Transport bound**: `moq_net::group::MAX_CACHE_BYTES` is 33,554,432 bytes - in `rs/moq-net/src/model/group.rs`. `create_frame` and `write_frame` - enforce it, including output container overhead. It bounds a published - frame, not an MKV master or ignored input. -- **Supported-input counterexample**: complete WebM files containing a VP8 - track and frame, with an EBML Void payload of 33,554,431 or 33,554,433 - bytes, both decode and finish successfully when fed in 1 KiB chunks. - Thus the transport limit cannot be used as a blanket element cap. - ebml-iterable 0.7.1 already defaults to a 4,000,000,000-byte sized-tag - limit; increasing a new cap to that ceiling does not solve the reproduced - memory exhaustion. Its setter also limits sized Segments and Clusters, - so lowering it would restrict total file and cluster sizes. -- **Two owners, not one scratch buffer**: `Import::decode` retains an - incomplete leaf in `self.buffer`. The decoder separately captures - completed children of `Ebml`, `Info`, `Tracks`, `TrackEntry`, `Audio`, - `Video`, and `BlockGroup` in its private `OpenTag.children` vectors until - the master completes. Feeding 64 complete 1 KiB Void children under the - same 1 GiB master leaves scratch length zero, with decoder position - 66,121. `TagDecoder::deliver` still pushes every child into the captured - master. A scratch-length check misses that memory, including per-child - allocation overhead. Removing the capture list alone still buffers leaves; - 0.7.1 offers no skip/stream hook. -- No production code, API, wire, dependencies, or versions changed. No - release implementation PR exists. The temporary diagnostic probes were - removed after running them; they must not become tests that require the - vulnerability to persist. The main bookkeeping attempt is - [moq-dev/moq#7718](https://github.com/moq-dev/moq/pull/7718). - -After the course decision, remaining work is the release fix, bounded -retention and accepted-input boundary/large-file coverage, release checks, -semver verification, and local `/quest-review`. Preserve CMAF, skipped track -roles, and video BlockGroups with trailing fields; do not blindly cherry-pick -the full breaking rewrite. Version bumps and publication remain out of scope. - -Public API: none; if refusing needs a new `mkv::Error` variant on -`release`, report it, since the enum is published. Wire: none. - -## Waiting - -- Maintainer: approve replacing the size-bound-only plan with a support-preserving - incremental parser backport (recommended), or defer the patch until a release - cut containing the main fix. The former must discard ignored leaves and - metadata children as they arrive and bound retained block/config data by - actual output ownership, while keeping release's accepted BlockGroup layouts - and CMAF output. A new blanket refusal conflicts with both supported-input - preservation and the repository's bounded-ownership rule. Clear this wait - when the chosen course is recorded here or on - [moq-dev/moq#7718](https://github.com/moq-dev/moq/pull/7718). - -## Related - -- [Backport the relay and auth hardening to release](/quest/m0/backport-release-hardening.md) - other security backports for the same patch release diff --git a/quest/m2/release/cut.md b/quest/m2/release/cut.md index 0633405d1ad..dce91cba871 100644 --- a/quest/m2/release/cut.md +++ b/quest/m2/release/cut.md @@ -27,6 +27,13 @@ group seed: the merge of release into main keeps main's `rs/moq-stats` and `doc/concept/stats.md`. MoQ Pro's VOD `storage.json` moves from the same seed to its own epoch. +This cut ships [moq-dev/moq#7234](https://github.com/moq-dev/moq/pull/7234)'s +MKV buffering fix to release, which stays exposed until then. Backport it to +`release` separately? ✅ No: defer it to the next main → release cut +(maintainer, 2026-11-03). A blanket size cap would refuse supported inputs and +miss the decoder's captured children, as +[moq-dev/moq#7718](https://github.com/moq-dev/moq/pull/7718) measured. + Recurs once per major release (decided 2026-10-19). The PR that finishes the cut shifts the milestones up, as the root [questline](/quest/README.md) describes. This epic's remaining publication and adoption quests move to m0,