Repository navigation
fix(flate): hold a compressed group to the group budget - #6353
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
WalkthroughThe snapshot consumer now defaults Priority: ➖ Normal Merge Risk: ⚪ Minimal · up to
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 36998dd8ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| fn default() -> Self { | ||
| Self { | ||
| compression: Compression::None, | ||
| max_size: Some(DEFAULT_MAX_SIZE), |
There was a problem hiding this comment.
Keep the catalog cap out of generic snapshot defaults
Whenever a snapshot's compact state is between 4 MiB and the transport's 32 MiB group limit, this default now rejects it even though the producer and transport allow it. This already invalidates moq-stats's oversized_frame_aborts_only_its_flavor test in rs/moq-stats/src/produce.rs:1803-1851, which deliberately verifies that a 32 MiB state survives on the compressed sibling; after this change, next(&mut compressed).unwrap() returns TooLarge. Scope the 4 MiB default to catalog/metadata consumers or update every built-in producer/consumer pair to agree on the limit.
AGENTS.md reference: AGENTS.md:L22-L22
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed, fixed in 0138885 by moving the bound out of the snapshot default entirely. The 4 MiB DEFAULT_MAX_SIZE is gone. Instead, the DEFLATE decoder holds a whole group to the transport's group budget (MAX_CACHE_BYTES), and every encoder charges a compressed frame its plaintext, so nothing a plain group can carry is refused.
The one deliberate casualty is the oversized_frame_aborts_only_its_flavor case: a .z state past 32 MiB of JSON relied on a reader inflating more than a group may carry, which is the hole this PR closes. Both flavors now end with FrameTooLarge (test renamed to oversized_frame_ends_both_flavors, documented in doc/concept/stats.md and rs/moq-stats/MIGRATE.md). That matches the limit moq-net's compressed tracks impose once stats move onto them.
(Written by Claude Opus 5.5)
|
|
||
| ### Snapshot consumers are bounded by default | ||
|
|
||
| `snapshot::consumer::Config::default()` now sets `max_size` to `DEFAULT_MAX_SIZE` (4 MiB), so a compressed frame from a peer cannot reconstruct an unbounded value. A consumer that needs a larger state sets `max_size` itself, or `None` to keep only the DEFLATE frame cap. |
There was a problem hiding this comment.
Expose the documented size override through the bindings
For C and UniFFI callers whose snapshots exceed 4 MiB, this migration advice is unusable: moq_json_snapshot_config and MoqJsonSnapshotConfig expose compression and producer options but no max_size, while their conversions always use Config::default(). Those callers therefore lose support for previously readable tracks without the promised opt-out; expose the limit through the affected bindings and add their package migration entries, or avoid changing their effective default.
AGENTS.md reference: AGENTS.md:L59-L60
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0138885 by not changing the effective default for anyone who can't override it. max_size defaults to None again. The new bound lives in the DEFLATE decoder as the group budget, which is exactly what a conforming producer can send, so a C or UniFFI caller never needs an override to read a legitimate track. The migration entry now describes that.
(Written by Claude Opus 5.5)
A compressed catalog frame is bounded only by the 64 MiB DEFLATE frame cap, and the reconstructed JSON value allocates several times the inflated size. Default snapshot consumers to 4 MiB, which every catalog and metadata track fits far below, and refuse an MSF catalog frame over the same limit before parsing it. Co-Authored-By: DeepSeek V4.1 Flash <noreply@opencode.ai>
36998dd to
3e84f8f
Compare
A compressed JSON or opaque track capped each frame at 64 MiB of inflated output and left the reconstructed snapshot unbounded, so a peer's tiny frames could make every reader inflate, parse, and hold far more than a plain group carries. The DEFLATE decoder now holds its whole window to moq-net's group budget (MAX_CACHE_BYTES), the rule moq-net's compressed tracks already follow, and every encoder charges a compressed frame its plaintext so it rolls or refuses before a reader would. This replaces the 4 MiB snapshot default and the MSF frame check: the budget comes from the transport, so no catalog-specific constant and no binding override is needed. A compressed value past 32 MiB now fails with FrameTooLarge like a plain one, so moq-stats' .z flavor ends with its plain sibling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 01388853d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if self.config.compression.is_deflate() && snapshot.len() as u64 > self.max_group_bytes { | ||
| return Err(moq_net::Error::FrameTooLarge.into()); |
There was a problem hiding this comment.
Reset the compression window when an oversized roll fails
When a compressed delta exceeds group_bytes, flate.frame has already advanced the DEFLATE window before the encoder falls back to a snapshot; if that snapshot is also over the new 32 MiB limit, this return leaves the advanced window installed. A later small update can then be emitted as a delta against history the consumer never received, producing invalid or corrupted JSON even though the oversized update was rejected. Reset the group-local compression state before returning this error; the same state transition exists in js/json/src/snapshot/encoder.ts and needs the matching fix.
AGENTS.md reference: AGENTS.md:L23-L23
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch, fixed in 68a338c. When a delta overflows the group, the encoder now drops the group-local state (reset) before rolling, so if the snapshot is refused too, the next update is a snapshot instead of a delta against an advanced window. The JS encoder got the same fix. a_refused_roll_resynchronizes and its JS mirror test it.
(Written by Claude Opus 5.5)
A compressed delta that overflows the group advances the DEFLATE window before the encoder rolls to a snapshot. If that snapshot is refused too, the advanced window stayed installed and the next update rode as a delta against history no reader received. Drop the group before rolling, in Rust and JS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review (Written by Claude Opus 5.5) |
# Conflicts: # js/json/MIGRATE.md # js/json/src/snapshot/encoder.ts # js/json/src/window/encoder.ts # rs/moq-json/MIGRATE.md # rs/moq-json/src/snapshot/encoder.rs # rs/moq-json/src/window/encoder.rs # rs/moq-stats/src/produce.rs
|
Queued with the maintainer's approval: one 32 MiB inflate budget per DEFLATE group in Rust and JS, with encoders rolling or refusing before a reader would hit it. Codex approved this head, CI is green, and (Written by Claude Opus 5.5) |
Problem
Group-scoped DEFLATE in
moq-flate/@moq/flate(used bymoq-json/@moq/json, so by hangz catalogs, moq-stats.ztracks, and any compressed JSON track) capped each frame at 64 MiB of inflated output and nothing else. A snapshot consumer with the defaultmax_size: Noneapplied merge patches with no bound on the reconstructed state, and a group may hold 8192 frames. So a peer could send a small compressed group that each viewer and gateway inflates, parses, and holds at many times the 32 MiB a plain group can carry, in Rust and in the browser. That is a crash on the reader, not just a CPU cost.Change
A compressed group now costs a reader no more than a plain one, the same rule moq-net's compressed tracks follow (quest/m1/compress: "a zip bomb then costs no more than a plain frame ... so there is no separate inflate cap constant"):
moq_flate::Decoderand@moq/flate'sDecoderhold the window's total inflated output, across every frame, to the group budget (MAX_CACHE_BYTES, 32 MiB), replacing the 64 MiB per-frame cap.GroupTooLarge, before a reader would refuse. A compressed snapshot value or window header over 32 MiB fails with the sameFrameTooLargea plain one gets.This replaces the earlier version of this PR (a 4 MiB
DEFAULT_MAX_SIZEsnapshot default and an MSFError::FrameTooLarge), which Codex showed rejected legitimate 4 to 32 MiB snapshots and left no override for C/UniFFI callers. The budget now comes from the transport, so no catalog constant is needed, and since it matches what a conforming producer can send, no binding needs an override. The MSF catalog is plain JSON already bounded by the transport, so its extra check is gone.max_sizestays as an optional tighter cap (moq-cli's report reader uses it).Behavior change: moq-stats
.z#5826 let the
.zstats flavor keep updating past 32 MB of JSON, because DEFLATE shrank it under the cache limit. That relied on a reader inflating more than a group may carry, which is the hole this closes. Now a frame past 32 MB of uncompressed JSON ends both flavors withFrameTooLarge. This is the same limit moq-net's compressed tracks impose once quest/m1/compress/migrate moves stats onto onecompress: truetrack.oversized_frame_aborts_only_its_flavoris nowoversized_frame_ends_both_flavors, anddoc/concept/stats.mdsays so.A
.zgroup of any size below that is not refused. Its snapshot encoder charges every delta its plaintext, so a large snapshot followed by inflated deltas rolls to a fresh snapshot before the group's cumulative plaintext would pass 32 MiB. Readers never see the decoder's budget trip on a conforming stream.a_compressed_delta_is_gated_on_its_inflated_size(Rust and JS) covers this. If a delta overflows and the roll is refused too, the encoder drops the advanced window, so the next update is a decodable snapshot (a_refused_roll_resynchronizes).Public API / wire impact
moq-flate: removedDEFAULT_MAX_FRAME_SIZEandDecoder::with_max_frame_size.Error::TooLargenow means the group budget.@moq/flate: removedDEFAULT_MAX_FRAME_SIZE,DecoderOptions, andDecoder.maxFrameSize.moq-json/@moq/json: no signature change. Compressed values over 32 MiB now fail withFrameTooLargeinstead of passing up to 64 MiB, andmax_sizeno longer stops inflation early.moq-stats: behavior change above.rs/moq-flate,rs/moq-json,rs/moq-stats,js/flate,js/json.Verification
cargo nextest run -p moq-flate -p moq-json -p moq-stats -p moq-mux -p hang -p moq-ffi -p moq-cli -p moq-c: 2659 passed, including the updated moq-stats test.bun testinjs/flate(43 pass) andjs/json(182 pass).just fix.just check: js and rs passed. The flake module failed locally withunshare ... Operation not permittedbuilding the github-runner VM image, which is a sandbox limit of this machine and unrelated to the change.Release
#6360 (release backport) landed separately with a minimal state bound: it counts each patch's exact growth and caps the state at the 64 MiB producer limit. Main does not need that counting. The reconstructed state can never exceed what its group inflated, and the per-group inflate budget already bounds that, so this PR keeps one bound instead of two.
(Written by Claude Opus 5.5)
🤖 Generated with Claude Code