Skip to content

fix(flate): hold a compressed group to the group budget - #6353

Merged
kixelated merged 5 commits into
mainfrom
fix/catalog-decode-bounds
Oct 22, 2026
Merged

kixelated merged 5 commits into
mainfrom
fix/catalog-decode-bounds

Conversation

@kixelated

@kixelated kixelated commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Group-scoped DEFLATE in moq-flate / @moq/flate (used by moq-json / @moq/json, so by hangz catalogs, moq-stats .z tracks, and any compressed JSON track) capped each frame at 64 MiB of inflated output and nothing else. A snapshot consumer with the default max_size: None applied merge patches with no bound on the reconstructed state, and a group may hold 8192 frames. So a peer could send a small compressed group that each viewer and gateway inflates, parses, and holds at many times the 32 MiB a plain group can carry, in Rust and in the browser. That is a crash on the reader, not just a CPU cost.

Change

A compressed group now costs a reader no more than a plain one, the same rule moq-net's compressed tracks follow (quest/m1/compress: "a zip bomb then costs no more than a plain frame ... so there is no separate inflate cap constant"):

  • moq_flate::Decoder and @moq/flate's Decoder hold the window's total inflated output, across every frame, to the group budget (MAX_CACHE_BYTES, 32 MiB), replacing the 64 MiB per-frame cap.
  • Every encoder (Rust and JS: flate snapshot/stream, json snapshot/stream/window) charges a compressed frame the larger of its slice and its plaintext against that same budget. Snapshot deltas and window ops roll a new group, while a stream refuses with GroupTooLarge, before a reader would refuse. A compressed snapshot value or window header over 32 MiB fails with the same FrameTooLarge a plain one gets.
  • The snapshot state is bounded as a result: it never exceeds what the group inflated.

This replaces the earlier version of this PR (a 4 MiB DEFAULT_MAX_SIZE snapshot default and an MSF Error::FrameTooLarge), which Codex showed rejected legitimate 4 to 32 MiB snapshots and left no override for C/UniFFI callers. The budget now comes from the transport, so no catalog constant is needed, and since it matches what a conforming producer can send, no binding needs an override. The MSF catalog is plain JSON already bounded by the transport, so its extra check is gone. max_size stays as an optional tighter cap (moq-cli's report reader uses it).

Behavior change: moq-stats .z

#5826 let the .z stats flavor keep updating past 32 MB of JSON, because DEFLATE shrank it under the cache limit. That relied on a reader inflating more than a group may carry, which is the hole this closes. Now a frame past 32 MB of uncompressed JSON ends both flavors with FrameTooLarge. This is the same limit moq-net's compressed tracks impose once quest/m1/compress/migrate moves stats onto one compress: true track. oversized_frame_aborts_only_its_flavor is now oversized_frame_ends_both_flavors, and doc/concept/stats.md says so.

A .z group of any size below that is not refused. Its snapshot encoder charges every delta its plaintext, so a large snapshot followed by inflated deltas rolls to a fresh snapshot before the group's cumulative plaintext would pass 32 MiB. Readers never see the decoder's budget trip on a conforming stream. a_compressed_delta_is_gated_on_its_inflated_size (Rust and JS) covers this. If a delta overflows and the roll is refused too, the encoder drops the advanced window, so the next update is a decodable snapshot (a_refused_roll_resynchronizes).

Public API / wire impact

  • Wire: none.
  • moq-flate: removed DEFAULT_MAX_FRAME_SIZE and Decoder::with_max_frame_size. Error::TooLarge now means the group budget.
  • @moq/flate: removed DEFAULT_MAX_FRAME_SIZE, DecoderOptions, and Decoder.maxFrameSize.
  • moq-json / @moq/json: no signature change. Compressed values over 32 MiB now fail with FrameTooLarge instead of passing up to 64 MiB, and max_size no longer stops inflation early.
  • moq-stats: behavior change above.
  • MIGRATE.md entries in rs/moq-flate, rs/moq-json, rs/moq-stats, js/flate, js/json.

Verification

  • cargo nextest run -p moq-flate -p moq-json -p moq-stats -p moq-mux -p hang -p moq-ffi -p moq-cli -p moq-c: 2659 passed, including the updated moq-stats test.
  • bun test in js/flate (43 pass) and js/json (182 pass).
  • just fix. just check: js and rs passed. The flake module failed locally with unshare ... Operation not permitted building the github-runner VM image, which is a sandbox limit of this machine and unrelated to the change.
  • New tests: a decoder's budget spans the whole window; a compressed log, delta, snapshot, and window header are each charged or refused by their inflated size (Rust and JS mirrors).

Release

#6360 (release backport) landed separately with a minimal state bound: it counts each patch's exact growth and caps the state at the 64 MiB producer limit. Main does not need that counting. The reconstructed state can never exceed what its group inflated, and the per-group inflate budget already bounds that, so this PR keeps one bound instead of two.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 339693eb-a478-4a69-9620-30d9f2d79f8e
📥 Commits

Reviewing files that changed from the base of the PR and between be8edae and 3e84f8f.

📒 Files selected for processing: 4
  • rs/moq-json/MIGRATE.md
  • rs/moq-json/src/snapshot/consumer.rs
  • rs/moq-mux/src/catalog/msf/consumer.rs
  • rs/moq-mux/src/catalog/msf/mod.rs

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

Walkthrough

The snapshot consumer now defaults max_size to Some(DEFAULT_MAX_SIZE), set to 4 MiB. The migration guide documents how to set a larger limit or use None. MSF catalog parsing now rejects payloads above the same 4 MiB limit before UTF-8 decoding or JSON parsing. It returns FrameTooLarge with the payload size and maximum.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 3e84f

The new decoding limits are ready to merge after normal checks; no actionable merge-blocking risk remains.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Title check Warning The title describes a compressed-group budget change in flate, but the changeset adds a 4 MiB snapshot default and bounds MSF catalog frames. Update the title to describe the actual changes, such as: "fix(catalog): bound snapshot and MSF catalog decoding"
Passed Passed checks: 4 passed
Check name Status Explanation
Docstring Coverage Passed Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 3 files. (1 skipped: 1 …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description addresses snapshot and catalog decoding limits, which are directly related to the changeset. It describes a broader or different revision than the summarized files, but it is not off-t…
Full details: Docstring Coverage

Explanation

Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 3 files. (1 skipped: 1 unsupported.)

Full details: Description check

Explanation

The description addresses snapshot and catalog decoding limits, which are directly related to the changeset. It describes a broader or different revision than the summarized files, but it is not off-topic.

✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-22T20:31:12.415082Z 70f6f1c New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 36998dd8ab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-json/src/snapshot/consumer.rs Outdated
fn default() -> Self {
Self {
compression: Compression::None,
max_size: Some(DEFAULT_MAX_SIZE),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep the catalog cap out of generic snapshot defaults

Whenever a snapshot's compact state is between 4 MiB and the transport's 32 MiB group limit, this default now rejects it even though the producer and transport allow it. This already invalidates moq-stats's oversized_frame_aborts_only_its_flavor test in rs/moq-stats/src/produce.rs:1803-1851, which deliberately verifies that a 32 MiB state survives on the compressed sibling; after this change, next(&mut compressed).unwrap() returns TooLarge. Scope the 4 MiB default to catalog/metadata consumers or update every built-in producer/consumer pair to agree on the limit.

AGENTS.md reference: AGENTS.md:L22-L22

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 0138885 by moving the bound out of the snapshot default entirely. The 4 MiB DEFAULT_MAX_SIZE is gone. Instead, the DEFLATE decoder holds a whole group to the transport's group budget (MAX_CACHE_BYTES), and every encoder charges a compressed frame its plaintext, so nothing a plain group can carry is refused.

The one deliberate casualty is the oversized_frame_aborts_only_its_flavor case: a .z state past 32 MiB of JSON relied on a reader inflating more than a group may carry, which is the hole this PR closes. Both flavors now end with FrameTooLarge (test renamed to oversized_frame_ends_both_flavors, documented in doc/concept/stats.md and rs/moq-stats/MIGRATE.md). That matches the limit moq-net's compressed tracks impose once stats move onto them.

(Written by Claude Opus 5.5)

Comment thread rs/moq-json/MIGRATE.md Outdated

### Snapshot consumers are bounded by default

`snapshot::consumer::Config::default()` now sets `max_size` to `DEFAULT_MAX_SIZE` (4 MiB), so a compressed frame from a peer cannot reconstruct an unbounded value. A consumer that needs a larger state sets `max_size` itself, or `None` to keep only the DEFLATE frame cap.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Expose the documented size override through the bindings

For C and UniFFI callers whose snapshots exceed 4 MiB, this migration advice is unusable: moq_json_snapshot_config and MoqJsonSnapshotConfig expose compression and producer options but no max_size, while their conversions always use Config::default(). Those callers therefore lose support for previously readable tracks without the promised opt-out; expose the limit through the affected bindings and add their package migration entries, or avoid changing their effective default.

AGENTS.md reference: AGENTS.md:L59-L60

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0138885 by not changing the effective default for anyone who can't override it. max_size defaults to None again. The new bound lives in the DEFLATE decoder as the group budget, which is exactly what a conforming producer can send, so a C or UniFFI caller never needs an override to read a legitimate track. The migration entry now describes that.

(Written by Claude Opus 5.5)

A compressed catalog frame is bounded only by the 64 MiB DEFLATE frame cap,
and the reconstructed JSON value allocates several times the inflated size.
Default snapshot consumers to 4 MiB, which every catalog and metadata track
fits far below, and refuse an MSF catalog frame over the same limit before
parsing it.

Co-Authored-By: DeepSeek V4.1 Flash <noreply@opencode.ai>
kixelated and others added 2 commits October 22, 2026 11:28
A compressed JSON or opaque track capped each frame at 64 MiB of inflated
output and left the reconstructed snapshot unbounded, so a peer's tiny
frames could make every reader inflate, parse, and hold far more than a
plain group carries. The DEFLATE decoder now holds its whole window to
moq-net's group budget (MAX_CACHE_BYTES), the rule moq-net's compressed
tracks already follow, and every encoder charges a compressed frame its
plaintext so it rolls or refuses before a reader would.

This replaces the 4 MiB snapshot default and the MSF frame check: the
budget comes from the transport, so no catalog-specific constant and no
binding override is needed. A compressed value past 32 MiB now fails with
FrameTooLarge like a plain one, so moq-stats' .z flavor ends with its plain
sibling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title fix(catalog): bound catalog decoding from a peer fix(flate): hold a compressed group to the group budget Oct 22, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 01388853d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-json/src/snapshot/encoder.rs Outdated
Comment on lines +412 to +413
if self.config.compression.is_deflate() && snapshot.len() as u64 > self.max_group_bytes {
return Err(moq_net::Error::FrameTooLarge.into());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reset the compression window when an oversized roll fails

When a compressed delta exceeds group_bytes, flate.frame has already advanced the DEFLATE window before the encoder falls back to a snapshot; if that snapshot is also over the new 32 MiB limit, this return leaves the advanced window installed. A later small update can then be emitted as a delta against history the consumer never received, producing invalid or corrupted JSON even though the oversized update was rejected. Reset the group-local compression state before returning this error; the same state transition exists in js/json/src/snapshot/encoder.ts and needs the matching fix.

AGENTS.md reference: AGENTS.md:L23-L23

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in 68a338c. When a delta overflows the group, the encoder now drops the group-local state (reset) before rolling, so if the snapshot is refused too, the next update is a snapshot instead of a delta against an advanced window. The JS encoder got the same fix. a_refused_roll_resynchronizes and its JS mirror test it.

(Written by Claude Opus 5.5)

A compressed delta that overflows the group advances the DEFLATE window before the encoder rolls to a snapshot. If that snapshot is refused too, the advanced window stayed installed and the next update rode as a delta against history no reader received. Drop the group before rolling, in Rust and JS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

# Conflicts:
#	js/json/MIGRATE.md
#	js/json/src/snapshot/encoder.ts
#	js/json/src/window/encoder.ts
#	rs/moq-json/MIGRATE.md
#	rs/moq-json/src/snapshot/encoder.rs
#	rs/moq-json/src/window/encoder.rs
#	rs/moq-stats/src/produce.rs
@kixelated

Copy link
Copy Markdown
Collaborator Author

Queued with the maintainer's approval: one 32 MiB inflate budget per DEFLATE group in Rust and JS, with encoders rolling or refusing before a reader would hit it. Codex approved this head, CI is green, and quest check passes against current main.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit 43fc925 Oct 22, 2026
10 checks passed
@kixelated
kixelated deleted the fix/catalog-decode-bounds branch October 22, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant