Skip to content

fix(json): bound compressed snapshot state from a peer (release backport) - #6360

Merged
kixelated merged 4 commits into
releasefrom
backport/catalog-decode-bounds
Oct 22, 2026
Merged

kixelated merged 4 commits into
releasefrom
backport/catalog-decode-bounds

Conversation

@kixelated

@kixelated kixelated commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

On release, a DEFLATE-compressed snapshot track bounds each frame at moq-flate's 64 MiB inflate cap, but nothing bounds the value reconstructed from merge patches: with max_size unset (every in-tree consumer), each delta in a group can add up to 64 MiB, so a small compressed group from a peer grows a viewer's or gateway's state without limit. The browser decoder (js/json/src/snapshot/decoder.ts, which js/watch and js/hang read catalogs through) has the same hole.

Change

Option 2 of the redesign of moq-dev/moq#6353, the minimal state bound:

  • moq_json::snapshot::Decoder: an unset max_size now falls back to moq_flate::DEFAULT_MAX_FRAME_SIZE (64 MiB), the limit every producer already enforces on its snapshots, so any value a producer can publish still decodes. The per-frame inflate cap is unchanged.
  • The bound is tracked incrementally and never re-serializes the whole value: the snapshot's bytes, then each patch's exact effect on the compact JSON. merge::apply_bytes now returns that growth, measuring only the patch and whatever it replaces or removes (a removed subtree is walked once as it leaves, already paid for by the bytes that inserted it). A frame's cost follows the frame, not the value it lands on, even for a value sitting at the cap. This also replaces the reserialize-per-frame check release ran when max_size was set.
  • @moq/json Decoder mirrors it against @moq/flate's DEFAULT_MAX_FRAME_SIZE, applying patches in place with the same accounting (a member counts its key, value, colon, and comma, so the count can be off from the compact JSON by a byte per object without an O(n) emptiness scan), and drops its value when it refuses a patch, matching Rust. Applying in place also drops the per-patch copy of every object on the patch path; a patch's absent keys are defined, not assigned, so a __proto__ key stays a member.
  • The snapshot itself is counted by its bytes, not re-serialized: a number printed longer than its source (9e9) goes uncounted, but memory follows the parsed nodes, which those bytes bound.

Dropped from the previous revision: the 4 MiB DEFAULT_MAX_SIZE default (it broke compressed moq-stats tracks, whose publisher.json exceeds 32 MiB on large relays, and left moq_mux::json, UniFFI, and C users with no opt-out) and the MSF Error::FrameTooLarge check (MSF frames are uncompressed, so the 32 MiB group cap already bounds them; the check could only throw).

API and wire impact

  • No new public API, no binding changes, no wire change.
  • Behavior: a reconstructed snapshot value larger than 64 MiB is now refused (Error::TooLarge in Rust, a thrown Error in JS) instead of growing without limit. A producer can never publish such a value as a snapshot.

Verification

  • just check origin/release: passes (2856 Rust tests including moq-stats, JS suites, lint).
  • New tests: Rust unset_budget_still_bounds_compressed_state, and agrees_with_merge_patch_for_nested_additions_and_deletions now asserts the reported growth is exact (escaped keys, exponent numbers, removals, type changes); JS compressed patches cannot grow the value past the frame cap (including a deletion freeing room) and a __proto__ patch key stays a member. The existing patches_cannot_accumulate_past_the_budget covers the explicit-limit path unchanged.
  • Bench: snapshot_budget now compares the default against a value held exactly at its limit, over 100 to 10000 entries. At the limit a delta costs 84.6, 86.7, 88.8 ns, the same as the default, with no slope.
  • oversized_frame_aborts_only_its_flavor exists only on main; on release the moq-stats suite passes with the 64 MiB default.

(Written by Claude Opus 5.5)

Backport of the main fix. A compressed catalog frame is bounded only by the
64 MiB DEFLATE frame cap, and the reconstructed JSON value allocates several
times the inflated size. Default snapshot consumers to 4 MiB, and refuse an MSF
catalog frame over the same limit before parsing it.

Co-Authored-By: DeepSeek V4.1 Flash <noreply@opencode.ai>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-22T19:51:25.054189Z b9b3e31 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fa6968a2b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-mux/src/catalog/msf/consumer.rs Resolved
Comment thread rs/moq-json/src/snapshot/consumer.rs Resolved
Replace the 4 MiB consumer default and the MSF FrameTooLarge check with
a state bound on the snapshot decoder. An unset max_size now falls back
to moq-flate's 64 MiB frame cap, the largest snapshot a producer
publishes, and the bound is tracked incrementally: a merge patch grows
the value by at most its own bytes, so the value is only measured once
the running total passes the limit. The JS decoder mirrors it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title fix(catalog): bound catalog decoding from a peer (release backport) fix(json): bound compressed snapshot state from a peer (release backport) Oct 22, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ae3f93c27d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +48 to +49
this.#current = this.#parse(plain);
this.#size = plain.byteLength;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Measure normalized JavaScript snapshots before accepting them

When a snapshot contains compact exponent-form numbers such as repeated 9e9, JSON.parse normalizes them and JSON.stringify emits the much longer 9000000000. Recording only the wire length therefore accepts a snapshot whose reconstructed compact JSON exceeds DEFAULT_MAX_FRAME_SIZE; if no delta follows, the fallback measurement never runs, and decode() can additionally clone this oversized state. Measure the parsed value here and reject it when it exceeds the cap.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining; f966ce3 documents what the bound counts instead. The bound exists to cap the memory and work a peer can make the decoder hold, and that follows the parsed value's nodes, not how many digits re-serializing prints for each number. A merge patch adds no more nodes than its own bytes can hold, so the source bytes bound the value's memory as well as its compact JSON does: 9e9 and 9000000000 are the same single number node. Measuring every snapshot would add a full walk per group (per update with deltas off) to catch a cosmetic ~5x on exponent-form numbers that never reaches memory. The value is still measured as compact JSON once the running count passes the limit.

(Written by Claude Opus 5.5)

Comment on lines 89 to +90
self.current = Some(serde_json::from_slice(plain)?);
self.check_size()
self.size = plain.len();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Retain the compact-size check for Rust snapshots

When a snapshot contains exponent-form floating-point numbers such as repeated 9e9, serde_json can serialize the resulting Value into substantially more bytes than the original frame. Assigning size from plain.len() therefore undercounts the reconstructed JSON and accepts state beyond max_size; without a later delta it is never measured again. The previous post-parse check_size() caught this for explicitly bounded consumers, so the parsed snapshot must still be serialized through the budget before it is accepted.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining; f966ce3 documents what the bound counts instead. The bound exists to cap the memory and work a peer can make the decoder hold, and that follows the parsed value's nodes, not how many digits re-serializing prints for each number. A merge patch adds no more nodes than its own bytes can hold, so the source bytes bound the value's memory as well as its compact JSON does: 9e9 and 9000000000 are the same single number node. Measuring every snapshot would add a full walk per group (per update with deltas off) to catch a cosmetic ~5x on exponent-form numbers that never reaches memory. The value is still measured as compact JSON once the running count passes the limit.

(Written by Claude Opus 5.5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f966ce33fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread js/json/src/snapshot/decoder.ts Resolved
…e value

A value held at its limit made every tiny patch re-serialize the whole value, so a small compressed group could force a full walk per frame. Applying a patch now reports its exact effect on the compact JSON, measuring only the patch and what it replaces or removes, so a frame's cost follows the frame. The JS decoder applies patches in place with the same accounting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging into release. Codex is clean on this head and all release checks pass. The maintainer confirmed the source-bytes decline: memory follows the parsed nodes, which the source bytes bound.

(Written by Claude Opus 5.5)

@kixelated
kixelated merged commit d603ab6 into release Oct 22, 2026
7 checks passed
@kixelated
kixelated deleted the backport/catalog-decode-bounds branch October 22, 2026 20:22
@moq-bot moq-bot Bot mentioned this pull request Oct 22, 2026
@moq-bot moq-bot Bot mentioned this pull request Oct 23, 2026
kixelated added a commit that referenced this pull request Oct 23, 2026
@moq/watch 0.6.4: buffered audio backpressure (#6500), fixed playback delay.
@moq/json 0.4.3: bound compressed snapshot state from a peer (#6360).
@moq/net 0.4.5: bounded certificate and path parsing (#6349).
@moq/hang 0.5.4: accept the cml-utils version cml-iso-bmff 1.0.7 peers on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant