Repository navigation
fix(json): bound compressed snapshot state from a peer (release backport) - #6360
Conversation
Backport of the main fix. A compressed catalog frame is bounded only by the 64 MiB DEFLATE frame cap, and the reconstructed JSON value allocates several times the inflated size. Default snapshot consumers to 4 MiB, and refuse an MSF catalog frame over the same limit before parsing it. Co-Authored-By: DeepSeek V4.1 Flash <noreply@opencode.ai>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fa6968a2b9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Replace the 4 MiB consumer default and the MSF FrameTooLarge check with a state bound on the snapshot decoder. An unset max_size now falls back to moq-flate's 64 MiB frame cap, the largest snapshot a producer publishes, and the bound is tracked incrementally: a merge patch grows the value by at most its own bytes, so the value is only measured once the running total passes the limit. The JS decoder mirrors it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ae3f93c27d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| this.#current = this.#parse(plain); | ||
| this.#size = plain.byteLength; |
There was a problem hiding this comment.
Measure normalized JavaScript snapshots before accepting them
When a snapshot contains compact exponent-form numbers such as repeated 9e9, JSON.parse normalizes them and JSON.stringify emits the much longer 9000000000. Recording only the wire length therefore accepts a snapshot whose reconstructed compact JSON exceeds DEFAULT_MAX_FRAME_SIZE; if no delta follows, the fallback measurement never runs, and decode() can additionally clone this oversized state. Measure the parsed value here and reject it when it exceeds the cap.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining; f966ce3 documents what the bound counts instead. The bound exists to cap the memory and work a peer can make the decoder hold, and that follows the parsed value's nodes, not how many digits re-serializing prints for each number. A merge patch adds no more nodes than its own bytes can hold, so the source bytes bound the value's memory as well as its compact JSON does: 9e9 and 9000000000 are the same single number node. Measuring every snapshot would add a full walk per group (per update with deltas off) to catch a cosmetic ~5x on exponent-form numbers that never reaches memory. The value is still measured as compact JSON once the running count passes the limit.
(Written by Claude Opus 5.5)
| self.current = Some(serde_json::from_slice(plain)?); | ||
| self.check_size() | ||
| self.size = plain.len(); |
There was a problem hiding this comment.
Retain the compact-size check for Rust snapshots
When a snapshot contains exponent-form floating-point numbers such as repeated 9e9, serde_json can serialize the resulting Value into substantially more bytes than the original frame. Assigning size from plain.len() therefore undercounts the reconstructed JSON and accepts state beyond max_size; without a later delta it is never measured again. The previous post-parse check_size() caught this for explicitly bounded consumers, so the parsed snapshot must still be serialized through the budget before it is accepted.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining; f966ce3 documents what the bound counts instead. The bound exists to cap the memory and work a peer can make the decoder hold, and that follows the parsed value's nodes, not how many digits re-serializing prints for each number. A merge patch adds no more nodes than its own bytes can hold, so the source bytes bound the value's memory as well as its compact JSON does: 9e9 and 9000000000 are the same single number node. Measuring every snapshot would add a full walk per group (per update with deltas off) to catch a cosmetic ~5x on exponent-form numbers that never reaches memory. The value is still measured as compact JSON once the running count passes the limit.
(Written by Claude Opus 5.5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f966ce33fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…e value A value held at its limit made every tiny patch re-serialize the whole value, so a small compressed group could force a full walk per frame. Applying a patch now reports its exact effect on the compact JSON, measuring only the patch and what it replaces or removes, so a frame's cost follows the frame. The JS decoder applies patches in place with the same accounting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merging into (Written by Claude Opus 5.5) |
@moq/watch 0.6.4: buffered audio backpressure (#6500), fixed playback delay. @moq/json 0.4.3: bound compressed snapshot state from a peer (#6360). @moq/net 0.4.5: bounded certificate and path parsing (#6349). @moq/hang 0.5.4: accept the cml-utils version cml-iso-bmff 1.0.7 peers on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
On
release, a DEFLATE-compressed snapshot track bounds each frame at moq-flate's 64 MiB inflate cap, but nothing bounds the value reconstructed from merge patches: withmax_sizeunset (every in-tree consumer), each delta in a group can add up to 64 MiB, so a small compressed group from a peer grows a viewer's or gateway's state without limit. The browser decoder (js/json/src/snapshot/decoder.ts, whichjs/watchandjs/hangread catalogs through) has the same hole.Change
Option 2 of the redesign of moq-dev/moq#6353, the minimal state bound:
moq_json::snapshot::Decoder: an unsetmax_sizenow falls back tomoq_flate::DEFAULT_MAX_FRAME_SIZE(64 MiB), the limit every producer already enforces on its snapshots, so any value a producer can publish still decodes. The per-frame inflate cap is unchanged.merge::apply_bytesnow returns that growth, measuring only the patch and whatever it replaces or removes (a removed subtree is walked once as it leaves, already paid for by the bytes that inserted it). A frame's cost follows the frame, not the value it lands on, even for a value sitting at the cap. This also replaces the reserialize-per-frame checkreleaseran whenmax_sizewas set.@moq/jsonDecodermirrors it against@moq/flate'sDEFAULT_MAX_FRAME_SIZE, applying patches in place with the same accounting (a member counts its key, value, colon, and comma, so the count can be off from the compact JSON by a byte per object without an O(n) emptiness scan), and drops its value when it refuses a patch, matching Rust. Applying in place also drops the per-patch copy of every object on the patch path; a patch's absent keys are defined, not assigned, so a__proto__key stays a member.9e9) goes uncounted, but memory follows the parsed nodes, which those bytes bound.Dropped from the previous revision: the 4 MiB
DEFAULT_MAX_SIZEdefault (it broke compressed moq-stats tracks, whosepublisher.jsonexceeds 32 MiB on large relays, and leftmoq_mux::json, UniFFI, and C users with no opt-out) and the MSFError::FrameTooLargecheck (MSF frames are uncompressed, so the 32 MiB group cap already bounds them; the check could only throw).API and wire impact
Error::TooLargein Rust, a thrownErrorin JS) instead of growing without limit. A producer can never publish such a value as a snapshot.Verification
just check origin/release: passes (2856 Rust tests including moq-stats, JS suites, lint).unset_budget_still_bounds_compressed_state, andagrees_with_merge_patch_for_nested_additions_and_deletionsnow asserts the reported growth is exact (escaped keys, exponent numbers, removals, type changes); JScompressed patches cannot grow the value past the frame cap(including a deletion freeing room) anda __proto__ patch key stays a member. The existingpatches_cannot_accumulate_past_the_budgetcovers the explicit-limit path unchanged.snapshot_budgetnow compares the default against a value held exactly at its limit, over 100 to 10000 entries. At the limit a delta costs 84.6, 86.7, 88.8 ns, the same as the default, with no slope.oversized_frame_aborts_only_its_flavorexists only onmain; onreleasethe moq-stats suite passes with the 64 MiB default.(Written by Claude Opus 5.5)