Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
8c3d3c1
quest(m1): claim compress/core
kixelated Oct 19, 2026
837b312
moq-net: compress tracks with per-group DEFLATE
kixelated Oct 19, 2026
b80bc74
moq-net: test compressed groups, resume, and relaying
kixelated Oct 19, 2026
9714837
compress: carry the flag across js/net, bindings, drafts, and docs
kixelated Oct 19, 2026
265b643
compress: format, gate the raw-read test on async, and quiet clippy
kixelated Oct 19, 2026
0e2acb0
quest(m1): finish compress/core
kixelated Oct 19, 2026
57b9d0c
compress: link the MIGRATE entries to the PR
kixelated Oct 19, 2026
4b1cf9f
Merge remote-tracking branch 'origin/main' into quest/m1/compress/core
kixelated Oct 19, 2026
b5e0bf8
Merge remote-tracking branch 'origin/main' into compress-core-iter
kixelated Oct 19, 2026
811f50f
compress: codec id on the wire, inflate outside the group guard, bill…
kixelated Oct 19, 2026
3f617ff
Merge branch 'main' of github.com:moq-dev/moq into compress-core-iter
kixelated Oct 19, 2026
dc92e02
Merge origin/main into quest/m1/compress/core
kixelated Oct 19, 2026
5af7a6f
Merge origin/main into quest/m1/compress/core
kixelated Oct 19, 2026
d3aa99b
compress: a held copy continues an inflating reader from frame 0 too
kixelated Oct 19, 2026
555f076
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
b413d65
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
08d2da8
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
c0437ef
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
90c1a0e
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
11bf6a0
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
baa4465
Merge branch 'main' of github.com:moq-dev/moq into quest/m1/compress/…
kixelated Oct 20, 2026
9047b01
fix(net): resume compressed groups through main's head handoff
kixelated Oct 20, 2026
515b510
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
58eab0d
fix(net): bill a compressed group's codec state to the cache pool
kixelated Oct 20, 2026
56eb6d1
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
a197b34
fix(net): settle debt after inflating, meter compressed writes as stored
kixelated Oct 20, 2026
15bb446
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
1417eaf
test(net): gate the compressed metering test on async, drop an unused…
kixelated Oct 20, 2026
116d2eb
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
014c577
fix(net): meter compressed datagrams in the form the wire carries
kixelated Oct 20, 2026
ed5a60a
docs(net): received compressed datagrams count toward inflated
kixelated Oct 20, 2026
e10686c
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
f6c9004
test(lite): decode the TRACK_INFO body, not its length prefix
kixelated Oct 20, 2026
54a28db
fix(net): allow Stored::sent without the async feature
kixelated Oct 20, 2026
1d9c479
Merge remote-tracking branch 'origin/main' into compress-core-takeover
kixelated Oct 20, 2026
1698dbc
fix(net): reject final DEFLATE blocks, stop resuming malformed groups
kixelated Oct 20, 2026
4bf436b
Merge origin/main into quest/m1/compress/core
kixelated Oct 20, 2026
2061397
docs(go): note TrackInfo.Compress in the migration guide
kixelated Oct 20, 2026
66f4655
style(net): format the merge resolution
kixelated Oct 20, 2026
aa40b06
Merge origin/main into quest/m1/compress/core
kixelated Oct 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 14 additions & 0 deletions dart/moq_ffi/lib/src/moq.dart
Original file line number Diff line number Diff line change
Expand Up @@ -2640,11 +2640,13 @@ class MoqTrackInfo {
final int? maxAgeUs;
final int? refresh;
final int? timescale;
final bool compress;
MoqTrackInfo({
this.priority = 127,
this.maxAgeUs = null,
this.refresh = null,
this.timescale = null,
this.compress = false,
});
}

Expand Down Expand Up @@ -2675,12 +2677,18 @@ class FfiConverterMoqTrackInfo {
);
final timescale = timescale_lifted.value;
new_offset += timescale_lifted.bytesRead;
final compress_lifted = FfiConverterBool.read(
Uint8List.view(buf.buffer, new_offset),
);
final compress = compress_lifted.value;
new_offset += compress_lifted.bytesRead;
return LiftRetVal(
MoqTrackInfo(
priority: priority,
maxAgeUs: maxAgeUs,
refresh: refresh,
timescale: timescale,
compress: compress,
),
new_offset - buf.offsetInBytes,
);
Expand All @@ -2692,6 +2700,7 @@ class FfiConverterMoqTrackInfo {
FfiConverterOptionalUInt64.allocationSize(value.maxAgeUs) +
FfiConverterOptionalUInt32.allocationSize(value.refresh) +
FfiConverterOptionalUInt64.allocationSize(value.timescale) +
FfiConverterBool.allocationSize(value.compress) +
0;
final buf = Uint8List(total_length);
write(value, buf);
Expand All @@ -2716,6 +2725,10 @@ class FfiConverterMoqTrackInfo {
value.timescale,
Uint8List.view(buf.buffer, new_offset),
);
new_offset += FfiConverterBool.write(
value.compress,
Uint8List.view(buf.buffer, new_offset),
);
return new_offset - buf.offsetInBytes;
}

Expand All @@ -2724,6 +2737,7 @@ class FfiConverterMoqTrackInfo {
FfiConverterOptionalUInt64.allocationSize(value.maxAgeUs) +
FfiConverterOptionalUInt32.allocationSize(value.refresh) +
FfiConverterOptionalUInt64.allocationSize(value.timescale) +
FfiConverterBool.allocationSize(value.compress) +
0;
}
}
Expand Down
8 changes: 8 additions & 0 deletions doc/concept/moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,14 @@ untimed. moq-lite 05 and 06 cannot declare an untimed track, so there its
frames go out stamped with their send time; the work-in-progress 07 carries it
untimed.

A publisher may compress a track, which pays off for repetitive data such as
JSON and not for media. Each group's frames become one DEFLATE stream
([moq-flate](/draft/moq-flate)), and a datagram
compresses alone. Only the work-in-progress moq-lite 07 carries the compressed
bytes. A relay forwards them as received and inflates each group once for every
other subscriber, moq-transport included, so readers always see the original
frames. Rust publishes and relays compressed tracks; js/net refuses them for now.

A relay cancels its upstream subscription once nobody subscribes, but keeps its
copy of the track for 30 seconds after the last reader leaves, so a returning
reader or the next fetch finds it. That copy is not live meanwhile: readers get
Expand Down
2 changes: 2 additions & 0 deletions doc/concept/stats.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,7 @@ stale.
"subscriptions_started": 6, "subscriptions_ended": 2,
"fetches": 0,
"bytes": 1048576, "frames": 900, "groups": 30, "datagrams": 0,
"inflated": 0,
"stale": { "bytes": 0, "frames": 0, "groups": 0, "datagrams": 0 },
"announced": 1, "announced_closed": 0,
"broadcasts": 3, "broadcasts_closed": 1,
Expand All @@ -232,6 +233,7 @@ stale.
| `fetches` | One-shot group fetches requested, including ones that found nothing. Their payload counts in `bytes`, `frames`, and `groups`. |
| `bytes` / `frames` / `groups` | Payload delivered. |
| `datagrams` | Groups delivered as an unreliable datagram. A subset of `groups`. |
| `inflated` | Bytes inflated from compressed groups for plain readers, counted once per group however many share the copy, plus received compressed datagrams, which inflate on arrival. Not part of `bytes`, which counts payload as stored or sent. |
| `stale` | Payload skipped because it aged past a subscriber's latency budget, with the same four fields. Disjoint from the top-level payload counters. |

The last six fields are legacy spellings of the `*_started` and `*_ended`
Expand Down
1 change: 1 addition & 0 deletions doc/lib/c/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ inside. The header is the reference: each function carries a doc comment.
- **Errors.** Negative return codes are named by `MOQ_ERROR_*`, and `moq_error()` gives the reason on the calling thread. Auth rejections have their own codes so you don't retry them. For the peer's protocol code use `moq_error_protocol()`; do not parse `moq_error()`. A panic in a binding background task reports `MOQ_ERROR_PANIC` through its terminal callback; unrelated objects keep running. A session closed because its credential expired has kind `MOQ_PROTOCOL_KIND_EXPIRED`, and a request refused for an expired token returns `MOQ_ERROR_EXPIRED`; refresh the token before retrying. A catalog that needs a newer moq-hang draft than this build reads returns `MOQ_ERROR_UNSUPPORTED_CATALOG`; upgrade the library to read it. A request-cap closure has kind `MOQ_PROTOCOL_KIND_TOO_MANY_REQUESTS`; reduce the workload or adjust the relay's [session limits](/bin/relay/config#session-and-peer) before reconnecting.
- **Threading.** Call any function from any thread. Raw publish calls block until the codec takes the frame, which paces a publisher.
- **Zeroed config is the default.** A zeroed `moq_client_config` means the default for every field, so new fields don't break callers. A field whose default isn't zero has a `has_*` flag: `backoff_timeout_us = 0` means "retry forever" only with `has_backoff_timeout = true`.
- **Compressed tracks.** `moq_track_info.compress` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames. `@moq/net` cannot read a compressed track yet.
- **Audio needs cuts.** Video groups at its keyframes, but audio forms a group only where you call `moq_publish_media_cut`: after every frame, or at a segment cadence to align with video.
- **Live encoder timing.** After writing a frame you encoded yourself, call `moq_publish_media_flush` with the same timestamp so the catalog advertises your jitter. Skip it for file and network imports. On a seek or pause, call `moq_publish_media_discontinuity`, then keep timestamps moving forward and resume video on a keyframe.
- **JSON values.** `moq_consume_json_value` fills `moq_json_value` with the document and its frame's `timestamp_us`, rounded down to a whole microsecond; `timestamp_present` is false on an untimed track. A snapshot consumer's callback fires for each state in order, but a group superseded before it is read is skipped.
Expand Down
4 changes: 4 additions & 0 deletions doc/lib/cpp/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,10 @@ Audio and video convenience decoders reject a catalog configuration with `enable

Hold the owned `BandwidthConsumer` returned by `session->recv_bandwidth()` to receive bandwidth estimates. `peek()` reads the current estimate in bits per second and `changed()` waits for a new one. Releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected.

## Compressed tracks

`moq::TrackInfo::compress` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames. `@moq/net` cannot read a compressed track yet.

## Track demand

`demand()->wanted()` is the group subscribers want, or `std::nullopt` while nobody subscribes, and `wanted_changed()` waits for it to move: start a new group once it passes your largest. Set `TrackInfo::refresh` to have a joining subscriber ask for a new group once the newest has grown past `refresh` times its first frame.
Expand Down
1 change: 1 addition & 0 deletions doc/lib/dart/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ A missing data timestamp stays untimed; the library never substitutes zero or no
`package:moq/json.dart` consumers return `Value` (`value`, `timestampUs`) from `next()`; a snapshot consumer
yields each state in order, and `latest()` skips to the newest. Read timestamps are rounded down to a whole microsecond.
Raw tracks with a `null` timescale are untimed on both publish and receive.
`MoqTrackInfo.compress` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames. `@moq/net` cannot read a compressed track yet.
Whole-frame media writes require caller time; container imports keep their own timeline handling.
Timed raw/data tracks require caller-supplied timestamps. Use `CatalogProducer.now()` to stamp a locally
created event on the broadcast clock; reading it fixes that clock for later media producers.
Expand Down
1 change: 1 addition & 0 deletions doc/lib/go/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ and a `Named` or `Requested` target. `NewCatalogProducer` holds its broadcast we
catalog writes fail with `ErrClosed` after the broadcast closes. Media timestamps use `time.Duration`.
A missing data timestamp stays untimed; the library never substitutes zero or now.
Raw tracks with a `nil` timescale are untimed on both publish and receive.
`TrackInfo.Compress` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames. `@moq/net` cannot read a compressed track yet.
`moq.dev/moq/flate` publishes opaque snapshot and stream payloads; its `Update` / `Append`
and the JSON producers take a final `*time.Duration` timestamp argument (`nil` for untimed).
JSON consumers return a `*Timed` holding the value and that timestamp; `SnapshotConsumer.Latest`
Expand Down
1 change: 1 addition & 0 deletions doc/lib/kt/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ A missing data timestamp stays untimed; the library never substitutes zero or no
`dev.moq.json` consumers' `values()` emits `Value` (`value`, `timestamp`) and `valuesAs<T>()` emits `Timed<T>`;
a snapshot consumer yields each state in order, and `latest()` skips to the newest. Read timestamps are rounded down to a whole microsecond.
Raw tracks with a `null` timescale are untimed on both publish and receive.
`MoqTrackInfo.compress` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames. `@moq/net` cannot read a compressed track yet.
Whole-frame media writes require caller time; container imports keep their own timeline handling.
Timed raw/data tracks require caller-supplied timestamps. Use `CatalogProducer.now()` to stamp a locally
created event on the broadcast clock; reading it fixes that clock for later media producers.
Expand Down
1 change: 1 addition & 0 deletions doc/lib/py/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ A missing data timestamp stays untimed; the library never substitutes zero or no
JSON consumers yield `moq.json.Timed(value, timestamp)`; a snapshot consumer yields each state in order,
and `await consumer.latest()` skips to the newest. Read timestamps are rounded down to a whole microsecond.
Raw tracks with a `None` timescale are untimed on both publish and receive.
`TrackInfo(compress=True)` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames. `@moq/net` cannot read a compressed track yet.
`moq.flate.SnapshotProducer` / `StreamProducer` publish opaque payloads; their `update` / `append`
and the JSON producers accept `timestamp=timedelta(...)` (`None` for untimed).
Whole-frame media writes require caller time; container imports keep their own timeline handling.
Expand Down
1 change: 1 addition & 0 deletions doc/lib/swift/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ A missing data timestamp stays untimed; the library never substitutes zero or no
`Json` consumers yield `Json.Timed` (`value`, `timestampUs`); a snapshot consumer yields each state in order,
and `latest()` skips to the newest. Read timestamps are rounded down to a whole microsecond.
Raw tracks with a `nil` timescale are untimed on both publish and receive.
`MoqTrackInfo.compress` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames. `@moq/net` cannot read a compressed track yet.
Whole-frame media writes require caller time; container imports keep their own timeline handling.
Timed raw/data tracks require caller-supplied timestamps. Use `Media.CatalogProducer.now()` to stamp a locally
created event on the broadcast clock; reading it fixes that clock for later media producers.
Expand Down
31 changes: 22 additions & 9 deletions drafts/draft-lcurley-moq-flate.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ informative:
This document specifies how a MoQ Transport {{moqt}} track carries DEFLATE-compressed payloads.
Each subgroup is one raw DEFLATE stream, sync flushed at each object boundary, so every object stays self-delimited while later objects compress against the earlier ones in the subgroup.
Small repetitive payloads compress several times better than they do alone, and a dropped group costs nothing beyond itself because the window never spans one.
Nothing is added to the wire: the application declares the track compressed, and a relay forwards it unchanged.
The transport declares a track compressed and carries each object's inflated size, so a relay forwards the compressed bytes to a peer that understands them and inflates them for one that does not.

--- middle

Expand All @@ -53,7 +53,7 @@ Each subgroup is a separate DEFLATE stream: its object payloads share one window

The application decides which subgroups a track uses; a consumer knows which to expect and decodes each one's objects in order.

A publisher MUST NOT send a compressed track in datagrams, which are neither ordered nor reliable.
A datagram carries a group of one object, so it is a stream of its own: compressed alone, starting cold.

An object with no payload is skipped, neither advancing nor resetting the window.

Expand All @@ -68,19 +68,23 @@ A publisher MUST omit it from each object and a consumer MUST append it before d

A zero-length payload is neither compressed nor decompressed.

The stream is never terminated: no object ends in a final block, so a consumer decompresses incrementally and MUST NOT treat the absent end of stream as truncation.
The stream is never terminated: a publisher MUST NOT emit a final block, and a consumer MUST treat one as malformed and abandon the subgroup.
A consumer decompresses incrementally and MUST NOT treat the absent end of stream as truncation.

A consumer missing an object MUST abandon the rest of the subgroup, which it can no longer decompress.
The compression level is a publisher's choice; any conformant stream decodes.


# Declaring Compression {#declare}
The application declares a track compressed, conventionally with a `.z` suffix on the track name, or in its catalog.
The transport declares a track compressed, and gives each object's inflated size beside its compressed one.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
{{moql}} does both from moq-lite-07: a Compression codec id in TRACK_INFO, and an Inflated Size on each FRAME and datagram.
A consumer MUST NOT infer compression from the payload: raw DEFLATE has no magic number, so a wrong guess yields plausible garbage.

There is deliberately no transport signal.
A relay drops objects and re-frames what it forwards, so a transport that knew about the window would make the relay responsible for emitting a stream that still decodes, which means carrying a DEFLATE implementation.
Instead a relay forwards opaque bytes, and an endpoint that has never heard of this document never asks for a compressed track.
A consumer MUST treat an object that inflates to more or less than its declared size as malformed, and abandon the subgroup.

A relay forwards the compressed objects as received to a peer whose transport declares compression.
For any other peer it inflates them, once per subgroup however many such peers it serves, and sends the original objects without the declaration.
A peer is never sent compressed bytes it was not told about.


# Security Considerations
Expand All @@ -89,18 +93,27 @@ The compressed size of one payload reveals how much it has in common with the pa
A publisher MUST NOT compress a secret and attacker-influenced data in the same subgroup.
Encryption does not mitigate this, since the sizes are visible to anyone on the path.

A few bytes can inflate to gigabytes, so a consumer MUST bound the decompressed size of each object and abandon the subgroup once that bound is exceeded.
A few bytes can inflate to gigabytes, so a consumer MUST check each declared inflated size against its limits before decompressing, and never inflate past it.
Each open subgroup also holds a window of up to 32 KiB, so a consumer SHOULD bound how many it decompresses at once.


# IANA Considerations

This document requests no registrations.
The format lives inside object payloads, which {{moqt}} treats as opaque.
The format lives inside object payloads, and the declaration in the transport that carries them.


--- back

# Changelog

## Since draft-lcurley-moq-flate-00 (in progress)

- Compression is declared by the transport instead of the application: {{moql}} carries a Compression codec in TRACK_INFO and each object's Inflated Size on FRAME and datagrams.
- A relay inflates for a peer whose transport does not declare compression.
- A datagram is compressed alone, as a stream of its own.
- A consumer rejects an object that inflates to other than its declared size, or that ends in a final block.

# Acknowledgments
{:numbered="false"}

Expand Down
2 changes: 2 additions & 0 deletions drafts/draft-lcurley-moq-ietf.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,7 @@ DATAGRAM Body {
[Status (i)]
[Properties Length (i), Properties (..)]
[Timestamp (i)]
[Inflated Size (i)]
Payload (b)
}
~~~
Expand All @@ -240,6 +241,7 @@ A bit field, or 0 for a plain datagram, which has none of the optional fields be
A receiver MUST drop a datagram with any other bit, or a bit without RELAYED.

A datagram with a Status MUST have an empty Payload, as {{moqt}} sends no payload beside one; a Normal (0) Status is an empty object.
It has no Inflated Size ({{moql}} DATAGRAM), even on a compressed track, as there is no payload to inflate.
A non-zero Status carries no object: it has no Timestamp, even on a timed track.
A receiver MUST drop a datagram whose Status no {{moqt}} draft defines: only 0 (Normal), 1 (DOES_NOT_EXIST, drafts 14 and 15), 3 (END_OF_GROUP), and 4 (END_OF_TRACK) are relayed.
It MUST also drop one whose fields no {{moqt}} OBJECT_DATAGRAM Type can carry, such as STATUS with END_OF_GROUP.
Expand Down
Loading
Loading