From 7814783d1f9385a56f74f38e60d8c893b0944729 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 28 Oct 2026 00:01:44 -0700 Subject: [PATCH 1/4] chore(quest): claim quest/m1/compress/migrate.md Quest-Claim: 7e9616fbcd4a039d From 3c8d290f7f9f5219e1f7133dcc75fe8a01ca44e2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 28 Oct 2026 01:03:56 -0700 Subject: [PATCH 2/4] refactor!: one compressed track per name Every first-party compressed track is now one name with compress: true: catalog.json, the moq-stats tracks, HLS records, and the CLI's reports. catalog.json.z, the stats .z siblings, hang's catalog compression field, hangz, moq-flate, @moq/flate, moq-mux::binary, moq-json's Config::compression, and the bindings' flate namespace are deleted. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/release-gate.yml | 2 - .oss-scanner/Dockerfile | 2 +- .oss-scanner/threat_model.md | 2 +- Cargo.lock | 15 - Cargo.toml | 2 - bun.lock | 23 +- dart/moq/MIGRATE.md | 6 + dart/moq/lib/flate.dart | 13 - dart/moq/lib/json.dart | 18 +- dart/moq/test/moq_test.dart | 50 +- demo/carla/sim/moq_carla_sim/publish.py | 7 +- demo/carla/sim/tests/conftest.py | 7 +- demo/carla/sim/tests/test_runtime.py | 2 +- demo/carla/web/src/preview.ts | 3 +- demo/carla/web/src/truck.test.ts | 6 +- demo/carla/web/src/truck.ts | 9 +- doc/.vitepress/api/package.json | 1 - doc/.vitepress/config.ts | 2 - doc/bin/inspect.md | 11 +- doc/concept/hang.md | 20 +- doc/concept/stats.md | 58 +- doc/lib/c/index.md | 2 +- doc/lib/cpp/index.md | 2 +- doc/lib/dart/index.md | 4 +- doc/lib/go/index.md | 5 +- doc/lib/js/flate.md | 47 - doc/lib/js/index.md | 1 - doc/lib/js/json.md | 3 +- doc/lib/js/net.md | 2 +- doc/lib/js/watch.md | 2 +- doc/lib/kt/index.md | 4 +- doc/lib/layers.md | 22 +- doc/lib/py/index.md | 8 +- doc/lib/rs/index.md | 1 - doc/lib/rs/moq-flate.md | 57 -- doc/lib/rs/moq-json.md | 3 +- doc/lib/swift/index.md | 4 +- doc/setup/upgrade.md | 4 +- drafts/draft-lcurley-moq-e2ee.md | 6 +- drafts/draft-lcurley-moq-hang.md | 38 +- go/wrapper/MIGRATE.md | 6 + go/wrapper/README.md | 11 +- go/wrapper/data_timestamps_test.go | 44 +- go/wrapper/flate/flate.go | 66 -- go/wrapper/json/json.go | 58 +- go/wrapper/json/json_test.go | 22 +- js/flate/MIGRATE.md | 28 - js/flate/README.md | 53 - js/flate/api/index.api.md | 81 -- js/flate/bench/decoder.ts | 63 -- js/flate/bench/tsconfig.json | 5 - js/flate/package.json | 34 - js/flate/src/codec.test.ts | 219 ---- js/flate/src/codec.ts | 145 --- js/flate/src/compression.ts | 15 - js/flate/src/index.ts | 49 - js/flate/src/snapshot/consumer.ts | 129 --- js/flate/src/snapshot/index.ts | 19 - js/flate/src/snapshot/producer.ts | 82 -- js/flate/src/snapshot/snapshot.test.ts | 239 ----- js/flate/src/stream/consumer.ts | 152 --- js/flate/src/stream/index.ts | 26 - js/flate/src/stream/producer.ts | 117 --- js/flate/src/stream/stream.test.ts | 298 ------ js/flate/src/types.test.ts | 45 - js/flate/tsconfig.build.json | 7 - js/flate/tsconfig.json | 9 - js/hang/src/catalog/binary.ts | 6 +- js/hang/src/catalog/compression.ts | 27 - js/hang/src/catalog/consumer.test.ts | 2 +- js/hang/src/catalog/data.test.ts | 15 +- js/hang/src/catalog/format.ts | 5 +- js/hang/src/catalog/index.ts | 1 - js/hang/src/catalog/json.ts | 6 +- js/json/MIGRATE.md | 9 +- js/json/README.md | 2 +- js/json/bench/codec.ts | 11 +- js/json/bench/stream.ts | 14 +- js/json/package.json | 4 +- js/json/src/compression.ts | 15 - js/json/src/error.ts | 8 - js/json/src/index.ts | 3 +- js/json/src/snapshot/codec.test.ts | 34 +- js/json/src/snapshot/compression.test.ts | 155 --- js/json/src/snapshot/consumer.ts | 4 +- js/json/src/snapshot/decoder.ts | 27 +- js/json/src/snapshot/encoder.ts | 88 +- js/json/src/snapshot/producer.ts | 3 + js/json/src/snapshot/properties.test.ts | 7 +- js/json/src/snapshot/snapshot.test.ts | 93 +- js/json/src/stream/codec.test.ts | 134 +-- js/json/src/stream/consumer.ts | 96 +- js/json/src/stream/decoder.ts | 21 +- js/json/src/stream/encoder.ts | 96 +- js/json/src/stream/index.ts | 15 +- js/json/src/stream/producer.ts | 6 +- js/json/src/stream/stream.test.ts | 70 +- js/json/src/types.test.ts | 20 +- js/json/src/window/consumer.ts | 11 +- js/json/src/window/decoder.ts | 20 +- js/json/src/window/encoder.ts | 39 +- js/json/src/window/index.ts | 4 +- js/json/src/window/producer.ts | 2 +- js/json/src/window/properties.test.ts | 3 +- js/json/src/window/window.test.ts | 31 +- js/publish/src/broadcast.ts | 37 +- js/publish/src/catalog.ts | 11 +- js/watch/README.md | 2 +- js/watch/src/broadcast.test.ts | 14 +- js/watch/src/broadcast.ts | 25 +- js/watch/src/element.test.ts | 2 +- kt/moq/MIGRATE.md | 6 + .../kotlin/dev/moq/flate/Tracks.kt | 8 - .../kotlin/dev/moq/json/Tracks.kt | 13 +- .../kotlin/dev/moq/SmokeTest.kt | 53 +- package.json | 1 - py/moq-rs/MIGRATE.md | 5 + py/moq-rs/README.md | 5 +- py/moq-rs/docs/index.md | 13 - py/moq-rs/moq/__init__.py | 1 - py/moq-rs/moq/flate.py | 60 -- py/moq-rs/moq/json.py | 53 +- py/moq-rs/tests/test_data_timestamps.py | 22 - py/moq-rs/tests/test_local.py | 16 +- quest/m1/api-audit/README.md | 6 +- quest/m1/compress/README.md | 6 +- quest/m1/compress/migrate.md | 59 -- quest/m1/unbounded-groups/README.md | 2 +- quest/m1/unbounded-groups/front-eviction.md | 4 +- quest/m1/unbounded-groups/js.md | 2 +- quest/m2/e2ee/rust-publish.md | 2 +- quest/m2/net-wrap/e2ee.md | 2 +- quest/m2/net-wrap/mux.md | 2 +- quest/m2/release/migrate-audit.md | 4 +- quest/m2/stats/README.md | 4 +- quest/m2/stats/js.md | 2 +- quest/m3/ladder/fetch.md | 4 +- quest/m3/teleop-mavlink.md | 2 +- quest/m3/teleop-robot.md | 4 +- rs/hang/MIGRATE.md | 7 + rs/hang/src/catalog/binary.rs | 15 +- rs/hang/src/catalog/compression.rs | 63 -- rs/hang/src/catalog/consumer.rs | 15 +- rs/hang/src/catalog/json.rs | 15 +- rs/hang/src/catalog/mod.rs | 2 - rs/hang/src/catalog/root.rs | 34 +- rs/hang/src/hls.rs | 2 +- rs/moq-c/MIGRATE.md | 6 + rs/moq-c/src/api.rs | 203 +--- rs/moq-c/src/consume.rs | 22 +- rs/moq-c/src/publish.rs | 108 +- rs/moq-c/src/test.rs | 185 +--- rs/moq-cli/src/play/args.rs | 4 +- rs/moq-cli/src/report.rs | 117 +-- rs/moq-cli/src/subscribe.rs | 5 +- rs/moq-ffi/MIGRATE.md | 6 + rs/moq-ffi/src/flate.rs | 180 ---- rs/moq-ffi/src/json.rs | 127 +-- rs/moq-ffi/src/lib.rs | 1 - rs/moq-ffi/src/test.rs | 182 +--- rs/moq-flate/CHANGELOG.md | 32 - rs/moq-flate/Cargo.toml | 29 - rs/moq-flate/MIGRATE.md | 33 - rs/moq-flate/README.md | 25 - rs/moq-flate/clippy.toml | 23 - rs/moq-flate/src/lib.rs | 440 -------- rs/moq-flate/src/snapshot/consumer.rs | 276 ----- rs/moq-flate/src/snapshot/mod.rs | 315 ------ rs/moq-flate/src/snapshot/producer.rs | 125 --- rs/moq-flate/src/stream/consumer.rs | 159 --- rs/moq-flate/src/stream/mod.rs | 562 ---------- rs/moq-flate/src/stream/producer.rs | 199 ---- rs/moq-json/Cargo.toml | 1 - rs/moq-json/MIGRATE.md | 11 +- rs/moq-json/benches/allocations.rs | 29 +- rs/moq-json/benches/codec.rs | 132 +-- rs/moq-json/benches/table.rs | 6 +- rs/moq-json/examples/telemetry.rs | 102 +- rs/moq-json/src/lib.rs | 30 - rs/moq-json/src/snapshot/consumer.rs | 69 +- rs/moq-json/src/snapshot/decoder.rs | 158 +-- rs/moq-json/src/snapshot/encoder.rs | 314 ++---- rs/moq-json/src/snapshot/mod.rs | 106 +- rs/moq-json/src/snapshot/producer.rs | 3 + rs/moq-json/src/stream/consumer.rs | 15 +- rs/moq-json/src/stream/decoder.rs | 129 +-- rs/moq-json/src/stream/encoder.rs | 119 +-- rs/moq-json/src/stream/mod.rs | 102 +- rs/moq-json/src/stream/producer.rs | 9 +- rs/moq-json/src/window/consumer.rs | 5 +- rs/moq-json/src/window/decoder.rs | 39 +- rs/moq-json/src/window/encoder.rs | 89 +- rs/moq-json/src/window/mod.rs | 33 +- rs/moq-json/src/window/producer.rs | 2 +- rs/moq-json/src/window/properties.rs | 13 +- rs/moq-mux/Cargo.toml | 1 - rs/moq-mux/MIGRATE.md | 7 + rs/moq-mux/src/binary.rs | 961 ------------------ rs/moq-mux/src/catalog/consumer.rs | 12 +- rs/moq-mux/src/catalog/data.rs | 15 +- rs/moq-mux/src/catalog/entry.rs | 7 +- rs/moq-mux/src/catalog/format.rs | 8 +- rs/moq-mux/src/catalog/hang/consumer.rs | 23 +- rs/moq-mux/src/catalog/producer.rs | 92 +- rs/moq-mux/src/catalog/tracks.rs | 18 +- rs/moq-mux/src/container/fmp4/import_test.rs | 8 +- rs/moq-mux/src/container/ts/export_test.rs | 48 +- rs/moq-mux/src/error.rs | 8 - rs/moq-mux/src/hls.rs | 7 +- rs/moq-mux/src/json.rs | 95 +- rs/moq-mux/src/lib.rs | 13 - rs/moq-relay/src/stats.rs | 9 +- rs/moq-stats/MIGRATE.md | 13 +- rs/moq-stats/src/aggregate.rs | 45 +- rs/moq-stats/src/consume.rs | 83 +- rs/moq-stats/src/lib.rs | 156 +-- rs/moq-stats/src/produce.rs | 555 +++++----- rs/moq-stats/src/produce_bench.rs | 10 +- swift/MIGRATE.md | 5 + swift/README.md | 4 +- swift/Sources/Moq/Flate.swift | 43 - swift/Sources/Moq/Json.swift | 52 +- swift/Tests/MoqTests/SmokeTests.swift | 35 +- 223 files changed, 1572 insertions(+), 9849 deletions(-) delete mode 100644 dart/moq/lib/flate.dart delete mode 100644 doc/lib/js/flate.md delete mode 100644 doc/lib/rs/moq-flate.md delete mode 100644 go/wrapper/flate/flate.go delete mode 100644 js/flate/MIGRATE.md delete mode 100644 js/flate/README.md delete mode 100644 js/flate/api/index.api.md delete mode 100644 js/flate/bench/decoder.ts delete mode 100644 js/flate/bench/tsconfig.json delete mode 100644 js/flate/package.json delete mode 100644 js/flate/src/codec.test.ts delete mode 100644 js/flate/src/codec.ts delete mode 100644 js/flate/src/compression.ts delete mode 100644 js/flate/src/index.ts delete mode 100644 js/flate/src/snapshot/consumer.ts delete mode 100644 js/flate/src/snapshot/index.ts delete mode 100644 js/flate/src/snapshot/producer.ts delete mode 100644 js/flate/src/snapshot/snapshot.test.ts delete mode 100644 js/flate/src/stream/consumer.ts delete mode 100644 js/flate/src/stream/index.ts delete mode 100644 js/flate/src/stream/producer.ts delete mode 100644 js/flate/src/stream/stream.test.ts delete mode 100644 js/flate/src/types.test.ts delete mode 100644 js/flate/tsconfig.build.json delete mode 100644 js/flate/tsconfig.json delete mode 100644 js/hang/src/catalog/compression.ts delete mode 100644 js/json/src/compression.ts delete mode 100644 js/json/src/snapshot/compression.test.ts delete mode 100644 kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/flate/Tracks.kt delete mode 100644 py/moq-rs/moq/flate.py delete mode 100644 quest/m1/compress/migrate.md delete mode 100644 rs/hang/src/catalog/compression.rs delete mode 100644 rs/moq-ffi/src/flate.rs delete mode 100644 rs/moq-flate/CHANGELOG.md delete mode 100644 rs/moq-flate/Cargo.toml delete mode 100644 rs/moq-flate/MIGRATE.md delete mode 100644 rs/moq-flate/README.md delete mode 100644 rs/moq-flate/clippy.toml delete mode 100644 rs/moq-flate/src/lib.rs delete mode 100644 rs/moq-flate/src/snapshot/consumer.rs delete mode 100644 rs/moq-flate/src/snapshot/mod.rs delete mode 100644 rs/moq-flate/src/snapshot/producer.rs delete mode 100644 rs/moq-flate/src/stream/consumer.rs delete mode 100644 rs/moq-flate/src/stream/mod.rs delete mode 100644 rs/moq-flate/src/stream/producer.rs delete mode 100644 rs/moq-mux/src/binary.rs delete mode 100644 swift/Sources/Moq/Flate.swift diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.yml index f8d91b7ef38..a641b23a8dd 100644 --- a/.github/workflows/release-gate.yml +++ b/.github/workflows/release-gate.yml @@ -73,7 +73,6 @@ jobs: - "rs/moq-loc/**" - "rs/moq-msf/**" - "rs/moq-json/**" - - "rs/moq-flate/**" - "rs/moq-pattern/**" - "rs/hang/**" - "rs/kio/**" @@ -234,7 +233,6 @@ jobs: - "rs/moq-cli/**" - "rs/moq-clock/**" - "rs/moq-congestion/**" - - "rs/moq-flate/**" - "rs/moq-hls/**" - "rs/moq-ice/**" - "rs/moq-json/**" diff --git a/.oss-scanner/Dockerfile b/.oss-scanner/Dockerfile index 5b368eed864..338e2689749 100644 --- a/.oss-scanner/Dockerfile +++ b/.oss-scanner/Dockerfile @@ -20,7 +20,7 @@ RUN rustup show active-toolchain || rustup toolchain install \ && curl -LsSf "https://get.nexte.st/latest/$nextest" | tar zxf - -C "$CARGO_HOME/bin" \ && cargo nextest --version -ARG CRATES="kio hang moq-auth moq-clock moq-e2ee moq-flate moq-hls moq-ice moq-json moq-loc moq-msf moq-mux moq-net moq-pattern moq-quic moq-relay moq-rtc moq-rtmp moq-rtsp moq-sock moq-srt moq-stats moq-tokio moq-uring moq-cli" +ARG CRATES="kio hang moq-auth moq-clock moq-e2ee moq-hls moq-ice moq-json moq-loc moq-msf moq-mux moq-net moq-pattern moq-quic moq-relay moq-rtc moq-rtmp moq-rtsp moq-sock moq-srt moq-stats moq-tokio moq-uring moq-cli" # Path specs, because the lockfile also has published copies of some workspace crates. # io-uring compiles the relay's io_uring QUIC workers (`--runtime-io-uring`), which the default features leave out. diff --git a/.oss-scanner/threat_model.md b/.oss-scanner/threat_model.md index cbde6250dc8..fc98d344028 100644 --- a/.oss-scanner/threat_model.md +++ b/.oss-scanner/threat_model.md @@ -35,7 +35,7 @@ Most important, in this order: 2. Authorization: a session publishing, subscribing, or seeing announcements outside its grant, or reaching another root (tenant) through path tricks, revalidation, mounts, or cluster links. 3. `rs/moq-mux` and `rs/hang`: container, codec, and catalog parsing of publisher-controlled bytes. 4. The ingest gateways: `rs/moq-rtmp`, `rs/moq-rtsp`, `rs/moq-srt`, `rs/moq-rtc`, `rs/moq-ice`, `rs/moq-hls`. -5. `rs/moq-e2ee`, `rs/moq-cli`, `rs/moq-flate`, `rs/moq-json`, `rs/moq-stats`, `rs/kio`, `rs/moq-clock`. +5. `rs/moq-e2ee`, `rs/moq-cli`, `rs/moq-json`, `rs/moq-stats`, `rs/kio`, `rs/moq-clock`. `unsafe` is rare and worth reading closely where it appears: `rs/moq-quic`, `rs/moq-net/src/model`, `rs/moq-net/src/util/cell.rs`, `rs/moq-net/src/transport.rs`, and `rs/moq-relay/src/uring.rs`. diff --git a/Cargo.lock b/Cargo.lock index d43faae9d44..ca0199dd469 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5319,19 +5319,6 @@ dependencies = [ "web-transport-wasm", ] -[[package]] -name = "moq-flate" -version = "0.2.0" -dependencies = [ - "bytes", - "flate2", - "futures", - "kio 0.6.2", - "moq-net", - "thiserror 2.0.21", - "tracing", -] - [[package]] name = "moq-gst" version = "0.4.12" @@ -5409,7 +5396,6 @@ dependencies = [ "futures", "json-patch", "kio 0.6.2", - "moq-flate", "moq-net", "proptest", "serde", @@ -5455,7 +5441,6 @@ dependencies = [ "libc", "memchr", "moq-clock", - "moq-flate", "moq-json", "moq-loc", "moq-msf", diff --git a/Cargo.toml b/Cargo.toml index c96c253118d..829babb00f7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -18,7 +18,6 @@ default-members = [ "rs/moq-e2ee", # "rs/moq-ffi", # requires Python/maturin # "rs/moq-gst", # requires GStreamer - "rs/moq-flate", "rs/moq-hls", "rs/moq-ice", "rs/moq-json", @@ -142,7 +141,6 @@ moq-auth = { version = "0.2.1", path = "rs/moq-auth" } moq-clock = { version = "0.11.0", path = "rs/moq-clock" } moq-clock-macros = { version = "0.1.0", path = "rs/moq-clock/macros" } moq-congestion = { version = "0.1.0", path = "rs/moq-congestion" } -moq-flate = { version = "0.2.0", path = "rs/moq-flate" } moq-hls = { version = "0.5.11", path = "rs/moq-hls", default-features = false } moq-ice = { version = "0.1.0", path = "rs/moq-ice" } moq-json = { version = "0.5.8", path = "rs/moq-json" } diff --git a/bun.lock b/bun.lock index c5ce21e30a9..e16e05b375a 100644 --- a/bun.lock +++ b/bun.lock @@ -113,7 +113,6 @@ "@moq/auth": "workspace:*", "@moq/boy": "workspace:*", "@moq/clock": "workspace:*", - "@moq/flate": "workspace:*", "@moq/hang": "workspace:*", "@moq/json": "workspace:*", "@moq/libav": "workspace:*", @@ -178,22 +177,6 @@ "typescript": "7.0.2", }, }, - "js/flate": { - "name": "@moq/flate", - "version": "0.1.3", - "dependencies": { - "@moq/net": "workspace:^", - "@moq/signals": "workspace:^", - "pako": "^3.0.2", - }, - "devDependencies": { - "@moq/clock": "workspace:^", - "@types/bun": "^1.4.3", - "fflate": "^0.8.2", - "rimraf": "^6.1.3", - "typescript": "7.0.2", - }, - }, "js/hang": { "name": "@moq/hang", "version": "0.5.2", @@ -219,8 +202,8 @@ "name": "@moq/json", "version": "0.4.2", "dependencies": { - "@moq/flate": "workspace:^", "@moq/net": "workspace:^", + "@moq/signals": "workspace:^", }, "devDependencies": { "@moq/clock": "workspace:^", @@ -878,8 +861,6 @@ "@moq/e2ee": ["@moq/e2ee@workspace:js/e2ee"], - "@moq/flate": ["@moq/flate@workspace:js/flate"], - "@moq/hang": ["@moq/hang@workspace:js/hang"], "@moq/interop-browser": ["@moq/interop-browser@workspace:test/interop/clients/js"], @@ -1366,8 +1347,6 @@ "fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="], - "fflate": ["fflate@0.8.3", "", {}, "sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA=="], - "focus-trap": ["focus-trap@7.8.0", "", { "dependencies": { "tabbable": "^6.4.0" } }, "sha512-/yNdlIkpWbM0ptxno3ONTuf+2g318kh2ez3KSeZN5dZ8YC6AAmgeWz+GasYYiBJPFaYcSAPeu4GfhUaChzIJXA=="], "foreground-child": ["foreground-child@3.3.1", "", { "dependencies": { "cross-spawn": "^7.0.6", "signal-exit": "^4.0.1" } }, "sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw=="], diff --git a/dart/moq/MIGRATE.md b/dart/moq/MIGRATE.md index 6dbf79168f8..003f152eabc 100644 --- a/dart/moq/MIGRATE.md +++ b/dart/moq/MIGRATE.md @@ -4,6 +4,12 @@ How to update your code for each breaking change, newest first. ## Unreleased +### Compression moves to the track + +`package:moq/flate.dart` is removed: publish opaque bytes on a `TrackProducer`. +`SnapshotConfig` and `WindowConfig` lose `compression`, and `StreamConfig` is removed. Publish the track with `MoqTrackInfo(compress: true)` instead. +`StreamProducer` drops its `config`, and `SnapshotConsumer`, `StreamConsumer`, and `WindowConsumer` take only the track. + ### Paths with an empty segment are refused ([#6648](https://github.com/moq-dev/moq/pull/6648)) Publishing, serving, or requesting a broadcast at a path with a leading, trailing, or doubled `/` throws instead of collapsing it. Pass the canonical path, such as `room/cam`. diff --git a/dart/moq/lib/flate.dart b/dart/moq/lib/flate.dart deleted file mode 100644 index f5afdf327de..00000000000 --- a/dart/moq/lib/flate.dart +++ /dev/null @@ -1,13 +0,0 @@ -/// Opaque snapshot and stream tracks, mirroring the moq-flate crate. -library; - -import 'package:moq_ffi/moq_ffi.dart'; - -/// Publishes the latest opaque payload on a track it takes over. -typedef SnapshotProducer = MoqFlateSnapshotProducer; - -/// Publishes an ordered opaque payload log on a track it takes over. -typedef StreamProducer = MoqFlateStreamProducer; - -/// Configures compression and the opaque payload media type. -typedef Config = MoqFlateConfig; diff --git a/dart/moq/lib/json.dart b/dart/moq/lib/json.dart index 73b62c37347..a3df4dfb0c8 100644 --- a/dart/moq/lib/json.dart +++ b/dart/moq/lib/json.dart @@ -3,15 +3,20 @@ /// Each type wraps a track: a producer takes over a `TrackProducer` (the /// snapshot and stream producers also advertise it in the broadcast's catalog, /// which has no window mode), and a consumer takes over a `TrackConsumer` that -/// has not read a group yet. Values cross as JSON -/// strings. Import it with a prefix, since `dart:convert` has a `json` too: +/// has not read a group yet. Values cross as JSON strings. Compression is the +/// track's: publish JSON with `MoqTrackInfo(compress: true)`, and a consumer +/// reads it with no option of its own. Import it with a prefix, since +/// `dart:convert` has a `json` too: /// /// ```dart /// import 'package:moq/json.dart' as moq_json; /// /// final status = moq_json.SnapshotProducer( /// broadcast: broadcast, -/// track: broadcast.publishTrack(name: 'status', info: null), +/// track: broadcast.publishTrack( +/// name: 'status', +/// info: MoqTrackInfo(compress: true), +/// ), /// config: moq_json.SnapshotConfig(), /// ); /// ``` @@ -26,7 +31,7 @@ typedef SnapshotProducer = MoqJsonSnapshotProducer; /// the newest with `latest()`. typedef SnapshotConsumer = MoqJsonSnapshotConsumer; -/// Configures a lossy latest-value JSON track. +/// Configures a lossy latest-value JSON producer. typedef SnapshotConfig = MoqJsonSnapshotConfig; /// Publishes a lossless stream of JSON records on a track it takes over. @@ -35,9 +40,6 @@ typedef StreamProducer = MoqJsonStreamProducer; /// Consumes a lossless stream of JSON records from a track it takes over. typedef StreamConsumer = MoqJsonStreamConsumer; -/// Configures a lossless JSON stream track. -typedef StreamConfig = MoqJsonStreamConfig; - /// A JSON string read from a track, with the timestamp of the frame that /// carried it in microseconds, or null when untimed. Rounded down to a whole /// microsecond, so a tick on a track like 90 kHz may not survive a read and @@ -52,7 +54,7 @@ typedef WindowProducer = MoqJsonWindowProducer; /// over. Group rolls never surface. typedef WindowConsumer = MoqJsonWindowConsumer; -/// Configures a JSON window track. +/// Configures a JSON window producer. typedef WindowConfig = MoqJsonWindowConfig; /// One change to a window: [WindowEventPush], [WindowEventPop], or diff --git a/dart/moq/test/moq_test.dart b/dart/moq/test/moq_test.dart index f9b085bcbf1..8aceeb239d5 100644 --- a/dart/moq/test/moq_test.dart +++ b/dart/moq/test/moq_test.dart @@ -4,7 +4,6 @@ import 'dart:typed_data'; import 'package:moq/audio.dart' as moq_audio; import 'package:moq/media.dart' as moq_media; -import 'package:moq/flate.dart' as moq_flate; import 'package:moq/json.dart' as moq_json; import 'package:moq/moq.dart'; import 'package:moq/video.dart' as moq_video; @@ -56,39 +55,6 @@ void main() { expect(() => catalog.now(), throwsA(isA())); }); - for (final timestamp in [null, 123456]) { - for (final mode in ['snapshot', 'stream']) { - test('flate $mode preserves timestamp $timestamp', () async { - final broadcast = BroadcastProducer(); - final info = timestamp == null ? null : TrackInfo(timescale: 1000000); - final track = broadcast.publishTrack(name: 'data', info: info); - final consumer = track.consume(subscription: null); - final payload = utf8.encode('payload'); - if (mode == 'snapshot') { - final producer = moq_flate.SnapshotProducer( - broadcast: broadcast, - track: track, - config: moq_flate.Config(), - ); - producer.update(payload: payload, timestampUs: timestamp); - producer.finish(); - } else { - final producer = moq_flate.StreamProducer( - broadcast: broadcast, - track: track, - config: moq_flate.Config(), - ); - producer.append(payload: payload, timestampUs: timestamp); - producer.finish(); - } - final frame = await consumer.readFrame().timeout(timeout); - expect(frame?.payload, payload); - expect(frame?.timestampUs, timestamp); - broadcast.close(); - }); - } - } - test('raw writes without timestamps stay untimed', () async { final broadcast = BroadcastProducer(); final track = broadcast.publishTrack(name: 'untimed', info: null); @@ -514,7 +480,7 @@ void main() { broadcast: broadcast, track: broadcast.publishTrack( name: 'status', - info: MoqTrackInfo(timescale: 1000000), + info: MoqTrackInfo(timescale: 1000000, compress: true), ), config: moq_json.SnapshotConfig(), ); @@ -522,10 +488,7 @@ void main() { .consume() .subscribeTrack(name: 'status', subscription: null) .timeout(timeout); - final consumer = moq_json.SnapshotConsumer( - track: track, - config: moq_json.SnapshotConfig(), - ); + final consumer = moq_json.SnapshotConsumer(track: track); for (var n = 1; n <= 4; n++) { producer.update(value: jsonEncode({'n': n}), timestampUs: n * 1000); @@ -551,18 +514,15 @@ void main() { final producer = moq_json.WindowProducer( track: broadcast.publishTrack( name: 'log', - info: MoqTrackInfo(timescale: 1000000), + info: MoqTrackInfo(timescale: 1000000, compress: true), ), - config: moq_json.WindowConfig(compression: true), + config: moq_json.WindowConfig(), ); final track = await broadcast .consume() .subscribeTrack(name: 'log', subscription: null) .timeout(timeout); - final consumer = moq_json.WindowConsumer( - track: track, - config: moq_json.WindowConfig(compression: true), - ); + final consumer = moq_json.WindowConsumer(track: track); for (var n = 0; n < 3; n++) { producer.push(value: jsonEncode({'n': n}), timestampUs: n * 1000); diff --git a/demo/carla/sim/moq_carla_sim/publish.py b/demo/carla/sim/moq_carla_sim/publish.py index b5607b62fb7..dc9ced6527b 100644 --- a/demo/carla/sim/moq_carla_sim/publish.py +++ b/demo/carla/sim/moq_carla_sim/publish.py @@ -177,14 +177,15 @@ def status_payload(status: Status) -> dict[str, Any]: class StatusPublisher: - """The `status` track: DEFLATE-compressed JSON snapshots. + """The `status` track: JSON snapshots on a compressed track. Snapshot rather than stream, because a late joiner wants the truck's current state, not the history of every command it ever accepted. """ def __init__(self, broadcast: moq.BroadcastProducer) -> None: - self._producer = moq.json.SnapshotProducer(broadcast, broadcast.publish_track("status"), compression=True) + track = broadcast.publish_track("status", moq.TrackInfo(compress=True)) + self._producer = moq.json.SnapshotProducer(broadcast, track) # Weak: watching demand neither keeps the track open nor locks it. self._demand = self._producer.demand() @@ -359,7 +360,7 @@ async def _pump_driver( log.info("driver %s connected (epoch %s)", driver_id, epoch) try: broadcast = await client.request_broadcast(path) - control = moq.json.SnapshotConsumer(await broadcast.subscribe_track("control"), compression=True) + control = moq.json.SnapshotConsumer(await broadcast.subscribe_track("control")) async for message in control: command = parse_command(driver_id, message, epoch) if command is not None: diff --git a/demo/carla/sim/tests/conftest.py b/demo/carla/sim/tests/conftest.py index 2ba51aa858c..77228ed632a 100644 --- a/demo/carla/sim/tests/conftest.py +++ b/demo/carla/sim/tests/conftest.py @@ -37,9 +37,8 @@ class AnnounceEventRestart(AnnounceEvent): class SnapshotProducer: """Records every published value; demand comes from the wrapped track.""" - def __init__(self, broadcast: object, track: SimpleNamespace, *, compression: bool = False) -> None: + def __init__(self, broadcast: object, track: SimpleNamespace) -> None: self.track = track - self.compression = compression self.values: list[object] = [] def demand(self) -> object: @@ -52,8 +51,7 @@ def update(self, value: object) -> None: class SnapshotConsumer: """Yields whatever the wrapped fake track yields, already parsed.""" - def __init__(self, track: object, *, compression: bool = False) -> None: - assert compression, "control is DEFLATE-compressed" + def __init__(self, track: object) -> None: self._track = track def __aiter__(self): @@ -68,6 +66,7 @@ def __aiter__(self): moq.AnnounceEventRestart = AnnounceEventRestart moq.AnnounceRoute = lambda **kwargs: SimpleNamespace(**kwargs) moq.mint_epoch = lambda: str(uuid.uuid4()) +moq.TrackInfo = lambda **kwargs: SimpleNamespace(**kwargs) moq_json = ModuleType("moq.json") moq_json.SnapshotProducer = SnapshotProducer diff --git a/demo/carla/sim/tests/test_runtime.py b/demo/carla/sim/tests/test_runtime.py index 298329becc3..ca0597d3f1a 100644 --- a/demo/carla/sim/tests/test_runtime.py +++ b/demo/carla/sim/tests/test_runtime.py @@ -371,7 +371,7 @@ def test_session_announces_with_the_run_epoch_after_populating_tracks() -> None: carries the run's epoch so a restarted worker reads as a restart.""" calls: list[object] = [] broadcast = SimpleNamespace( - publish_track=lambda name: calls.append(name) or SimpleNamespace(demand=SimpleNamespace), + publish_track=lambda name, info=None: calls.append(name) or SimpleNamespace(demand=SimpleNamespace), announce=lambda route: calls.append(route), ) route = SimpleNamespace(epoch="0190b2a4-0000-7000-8000-000000000000") diff --git a/demo/carla/web/src/preview.ts b/demo/carla/web/src/preview.ts index 4fec2ed2f6d..c342a52a85e 100644 --- a/demo/carla/web/src/preview.ts +++ b/demo/carla/web/src/preview.ts @@ -14,8 +14,7 @@ export function populatePreview(add: (path: Moq.Path.Valid) => Truck) { const truck = add(Moq.Path.from(`carla/truck${index + 1}/front.hang`)); const origin = new Moq.Broadcast.Producer(); const producer = new Json.Snapshot.Producer({ - track: origin.createTrack("status"), - compression: "deflate", + track: origin.createTrack("status", { compress: true }), }); (truck.broadcast.out.active as Moq.Signals.Signal).set(origin.consume()); producer.update({ diff --git a/demo/carla/web/src/truck.test.ts b/demo/carla/web/src/truck.test.ts index 9800aef2217..358d03867d3 100644 --- a/demo/carla/web/src/truck.test.ts +++ b/demo/carla/web/src/truck.test.ts @@ -61,8 +61,8 @@ describe("Truck status", () => { function statusOrigin() { const origin = new Moq.Broadcast.Producer(); - const track = origin.createTrack("status"); - const producer = new Json.Snapshot.Producer({ track, compression: "deflate" }); + const track = origin.createTrack("status", { compress: true }); + const producer = new Json.Snapshot.Producer({ track }); return { origin, producer }; } @@ -329,11 +329,9 @@ describe("Truck safety controls", () => { }); const front = new Json.Snapshot.Consumer>({ track: safety.broadcast.track("control").subscribe(), - compression: "deflate", }); const rear = new Json.Snapshot.Consumer>({ track: safety.broadcast.track("control").subscribe(), - compression: "deflate", }); try { diff --git a/demo/carla/web/src/truck.ts b/demo/carla/web/src/truck.ts index d3fc13a4a82..a7572fc066f 100644 --- a/demo/carla/web/src/truck.ts +++ b/demo/carla/web/src/truck.ts @@ -45,8 +45,8 @@ export class SafetyCommandBroadcast { constructor(command: Command, broadcast = new Moq.Broadcast.Producer()) { this.broadcast = broadcast; - const track = this.broadcast.createTrack("control"); - this.#producer = new Json.Snapshot.Producer>({ track, compression: "deflate" }); + const track = this.broadcast.createTrack("control", { compress: true }); + this.#producer = new Json.Snapshot.Producer>({ track }); this.#producer.update({ value: { ...command.cmd, sequence: command.sequence, timestamps: command.timestamps }, }); @@ -371,7 +371,6 @@ export class Truck { const consumer = new Json.Snapshot.Consumer({ track: statusTrack, schema: TruckStatusSchema, - compression: "deflate", }); effect.spawn(async () => { @@ -425,8 +424,8 @@ export class Truck { this.#sentAt.clear(); }); - const track = driverBroadcast.createTrack("control"); - const producer = new Json.Snapshot.Producer>({ track, compression: "deflate" }); + const track = driverBroadcast.createTrack("control", { compress: true }); + const producer = new Json.Snapshot.Producer>({ track }); effect.cleanup(() => producer.finish()); effect.run(this.#runControlTrack.bind(this, track, producer)); // A fresh epoch per lease: the truck binds its single driver to this path and epoch. diff --git a/doc/.vitepress/api/package.json b/doc/.vitepress/api/package.json index 0997140a5e5..fdeaf455106 100644 --- a/doc/.vitepress/api/package.json +++ b/doc/.vitepress/api/package.json @@ -8,7 +8,6 @@ "@moq/auth": "workspace:*", "@moq/boy": "workspace:*", "@moq/clock": "workspace:*", - "@moq/flate": "workspace:*", "@moq/hang": "workspace:*", "@moq/json": "workspace:*", "@moq/libav": "workspace:*", diff --git a/doc/.vitepress/config.ts b/doc/.vitepress/config.ts index 54eb955d650..dc6a49fba92 100644 --- a/doc/.vitepress/config.ts +++ b/doc/.vitepress/config.ts @@ -206,7 +206,6 @@ export default defineConfig({ { text: "moq-auth", link: "/lib/rs/moq-auth" }, { text: "moq-room", link: "/lib/rs/moq-room" }, { text: "moq-json", link: "/lib/rs/moq-json" }, - { text: "moq-flate", link: "/lib/rs/moq-flate" }, { text: "moq-clock", link: "/lib/rs/moq-clock" }, ], }, @@ -224,7 +223,6 @@ export default defineConfig({ { text: "@moq/signals", link: "/lib/js/signals" }, { text: "@moq/clock", link: "/lib/js/clock" }, { text: "@moq/json", link: "/lib/js/json" }, - { text: "@moq/flate", link: "/lib/js/flate" }, { text: "API reference", link: "/lib/js/api/" }, ], }, diff --git a/doc/bin/inspect.md b/doc/bin/inspect.md index db8ecf3a57a..d75262b5075 100644 --- a/doc/bin/inspect.md +++ b/doc/bin/inspect.md @@ -74,11 +74,10 @@ curl http://localhost:4443/fetch/.stats/node/local/host/totals.json | jq ``` The node name (`local/host` here) is `stats.node`. Each fetch returns the -newest snapshot of cumulative counters. `totals.json` sums each tier's traffic -and sessions. `group.json` (`/group.json` at `stats.depth` 1 or more) -carries a group's traffic per broadcast path and sessions per stats label: its -newest group is a full snapshot followed by merge-patch deltas, one per line. +newest group of cumulative counters: a full snapshot followed by merge-patch +deltas, one per line. `totals.json` sums each tier's traffic and sessions. +`group.json` (`/group.json` at `stats.depth` 1 or more) carries a +group's traffic per broadcast path and sessions per stats label. A subscriber may also ask for `/prefix.json`, one broadcast prefix rolled up with each child, produced only while held. Fetch twice and divide by the -interval for a rate. The `.json.z` twins carry compressed patches, which these -tools print as raw bytes. [Stats](/concept/stats) describes every field. +interval for a rate. [Stats](/concept/stats) describes every field. diff --git a/doc/concept/hang.md b/doc/concept/hang.md index 41802ac4890..84c8f431541 100644 --- a/doc/concept/hang.md +++ b/doc/concept/hang.md @@ -19,8 +19,9 @@ payloads follow [moq-e2ee](/draft/moq-e2ee). See [Encryption](#encryption). ## Catalog `catalog.json` is a JSON track listing the renditions of each media kind and -the decoder config for each. It updates live as tracks come and go, and a -compressed twin (`catalog.json.z`) is published alongside it. +the decoder config for each. It updates live as tracks come and go, and is a +[compressed track](/concept/moq-lite#subscriptions), so a reader sees plain JSON +whether or not its session carries the compression. ```json { @@ -113,7 +114,7 @@ as plain tracks, split by whether a generic consumer can parse the payload. { "json": { "tracks": { - "chat": { "mode": "stream", "compression": "deflate" }, + "chat": { "mode": "stream", "schema": "https://example.com/chat.schema.json" }, "status": { "mode": "snapshot" } } }, @@ -137,17 +138,16 @@ A sliding window, a bounded log a reader can join in the middle, is a framing in [`moq-json`](/lib/rs/moq-json) rather than a catalog mode. Both ends opt into it on a raw track. -The rest is descriptive: `compression` (`deflate`, the same group-scoped -`deflate-raw` the catalog uses), `schema` on a JSON track, `mime` on a binary -one, `bitrate`, `jitter`, and `delay` with the same meaning as for media, plus +The rest is descriptive: `schema` on a JSON track, `mime` on a binary one, `bitrate`, `jitter`, and `delay` with the same meaning as for media, plus the optional `broadcast` reference and `track` name. A publisher measures `delay` only from payloads stamped with their capture time on the broadcast clock. A -consumer that doesn't recognize a `mode` or `compression` ignores that track and -round-trips it verbatim. +consumer that doesn't recognize a `mode` ignores that track and round-trips it +verbatim. Compression is not a catalog field: a JSON track is a compressed track, +and moq-net hands every reader plain frames. The catalog entry lives as long as its producer: dropping it retracts the -track. [`moq-json`](/lib/rs/moq-json) and [`moq-flate`](/lib/rs/moq-flate) (and -their TypeScript twins) are what read and write the payloads. +track. [`moq-json`](/lib/rs/moq-json) (and its TypeScript twin) reads and writes +JSON payloads; a binary track's payloads are the frames themselves. An application with its own per-track fields can list a data track in its own root section instead, nesting the JSON or binary config in a `config` field diff --git a/doc/concept/stats.md b/doc/concept/stats.md index a7954b0bfd8..74ec948cd1f 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -1,6 +1,6 @@ --- title: Stats -description: The stats broadcasts a relay publishes, their tracks, and both JSON encodings +description: The stats broadcasts a relay publishes, their tracks, and their JSON encoding --- # Stats @@ -199,8 +199,7 @@ but a group's comes in two encodings: | `[