Repository navigation
Description
Problem
Groups are sent newest-first, so an older, finished group can be accepted after the live group has started decoding. The container consumer hands its frames to the video decode loop, which feeds them to the codec between the live group's deltas: the codec receives the older keyframe and deltas in the middle of the live reference chain. The expected effect is corrupted live deltas or a codec error; that effect was not run on a real codec (see Verified).
This needs the older group's oldest queued frame to be within maxAge of the live edge, which happens with a GOP shorter than the age budget: short GOPs, a large delay, or buffered mode where maxAge = delay + buffer. An ordinary 2 s GOP at a 100 ms budget is dropped on its first frame.
Where
At 2b689c242:
js/hang/src/container/consumer.ts:495-528:next()servesgroups[0]wheneverseq <= #active, and reports itcontinuousbecause nothing was dropped (#continuesDelivery,:299-302).consumer.ts:304-335(#checkMaxAge): shifts the older group on its first frame unless the span from its oldest queued frame to the live edge is withinmaxAge, which is the condition above.js/watch/src/video/decoder.ts:432-472(legacy loop) and:509-547(CMAF loop): feed whatever arrives.- The consumer's own comment (
consumer.ts:141-145) says newest-first delivery makes this ordering normal.
Root cause
The decode loops treat every delivered frame as the next frame of the stream they are decoding. continuous answers "was anything dropped before this frame", not "is this frame newer than what the codec already has", and a group that arrives below the one being decoded is continuous by that definition. The consumer keeps such groups on purpose, because audio places them in a timestamp-indexed ring, but nothing on the video side filters them before the codec.
Reproduce
A new test file, js/watch/src/video/decoder.order.test.ts. A fake VideoDecoder records the type and timestamp of every chunk, a real Moq.Broadcast.Producer and Track.Producer serve the track, the frames are legacy, and no scaled timers are used.
- "an older group served after the live group's keyframe never reaches the codec" (delay 500 ms): group 10 sends a key at 10.000 s and a delta at 10.033 s and both are fed; finished group 9 is then accepted with frames at 9.900 s and 9.933 s; group 10 sends another delta at 10.066 s. It asserts the codec received only
[10000000, 10033000, 10066000]. Red. - "a whole older GOP accepted behind the live GOP never reaches the codec" (delay 100 ms, the older group is a full 2 s GOP of 60 frames from 8.0 s). It asserts
{ older: 0, fed: 3 }. This passes today, because the span rule drops the group on its first frame; it is a guard for the ordinary case.
Failing test
diff --git a/js/watch/src/video/decoder.order.test.ts b/js/watch/src/video/decoder.order.test.ts
new file mode 100644
index 000000000..dafca1920
--- /dev/null
+++ b/js/watch/src/video/decoder.order.test.ts
@@ -0,0 +1,182 @@
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import * as Catalog from "@moq/hang/catalog";
+import * as Moq from "@moq/net";
+import { Time } from "@moq/net";
+import { Signal } from "@moq/signals";
+import type { Broadcast } from "../broadcast";
+import { Sync } from "../sync";
+import { Decoder } from "./decoder";
+import type { Source } from "./source";
+
+// What reaches the codec when an older, finished GOP is accepted
+// after the live GOP has already started decoding. WebCodecs is not in bun, so a fake codec records
+// every chunk it is handed.
+
+type Chunk = { type: string; timestamp: number };
+let decoded: Chunk[][] = [];
+
+const real = { VideoDecoder: globalThis.VideoDecoder, EncodedVideoChunk: globalThis.EncodedVideoChunk };
+
+beforeEach(() => {
+ decoded = [];
+ class FakeVideoDecoder {
+ state = "unconfigured";
+ readonly #chunks: Chunk[] = [];
+ constructor(_init: unknown) {
+ decoded.push(this.#chunks);
+ }
+ configure(): void {
+ this.state = "configured";
+ }
+ decode(chunk: Chunk): void {
+ this.#chunks.push({ type: chunk.type, timestamp: chunk.timestamp });
+ }
+ close(): void {
+ this.state = "closed";
+ }
+ static isConfigSupported(): Promise<{ supported: boolean }> {
+ return Promise.resolve({ supported: true });
+ }
+ }
+ class FakeEncodedVideoChunk {
+ readonly type: string;
+ readonly timestamp: number;
+ readonly byteLength: number;
+ constructor(init: { type: string; timestamp: number; data: Uint8Array }) {
+ this.type = init.type;
+ this.timestamp = init.timestamp;
+ this.byteLength = init.data.byteLength;
+ }
+ }
+ globalThis.VideoDecoder = FakeVideoDecoder as unknown as typeof VideoDecoder;
+ globalThis.EncodedVideoChunk = FakeEncodedVideoChunk as unknown as typeof EncodedVideoChunk;
+});
+
+afterEach(() => {
+ globalThis.VideoDecoder = real.VideoDecoder;
+ globalThis.EncodedVideoChunk = real.EncodedVideoChunk;
+});
+
+const flush = () => new Promise((resolve) => setTimeout(resolve, 0));
+async function settle(rounds = 10): Promise<void> {
+ for (let i = 0; i < rounds; i++) await flush();
+}
+
+const TRACK = "video";
+
+/** A legacy frame: varint timestamp then one payload byte. Wire timestamps are "now". */
+function write(group: Moq.Group.Producer, timestamp: number): void {
+ const header = Moq.Varint.encode(timestamp);
+ const body = new Uint8Array(header.length + 1);
+ body.set(header);
+ body[header.length] = 1;
+ group.writeFrame({ payload: body, timestamp: Time.Timestamp.now() });
+}
+
+function fixture(delay = Time.Milli(500)) {
+ const broadcast = new Moq.Broadcast.Producer();
+ const consumer = broadcast.consume();
+ const config = Catalog.VideoConfigSchema.parse({ codec: "avc1.640028", container: { kind: "legacy" } });
+ const catalog = new Signal<Catalog.Root | undefined>({
+ video: { renditions: { [TRACK]: config } },
+ } as Catalog.Root);
+ const source = {
+ in: {
+ broadcast: new Signal({ relativeBroadcast: () => consumer, out: { catalog } } as unknown as Broadcast),
+ },
+ out: {
+ track: new Signal<string | undefined>(TRACK),
+ config: new Signal<Catalog.VideoConfig | undefined>(config),
+ catalog: new Signal<Catalog.VideoConfig | undefined>(undefined),
+ },
+ } as unknown as Source;
+ // A generous fixed delay, so neither the wire nor the local age budget drops the older GOP:
+ // what is under test is what the decode loop does with it.
+ const sync = new Sync({ delay });
+ const decoder = new Decoder({ source, sync });
+ const track = new Moq.Track.Producer(TRACK).accept({});
+ broadcast.insertTrack(track);
+ return {
+ decoder,
+ sync,
+ track,
+ close(): void {
+ decoder.close();
+ sync.close();
+ broadcast.close();
+ },
+ };
+}
+
+test("an older group served after the live group's keyframe never reaches the codec", async () => {
+ const warn = console.warn;
+ console.warn = () => {};
+ const fx = fixture();
+ try {
+ for (let i = 0; i < 400 && decoded.length < 1; i++) await flush();
+ expect(decoded).toHaveLength(1);
+
+ // The live GOP (group 10) arrives first and the loop hands its keyframe and a delta to the codec.
+ const live = new Moq.Group.Producer(10);
+ fx.track.writeGroup(live);
+ write(live, 10_000_000);
+ write(live, 10_033_000);
+ await settle();
+ expect(decoded[0].map((c) => c.timestamp)).toEqual([10_000_000, 10_033_000]);
+
+ // The previous, finished GOP (group 9) is accepted behind it.
+ const older = new Moq.Group.Producer(9);
+ fx.track.writeGroup(older);
+ write(older, 9_900_000);
+ write(older, 9_933_000);
+ older.close();
+ await settle();
+
+ // The live GOP continues.
+ write(live, 10_066_000);
+ await settle();
+
+ // Only group-10 chunks reach the codec: group 9's keyframe would reset the reference chain
+ // the live deltas decode against.
+ expect(decoded[0].map((c) => c.timestamp)).toEqual([10_000_000, 10_033_000, 10_066_000]);
+ live.close();
+ } finally {
+ fx.close();
+ console.warn = warn;
+ }
+});
+
+// The realistic shape: the older group is a whole 2s GOP and the budget is an ordinary 100ms.
+// The span rule drops it on its first frame, so this passes today; it guards the ordinary case.
+test("a whole older GOP accepted behind the live GOP never reaches the codec", async () => {
+ const warn = console.warn;
+ console.warn = () => {};
+ const fx = fixture(Time.Milli(100));
+ try {
+ for (let i = 0; i < 400 && decoded.length < 1; i++) await flush();
+ expect(decoded).toHaveLength(1);
+
+ const live = new Moq.Group.Producer(10);
+ fx.track.writeGroup(live);
+ write(live, 10_000_000);
+ write(live, 10_033_333);
+ await settle();
+ expect(decoded[0].map((c) => c.timestamp)).toEqual([10_000_000, 10_033_333]);
+
+ const older = new Moq.Group.Producer(9);
+ fx.track.writeGroup(older);
+ for (let i = 0; i < 60; i++) write(older, 8_000_000 + i * 33_333);
+ older.close();
+ await settle();
+
+ write(live, 10_066_666);
+ await settle();
+
+ const fed = decoded[0].map((c) => c.timestamp);
+ expect({ older: fed.filter((ts) => ts < 10_000_000).length, fed: fed.length }).toEqual({ older: 0, fed: 3 });
+ live.close();
+ } finally {
+ fx.close();
+ console.warn = warn;
+ }
+});Command (from js/watch):
bun test src/video/decoder.order.test.ts
Red output at 2b689c242 (3 of 3 runs):
error: expect(received).toEqual(expected)
@@ -3,3 +3,4 @@
10033000,
+ 9900000,
+ 9933000,
10066000,
...
(fail) an older group served after the live group's keyframe never reaches the codec [46.12ms]
1 pass
1 fail
Suggested fix
diff --git a/js/watch/src/video/decoder.ts b/js/watch/src/video/decoder.ts
index 592eb898d..e6e5fd152 100644
--- a/js/watch/src/video/decoder.ts
+++ b/js/watch/src/video/decoder.ts
@@ -428,6 +428,8 @@ class DecoderTrack {
});
let previous: Time.Micro | undefined;
+ // The newest group fed to the codec; see the guard below.
+ let fed: number | undefined;
effect.spawn(async () => {
for (;;) {
@@ -435,11 +437,21 @@ class DecoderTrack {
if (!next) break;
// Publisher rewound: flush queued/in-flight video and re-anchor before decoding.
- if (this.#onDiscontinuity(next.discontinuity)) previous = undefined;
+ if (this.#onDiscontinuity(next.discontinuity)) {
+ previous = undefined;
+ fed = undefined;
+ }
const { frame } = next;
if (!frame) continue; // The group is done
+ // A group older than one the codec is already decoding was accepted behind it (groups
+ // are sent newest-first). Its keyframe would reset the reference chain the newer
+ // group's deltas decode against, and its pictures are late anyway: the renderer would
+ // drop them. Audio keeps such groups for its timestamp-indexed ring; video does not.
+ if (fed !== undefined && next.group < fed) continue;
+ fed = next.group;
+
// Mark that we received this frame right now.
const timestamp = Time.Milli.fromMicro(frame.timestamp as Time.Micro);
this.sync.received(timestamp, "video");
@@ -505,6 +517,8 @@ class DecoderTrack {
});
let previous: Time.Micro | undefined;
+ // The newest group fed to the codec; see the guard below.
+ let fed: number | undefined;
effect.spawn(async () => {
for (;;) {
@@ -512,11 +526,21 @@ class DecoderTrack {
if (!next) break;
// Publisher rewound: flush queued/in-flight video and re-anchor before decoding.
- if (this.#onDiscontinuity(next.discontinuity)) previous = undefined;
+ if (this.#onDiscontinuity(next.discontinuity)) {
+ previous = undefined;
+ fed = undefined;
+ }
const { frame } = next;
if (!frame) continue;
+ // A group older than one the codec is already decoding was accepted behind it (groups
+ // are sent newest-first). Its keyframe would reset the reference chain the newer
+ // group's deltas decode against, and its pictures are late anyway: the renderer would
+ // drop them. Audio keeps such groups for its timestamp-indexed ring; video does not.
+ if (fed !== undefined && next.group < fed) continue;
+ fed = next.group;
+
// Mark that we received this frame right now.
const timestamp = Time.Milli.fromMicro(frame.timestamp);
this.sync.received(timestamp, "video");In js/watch/src/video/decoder.ts only, in both decode loops (legacy and CMAF): fed tracks the newest group handed to the codec, a frame whose next.group < fed is skipped, and fed resets on a discontinuity (#onDiscontinuity). The consumer is unchanged, so audio still receives such groups for its timestamp-indexed ring. The skipped pictures are late anyway: the renderer would drop them. A consumer-level alternative (report continuous: false for a group below the delivered one) was not tested; it would also change what audio sees on every tune-in, where older groups routinely arrive after the live one.
Verified
- Red at
2b689c242with the test: test 1 fails 3 of 3 runs (the codec received[10000000, 10033000, 9900000, 9933000, 10066000]); test 2 passes, as expected from the span rule. - Green with the fix: both tests pass, 3 of 3 runs.
- js/watch with test and fix:
bun test235 pass, 0 fail across 24 files;tsc --noEmitexit 0;biome checkclean on the two files. No audio code changed, so no audio replay was run. - Not verified: no real codec was run. That the older IDR corrupts the following live deltas, or raises a codec error, comes from the codec's reference model, not from a run. The relay order was not reproduced on a live relay: the test builds the "older group accepted after the live group's first frames" ordering by hand, on the consumer comment's premise that newest-first delivery makes it normal.
Related
- This is not the defect in
quest/m1/3056-watch-video-decoder-captures-the-rewind-generation-at.mdorquest/m1/play-decode-schedule.md. Both are about output that outlives a declared discontinuity: 3056 hasjs/watchkeep decoding chunks queued before a publisher rewind (nodecoder.reset()in#onDiscontinuity, and a generation read at output time), and play-decode-schedule is the Rustmoq playpath (rs/moq-cli/src/play/video.rs: stale pictures after a rewind, a fixed 100 ms decode lead, a batch trimmed to three pictures). Here nothing rewinds and nothing is declared: the older group arrives with the same discontinuity count andcontinuous: true, so neither a codec reset on discontinuity nor a generation tag would fire. The harm is on the input side (an older keyframe and deltas enter the codec between the live group's deltas), and the fix is an input guard in the decode loops, which neither quest proposes. - Quest 3056's line numbers have moved. At
2b689c242:#onDiscontinuityisjs/watch/src/video/decoder.ts:557-564(the quest says:528-535), the generation read is:340(:312), the generation compare is:362(:335), the wait race is:360-361(:332-334), and the audio decoder's reset on a new epoch isjs/watch/src/audio/decoder.ts:387-390(:374-376). - fix(net): judge spliced staleness against the logical live edge #4103, fix(net): hold a parked track's warm cache until the upstream confirms it #4104 and fix(net): skip a stale warm cache on an IETF rejoin #4150 are relay-side; they do not change what the decode loop feeds the codec.
(Written by Claude Opus 5.5)
Activity