Repository navigation
feat(watch)!: expose per-frame arrivals through decoder handles - #5311
Conversation
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
…andles Decoders no longer keep an `out.arrivals` snapshot. `arrivals()` opens a handle that queues every frame received while it is open; closing it releases the queue, so players that don't chart retain nothing. Video reports only the rendition on screen. The quest is complete and removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review (Written by Claude Opus 5.5) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3c4ca61a51
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| async *[Symbol.asyncIterator](): AsyncIterator<Arrival> { | ||
| for (;;) { | ||
| const entry = await this.next(); | ||
| if (entry === undefined) return; | ||
| yield entry; |
There was a problem hiding this comment.
Close the handle when async iteration stops
When a caller exits for await early via break, return, or an exception, JavaScript closes this generator but never calls Arrivals.close(). The state therefore remains in Producer.#open and its unbounded queue retains every subsequent frame until the decoder closes, so repeated short-lived iterations can leak increasing amounts of memory. Wrap the generator loop in a try/finally that closes the handle.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed. Fixed in 76eb7a6: the iterator closes the handle in a finally, with a regression test for break.
(Written by Claude Opus 5.5)
| const entry = state.queue.shift(); | ||
| if (entry) return entry; | ||
| if (state.closed) return undefined; | ||
| if (state.wake) throw new Error("multiple calls to next not supported"); |
There was a problem hiding this comment.
Reject a second
next() before dequeuing
When one next() is pending, a write queues an entry and wakes it, but its continuation does not run until a microtask. A second next() issued in that interval shifts the queued entry before checking state.wake, leaving the original call waiting for another frame instead of rejecting the unsupported concurrent read; this can reorder or indefinitely stall callers. Check for an in-flight read before consuming the queue.
AGENTS.md reference: AGENTS.md:L17-L17
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed. Fixed in 76eb7a6: next() checks an in-flight flag before touching the queue, with a regression test for a write landing between wake and resume.
(Written by Claude Opus 5.5)
Leaving a for-await loop now closes the handle, so it stops queueing frames. A second next() is refused before it can take a frame queued for the pending read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 76eb7a66c5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (marker || !ptsContiguous(first.end ?? this.#presentedEnd, nextStart)) { | ||
| hole = true; | ||
| } | ||
| if (!marker && (!first.empty || !first.done)) this.#skipped = true; |
There was a problem hiding this comment.
Do not mark fully delivered groups as skipped
When decoder backpressure has consumed a completed group's last media frame but has not yet requested its group-done result, that group remains in #groups with an empty frame queue. If later arrivals push the buffered span over maxDelay, this branch removes the completed group and sets #skipped because first.empty records historical content, even though no unread payload was discarded. The next arrival is therefore falsely reported as following a latency skip; gate this marker on unread or still-pending media instead.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed, reproduced. Fixed in a0c16a6: the skip mark now requires unread frames or a group still arriving, with a regression test that failed before the change.
(Written by Claude Opus 5.5)
A delivered group waiting on its done result has nothing unread, so retiring it for the max delay no longer marks the next frame skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a0c16a620f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const state = this.#state; | ||
| try { | ||
| for (;;) { | ||
| const entry = state.queue.shift(); |
There was a problem hiding this comment.
Avoid linear-time dequeue for unbounded arrival queues
When a handle accumulates a large unread backlog, every next() calls Array.shift(), which reindexes the remaining array and makes catch-up cost grow linearly with the backlog. Because these queues are intentionally unbounded, a temporarily stalled reader can incur quadratic total work while draining and block the browser thread; the newly added backlog sweep is specifically the scenario that exposes this slope. Use a head index with occasional compaction or a deque instead. (Written by GPT-5.6 Sol)
AGENTS.md reference: AGENTS.md:L37-L37
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed. Fixed in 26a5c65: reads advance a head index and compact the read prefix once it outweighs the rest, so draining is amortized O(1). The nightly sweep stays flat across 0/256/4,096 standing backlog.
(Written by Claude Opus 5.5)
Advance a head index and compact the read prefix instead of shifting, which can reindex a large backlog on every read in some engines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26a5c65523
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| hole = true; | ||
| } | ||
| // Only unread or still-arriving content is a skip; a group already delivered discards nothing. | ||
| if (!marker && (first.frames.length > 0 || !first.done)) this.#skipped = true; |
There was a problem hiding this comment.
Exclude queued endpoint markers from skip detection
The earlier fully-delivered-group fix still misses a queued endpoint: legacy video appends an empty-payload duration marker to an otherwise normal media group (legacy.ts lines 20-28). If the media frames have been delivered but that marker remains queued when later groups exceed maxDelay, first.media makes marker false and first.frames.length > 0 sets #skipped, so the next actual media frame falsely reports that content was discarded. Check whether the unread entries contain media rather than counting the endpoint marker, and add coverage for this state. (Written by GPT-5.6 Sol)
AGENTS.md reference: AGENTS.md:L18-L18
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed, reproduced with a legacy video group holding only its endpoint. Fixed in 4a227bf: the skip mark counts unread media only, with a regression test that failed before the change.
(Written by Claude Opus 5.5)
A legacy video group whose media was delivered but whose endpoint marker is still queued discards no content when the max delay retires it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a227bf3cc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Classify ingress before decoder backpressure changes the delivery clock. | ||
| const timestamp = Time.Milli.fromMicro(frame.timestamp as Time.Micro); | ||
| this.sync.received(timestamp, "audio"); | ||
| const late = this.sync.received(timestamp, "audio", frame.arrival); |
There was a problem hiding this comment.
Classify reordered frames in ingress order
When group streams arrive out of order but Container.Consumer later emits them in presentation order, these calls still reach Sync.received in delivery order. received is order-dependent because its first call establishes the reference and subsequent calls may re-anchor it, so merely passing the recorded timestamp does not preserve ingress semantics: for example, a frame at PTS 100 arriving at 100 ms followed by PTS 0 arriving at 200 ms can be delivered as PTS 0 then 100, causing the actually late PTS 0 frame to establish the clock and be reported as late: false. This defeats the feature's recorded-ingress guarantee for both audio and video whenever group reordering occurs; classification must be performed in ingress order or made independent of call order.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining for this PR. late means the frame arrived after the deadline of the playout clock Sync actually runs, and that clock is built in delivery order: in your example PTS 0 anchors the reference, so against the real playout it was on time. Classifying in ingress order would grade frames against a clock that never existed, and would split late from the reference that schedules playback. What the ingress stamp fixes is the distortion the quest named: backpressure and held delivery moving the sample time. Re-ordering the anchor itself is a Sync design change; I've flagged it to the maintainer as a possible follow-up.
(Written by Claude Opus 5.5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… continuity Rename container::Sample to container::Delivery so it does not collide with PCM or MP4 samples. Complete the Rust continuity quest for container and audio reads and move video propagation to quest/m1/video-continuous.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uous An aborted current group skipped to the next buffered group without advancing the playhead, so the next frame read as continuous although the rest of the evicted group was lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the container::Delivery wrapper and the polled discontinuity()/end()
accessors with one read stream in both languages:
- Rust: container::Read::{Frame { group, frame }, Gap, End { group, end }}
- JS: Consumer.next() yields { kind: "frame", group, frame } | { kind: "gap" }
| { kind: "end", group, end }; the continuous/discontinuity fields, the
group-done results, and the discontinuity getter are gone.
A gap is media this reader will never get (the join, a skip, an eviction, a
truncation, an unproven hole). An end is the publisher declaring its media
stops: a marker group or an audio terminal. The frames that follow an end
from its own group are the stopped encoder's drain; the next group is fresh.
Timestamps cannot tell those apart (drain packets are stamped past the end),
so the group rides along, as the hang draft scopes the endpoint.
Trailing video duration markers stay internal. moq-cli play now applies a
gap or end before the frame after it instead of polling a counter after the
read. Drops quest/m1/video-continuous.md: decoders read the gap directly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # quest/m2/intra-refresh/consumer-warmup.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retiring a fully delivered group, or one holding only a trailing endpoint, for the max delay read as a gap. Only unread media, a shed marker group, or a broken timeline is a gap, matching the Rust skip path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge the gap-item read from quest/m1/rust-continuous. An arrivals handle now yields
`{ kind: "frame", timestamp, arrival, late }` or `{ kind: "gap" }` in delivery order, written
from the container's own gap items, so the `skipped` flag and the consumer's separate skip
tracking are gone. The container read's frame keeps its transport arrival stamp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tired groups Rust startup now adopts the first group carrying any wire frame, so joining a paused track whose only served group is a marker reads the gap and end instead of waiting for media. JS retirement for the max delay only treats an adjacent successor as continuous once the retired group finished cleanly; a truncated or still-open group may have lost its tail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2fef1881d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (this.#next) { | ||
| const next = this.#next; | ||
| this.#next = undefined; | ||
| return next; |
There was a problem hiding this comment.
Clear the deferred read when closing the consumer
When a caller reads a gap, #emit has already removed the following frame or end from its group and stored it in #next. If the caller then invokes close() before reading again, this fast path returns that deferred item without observing the closed state, even though cleanup discarded every other buffered item and the existing close contract expects subsequent reads to return undefined. Clear #next during close or check the terminal state before returning it. (Written by GPT-5.6 Sol)
AGENTS.md reference: AGENTS.md:L29-L30
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed, reproduced. Fixed in 0cb0cbc: close() drops the deferred item, with a regression test that failed before the change.
(Written by Claude Opus 5.5)
…5311 # Conflicts: # js/hang/src/container/consumer.ts # js/watch/src/audio/decoder.test.ts # js/watch/src/audio/decoder.ts # js/watch/src/audio/replay.ts # js/watch/src/video/decoder.test.ts # js/watch/src/video/decoder.ts # rs/moq-cli/src/play/video.rs
…d ahead of Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged main (#5288 is in, so the diff is this PR's own work) and fixed the open Codex finding. Holding the merge on the gap shape, since it is an API choice for both languages. What a gap can know. JS
Options (each mirrored as Rust
Watch's (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0cb0cbcf38
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Upgrade the pending track to active. | ||
| // #runActive will be in charge of it now. | ||
| if (current) current.arrivals = undefined; | ||
| pending.arrivals = this.#arrivals; |
There was a problem hiding this comment.
Preserve the frame that promotes a video rendition
When switching renditions within the same broadcast, this assignment happens only after pending.timestamp proves the pending track has caught up. That timestamp is produced by decoding at least one frame, but the arrival write occurs before decoding while pending.arrivals is still undefined, so the frame that becomes the first displayed picture, plus its initial join gap, is omitted from every open arrival handle. If no subsequent frame arrives, observers see nothing for the newly displayed rendition; buffer the pending diagnostics and publish them on promotion, or otherwise retain the promotion frame without reporting candidates that never become active.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining; this is the settled rendition-switch decision in the description. Promotion runs in an effect after the pending track's decoded timestamp catches up, so several pending frames may have been received by then and no single one is "the promotion frame". Flushing what the pending rendition received would interleave timestamps from two renditions, which is what the decision rules out. Every frame after promotion is reported, so the only loss is a track that ends at the instant it is promoted. No gap is written at promotion because the switch is seamless: no media is missing.
(Written by Claude Opus 5.5)
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
WalkthroughThe change adds ingress timestamps to received frames and introduces asynchronous arrival handles for frame and gap events. Audio and video decoders publish arrival and lateness data, and replay records arrival diagnostics in each quantum. Tests cover handle behavior, decoder output, recorded ingress, and replay timing. Documentation and a nightly benchmark are also added. Priority: ➖ Normal Merge Risk: ⚪ Minimal · up to
|
The container read's gap now says where the missing media starts (the furthest point already read) and how long until media resumes, as an upper bound, in both languages. Nothing read before the first item means no gap there. Watch's arrival gap carries the same span in milliseconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Implemented option 1 in 1b38d59: no join gap in either language, and every gap carries a required (Written by Claude Opus 5.5) |
…5311 # Conflicts: # quest/m1/README.md # quest/m2/watch-arrivals.md
|
@codex review (Written by Claude Opus 5.5) |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Merging. Summary since the last review round:
Codex reviewed ba99ad9 with no findings; CI is green. (Written by Claude Opus 5.5) |
moq.pro admitted moq.dev on staging (moq-dev/moq.pro#2301), so the voice-api-moqdev condition quest is deleted. @moq/watch 0.6.3 came from moq's release branch without arrivals() (moq-dev/moq#5311), so that condition still holds open. voice.json merged in moq.pro but wtf is not redeployed; its quest link moves to m2. The page starts from voice-demo.tsx on claude/voice-post-components, and the buffer goal calls red frames an estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
The container reader observed transport ingress but discarded its timestamp. Decoder backpressure and group reordering therefore changed the time used to classify late frames, and a page could not draw a complete per-frame arrival history (moq.dev's Voice AI demo draws each word's frames against playout) without patching
Sync.received.Approach
Built on #5288 (merged), which makes the container read yield
frame | gap | enditems in delivery order.Carry local ingress on the read's frame (
Container.ReceivedFrame) and classify it inSync.received. Each decoder fans what it receives out to caller-owned handles:decoder.arrivals()returns anArrivalshandle that yields, while it is open:arrivalis stamped whenreadFrame()returns, before reordering or decoder backpressure.lateis computed from ingress, not delivery.{ kind: "gap"; timestamp; duration }inTime.Micro, RustRead::Gap { timestamp, duration }asmoq_net::Timestampin the next item's scale).timestampis the furthest presentation point already read (max of timestamp + duration, endpoints included);durationruns to the item it precedes, clamped at zero. It is an upper bound: without a per-frame duration it starts one frame early, and it covers any declared pause in between. No gap is read before the first item, in either language.Decisions
Read shape shared with #5288 (maintainer, resolved there):
skippedflag and the consumer's separate skip bookkeeping are deleted.skippedboolean (previous revision): a second notion of loss beside the read's gap, and one more field every fake read result had to supply.Gap span (maintainer): every gap but the join knows its span, from the delivered edge to the item it precedes; the join has no start.
timestamp+durationon every gap, documented as an upper bound. The decoders that react to a gap (JSTerminal, Rust audiorestart(), CLI play'svideo_restarted(),replay.ts) start in the same state a gap resets them to, so dropping the join gap changes nothing for them.timestampwould mean the gap's end, unlikeFrame.timestamp.Handle vs. retained snapshot (maintainer):
out.arrivalssnapshots.Handle shape (follows the repo's Consumer convention, as in
Hang.Timeline.Consumer):decoder.arrivals(): Arrivalswithnext(), async iteration, andclose().Signal<readonly Arrival[]>snapshot: convenient to bind, but copying on every frame scales with retained history.Handle buffer bound (settled on the recommendation):
arrivals({ backlog }).Late frames under group reordering (Codex P1 declined; settled on the recommendation):
latein delivery order, against the playout reference Sync actually runs.Video rendition switch:
Impact
Public API (on top of #5288; no wire, catalog, or encoded container change):
@moq/hang: newContainer.ReceivedFrame extends Framewith a required localarrival; the frame item ofContainer.Readcarries it. ProducerFrameinputs are unchanged. Fake read results must supplyarrival.@moq/watch:Sync.received(timestamp, label?, arrival?)takes recorded ingress and returns whether the frame was late; an omitted arrival samples the current time as before.@moq/watch: newAudio.Decoder.arrivals()andVideo.Decoder.arrivals(); exports typesWatch.Arrival(the frame | gap union, the gap carryingtimestampanddurationinTime.Milli) andWatch.Arrivals. Noout.arrivalssignal.@moq/hang(breaking):Container.Read's gap is{ kind: "gap"; timestamp: Time.Micro; duration: Time.Micro }, andnext()no longer yields a gap before the first item.moq-mux(breaking):container::Read::Gap { timestamp, duration }(bothmoq_net::Timestamp), andConsumer::readno longer yields a gap before the first item. In-repo matches (moq-audio,moq-video,moq-cli,moq-c,moq-ffi,moq-gst,moq-rtc,moq-srt, thehangexample) only change their patterns; no FFI, C ABI, or binding surface changes.@moq/hang:Container.Consumer.close()drops an item deferred behind a returned gap, so reads after close returnundefined(a feat(mux): read gaps and declared ends as items in delivery order #5288 follow-up from review).Verification
just checkpasses. Container tests in both languages pin each gap's span (legacy without durations, CMAF with them, a hole after a declared end, the overlapping-edge clamp, cross-scale conversion in Rust) and that no gap precedes the first item. Watch tests cover handle fan-out, release on close (including leaving afor await), drain on decoder close, concurrentnext(), the gap item reaching an audio handle before the frame it precedes, audio and legacy/CMAF video decoder output, and recorded-trace replay where held delivery keeps the recorded arrival and lateness (no gap before the first frame). A nightly benchmark (just js arrivals-bench) sweeps 1/4/16 handles against 0/256/4,096 unread entries; per-handle write+read cost stays flat with no slope in the backlog.The completed quest
quest/m1/watch-arrivals.mdis removed.Follow-ups
quest/m1/voice-buffers.mdandwatch-arrivals-release.md(on Plan the Voice AI demo and its buffer view moq.dev#173) still describe "a signal with a bounded window"; they should name thearrivals()handle and its gap item.(Written by Claude Opus 5.5)
🤖 Generated with Claude Code