diff --git a/c/moq/MIGRATE.md b/c/moq/MIGRATE.md index 4326d19c6d0..98cd2db4c41 100644 --- a/c/moq/MIGRATE.md +++ b/c/moq/MIGRATE.md @@ -4,6 +4,11 @@ How to update your code for each breaking change, newest first. ## Unreleased +### `moq_connection_stats` counts unrouted datagrams ([#7951](https://github.com/moq-dev/moq/pull/7951)) + +`moq_connection_stats` gains `datagrams_unrouted` and `datagrams_unrouted_valid` at its end, which changes its ABI: recompile against the new header. +`datagrams_unrouted` counts the datagrams received for a subscription with no datagram track, and is valid on versions with datagrams (moq-lite 05 and newer). + ### Socket sessions The generated socket endpoint, incoming dial, and stream handles now expose TCP group halves and reliably subscribed track halves beside origin. Regenerate bindings and rebuild the matching native library together. Existing callers need no migration. Socket endpoints require lite-07 or moq-transport on every hop; there is no wire change. diff --git a/dart/moq_ffi/lib/src/moq.dart b/dart/moq_ffi/lib/src/moq.dart index c3864781f9b..e157887146f 100644 --- a/dart/moq_ffi/lib/src/moq.dart +++ b/dart/moq_ffi/lib/src/moq.dart @@ -4160,6 +4160,7 @@ class MoqConnectionStats { final int? packetsReceived; final int? packetsLost; final String? version; + final int? datagramsUnrouted; MoqConnectionStats({ this.rtt, this.estimatedSendRateBps, @@ -4170,6 +4171,7 @@ class MoqConnectionStats { this.packetsReceived, this.packetsLost, this.version = null, + this.datagramsUnrouted = null, }); } @@ -4225,6 +4227,11 @@ class FfiConverterMoqConnectionStats { ); final version = version_lifted.value; new_offset += version_lifted.bytesRead; + final datagramsUnrouted_lifted = FfiConverterOptionalUInt64.read( + Uint8List.view(buf.buffer, new_offset), + ); + final datagramsUnrouted = datagramsUnrouted_lifted.value; + new_offset += datagramsUnrouted_lifted.bytesRead; return LiftRetVal( MoqConnectionStats( rtt: rtt, @@ -4236,6 +4243,7 @@ class FfiConverterMoqConnectionStats { packetsReceived: packetsReceived, packetsLost: packetsLost, version: version, + datagramsUnrouted: datagramsUnrouted, ), new_offset - buf.offsetInBytes, ); @@ -4252,6 +4260,7 @@ class FfiConverterMoqConnectionStats { FfiConverterOptionalUInt64.allocationSize(value.packetsReceived) + FfiConverterOptionalUInt64.allocationSize(value.packetsLost) + FfiConverterOptionalString.allocationSize(value.version) + + FfiConverterOptionalUInt64.allocationSize(value.datagramsUnrouted) + 0; final buf = Uint8List(total_length); write(value, buf); @@ -4296,6 +4305,10 @@ class FfiConverterMoqConnectionStats { value.version, Uint8List.view(buf.buffer, new_offset), ); + new_offset += FfiConverterOptionalUInt64.write( + value.datagramsUnrouted, + Uint8List.view(buf.buffer, new_offset), + ); return new_offset - buf.offsetInBytes; } @@ -4309,6 +4322,7 @@ class FfiConverterMoqConnectionStats { FfiConverterOptionalUInt64.allocationSize(value.packetsReceived) + FfiConverterOptionalUInt64.allocationSize(value.packetsLost) + FfiConverterOptionalString.allocationSize(value.version) + + FfiConverterOptionalUInt64.allocationSize(value.datagramsUnrouted) + 0; } } diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index 9bd3e9c4f3f..ebb435ee7d7 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -120,7 +120,7 @@ inside. The header is the reference: each function carries a doc comment. - **JSON values.** `moq_consume_json_value` fills `moq_json_value` with the document and its frame's timestamp. A snapshot consumer's callback fires for each state in order, but a group superseded before it is read is skipped. - **JSON windows.** `moq_publish_json_window` publishes a sliding run of records: `moq_publish_json_window_push` appends at the back, `moq_publish_json_window_pop` drops from the front, and `moq_publish_json_window_cut` closes the open group at a natural boundary. The catalog has no window mode, so a reader names the track. `moq_consume_json_window_event` fills `moq_json_window_event` with a push (the record at `start`, in `json`), a pop of `[start, end)`, or a skip of `[start, end)` that this reader joined too late to receive. Set `checkpoint_records` to bound what each new group restates for a window you never pop. - **Decoded reads.** `moq_decode_audio` and `moq_decode_video` deliver read ids in order: a frame, a gap where media was lost, or an end where the publisher stopped its encoder. Check `kind` on what `moq_decode_audio_read` or `moq_decode_video_read` writes. Free each id with `moq_decode_audio_read_free` or `moq_decode_video_read_free` promptly: a held video frame holds a decoder buffer, and the decoder stalls. A decoder reads everything the track still holds unless a live player sets `start` to `MOQ_DECODE_START_LATEST`, which joins at the newest group and skips the cached backlog. A decoder setup refusal reports `MOQ_ERROR_UNSUPPORTED_CONFIG` on its terminal callback, so the caller can try another rendition; later peer or decode failures retain their own error codes. -- **Stats.** `moq_session_stats()` fills `moq_connection_stats` with `rtt_us`, `estimated_send_rate_bps`, and the byte and packet counters (`bytes_sent`, `bytes_received`, `bytes_lost`, `packets_sent`, `packets_received`, `packets_lost`). Each has a `_valid` flag, false when the transport does not report it, which is not the same as zero. `moq_session_snapshot()` adds the negotiated `protocol`. A failed dial or dropped connection reaches the session's `on_state` with its `failure` code, and `moq_error()` gives the reason during that callback. Hold the handle returned by `moq_session_recv_bandwidth()` to request receive estimates. `moq_bandwidth_consumer_peek()` reads the estimate and `moq_bandwidth_consumer_changed()` waits for a change. `moq_bandwidth_consumer_close()` releases demand; the last consumer stops PROBE. Stats snapshots do not request it. +- **Stats.** `moq_session_stats()` fills `moq_connection_stats` with `rtt_us`, `estimated_send_rate_bps`, and the byte and packet counters (`bytes_sent`, `bytes_received`, `bytes_lost`, `packets_sent`, `packets_received`, `packets_lost`), and `datagrams_unrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. Each has a `_valid` flag, false when the transport does not report it, which is not the same as zero. `moq_session_snapshot()` adds the negotiated `protocol`. A failed dial or dropped connection reaches the session's `on_state` with its `failure` code, and `moq_error()` gives the reason during that callback. Hold the handle returned by `moq_session_recv_bandwidth()` to request receive estimates. `moq_bandwidth_consumer_peek()` reads the estimate and `moq_bandwidth_consumer_changed()` waits for a change. `moq_bandwidth_consumer_close()` releases demand; the last consumer stops PROBE. Stats snapshots do not request it. - **Viewers.** `moq_session_viewers()` returns a handle that divides the receive estimate among the broadcasts you watch, max-min fair by each one's ceiling. `moq_viewers_watch()` returns a viewer claiming up to `max_bps`, the bitrate of the largest rendition you would switch to; `moq_viewer_rate()` reads its share in bits per second, with `present` false while there is no estimate (keep the current rendition), and `moq_viewer_update()` changes the ceiling. Holding a viewer requests the estimate like a `moq_session_recv_bandwidth()` consumer, and `moq_viewer_close()` releases the claim. The session sends the summed ceilings as its PROBE target; `moq_viewers_set_target()` with `present` true overrides that, a `0` target sends none, and `present` false follows the viewers again. `moq_viewers_watch()`, `moq_viewer_update()`, and `moq_viewers_set_target()` return `MOQ_ERROR_OUT_OF_RANGE`, changing nothing, when the ceilings or the target would pass 2^53 - 1 bps, the most a PROBE target carries. `moq_viewers_close()` releases the registry handle; viewers watched through it stay until each is closed. - **Demand.** A watcher from `moq_publish_track_demand()` (or the media, video, and audio variants) calls back `MOQ_DEMAND_WANTED` or `MOQ_DEMAND_UNWANTED`, again whenever the wanted group moves. `moq_publish_demand_wanted()` reads that group: start a new one once it passes your largest. Set `refresh` in `moq_track_info` to have a joining subscriber ask for a new group once the newest has grown past `refresh` times its first frame. A producer publishes everything you write, subscribed or not, so a late subscriber can fetch the recent past. Capturing only while the watcher reports `MOQ_DEMAND_WANTED` is an opt-in that saves the encode but leaves nothing from before the first subscriber. - **Disabled renditions.** Audio and video consumers and decoders reject a rendition with `enabled: false`. Query the latest catalog with `moq_consume_audio_enabled` or `moq_consume_video_enabled` before selecting a rendition and opening a decoder. Among the video renditions you can decode, prefer the highest `preference`; `0` is the default and `-1` marks a compatibility transcode. diff --git a/doc/lib/dart/index.md b/doc/lib/dart/index.md index dff6b40a757..abfc1fe159c 100644 --- a/doc/lib/dart/index.md +++ b/doc/lib/dart/index.md @@ -156,7 +156,7 @@ shows which module holds each. - **Decoded frames hold decoder buffers.** `dispose()` each frame from a `video.Consumer` promptly rather than waiting for the garbage collector, or the decoder stalls. `resize` is best effort, so read each frame's `width()` and `height()`. - **Closing.** `await moq.close()` drains for up to one second and reports its result. `session.shutdown()` requests the drain synchronously; `await session.closed()` waits for its result. `session.cancel(code: 0)` closes at once. Finish or abort live tracks first. Awaiting cancellation of an announcement, grant, or request subscription releases its pending native read. Announcements and grants also release their consumer; server listeners leave their shared server usable. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. - **Ending cleanly.** When the input ends, finish every track, then call `finish()` on its `CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.close()` or disposing its `MoqRoute` takes it offline. How long to keep it up is yours to decide. A track still open at the close, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). -- **Stats.** `session.stats()` reports `rtt`, `estimatedSendRateBps`, and the byte and packet counters (`bytesSent`, `bytesReceived`, `bytesLost`, `packetsSent`, `packetsReceived`, `packetsLost`). A field is `null` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe first-dial failures and reconnects. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recvBandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rateChanged()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. +- **Stats.** `session.stats()` reports `rtt`, `estimatedSendRateBps`, and the byte and packet counters (`bytesSent`, `bytesReceived`, `bytesLost`, `packetsSent`, `packetsReceived`, `packetsLost`), and `datagramsUnrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is `null` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe first-dial failures and reconnects. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recvBandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rateChanged()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `client.viewers()` divides the connection's receive estimate among the broadcasts you watch, max-min fair by each one's ceiling. `watch(maxBps: ...)` returns a `Viewer` claiming up to what the largest rendition the player would switch to needs; `rate()` reads its slice in bits per second, `null` while there is no estimate, so hold the current selection. `update(maxBps: ...)` changes the ceiling, such as when the catalog enables or disables a rendition. Holding a viewer requests PROBE, and the session sends the viewers' summed ceilings as the PROBE target; `dispose()` it once the broadcast stops being watched. `setTarget(targetBps: ...)` sends another target, `0` sends none, and `null` follows the viewers again. Ceilings or a target past 2^53 - 1 bps, the most a PROBE target carries, throw `OutOfRangeMoqException`. Client viewers survive reconnects and read `null` while disconnected. - **Authorization.** On `moq-lite-07-wip` (opt in with `ConnectOptions(versions: ...)`) and on moq-transport draft-18+ with the [MoQ Auth extension](/draft/moq-auth), listen to `moq.grants().updates()` for the `Grant` the relay sent back and each change, `null` until it answers and while a client session reconnects. - **Tokens.** Pass `ConnectOptions(tokens: [...])` instead of putting the credential in the URL. The first is the connection credential, sent in SETUP and, while an offered version can't carry it there (lite before 07), as the URL's `?jwt=`; each other token widens the grant over AUTH. `moq.tokens().set_(tokens: [...])` replaces them before they expire: the next dial presents them, a live AUTH session at once, and after an authorization refusal the session redials only if they changed. `await tokens().answered()` returns once the relay accepts them and throws its refusal with the relay's code. A `?jwt=` already in the URL stays the connection credential, which is how to reach a relay older than this release. diff --git a/doc/lib/go/index.md b/doc/lib/go/index.md index 219ac0e1a66..563210798c9 100644 --- a/doc/lib/go/index.md +++ b/doc/lib/go/index.md @@ -133,7 +133,7 @@ shows which module holds each. - **Decoded frames hold decoder buffers.** `Close` each `VideoDecodedFrame` promptly, or the decoder stalls. `Resize` is best effort, so read each frame's `Width()` and `Height()`. - **Closing.** `client.Close()` drains for up to one second and returns an error on failure. `session.Shutdown()` requests that drain immediately; `session.Closed(ctx)` waits for its result, and `session.Cancel(code)` closes at once. Finish or abort live tracks first. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. - **Ending cleanly.** When the input ends, finish every track, then call `Finish()` on its `media.CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.Close()` takes it offline. How long to keep it up is yours to decide. A track still open at the close, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). -- **Stats.** `Session().Stats()` reports `RTT`, `EstimatedSendRateBps`, and the byte and packet counters (`BytesSent`, `BytesReceived`, `BytesLost`, `PacketsSent`, `PacketsReceived`, `PacketsLost`). A field is nil when the transport does not report it, which is not the same as zero. `Version` names the negotiated protocol. `Dial` returns a connecting session; read `Session().State()` and wait with `StateChanged(ctx, state.Revision)` to observe first-dial failures and reconnects. Hold the owned `BandwidthConsumer` returned by `Session().RecvBandwidth()` to receive bandwidth estimates. `Rate()` reads the current estimate in bits per second and `RateChanged(ctx)` waits for a new one. Calling `Close()` on the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. +- **Stats.** `Session().Stats()` reports `RTT`, `EstimatedSendRateBps`, and the byte and packet counters (`BytesSent`, `BytesReceived`, `BytesLost`, `PacketsSent`, `PacketsReceived`, `PacketsLost`), and `DatagramsUnrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is nil when the transport does not report it, which is not the same as zero. `Version` names the negotiated protocol. `Dial` returns a connecting session; read `Session().State()` and wait with `StateChanged(ctx, state.Revision)` to observe first-dial failures and reconnects. Hold the owned `BandwidthConsumer` returned by `Session().RecvBandwidth()` to receive bandwidth estimates. `Rate()` reads the current estimate in bits per second and `RateChanged(ctx)` waits for a new one. Calling `Close()` on the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `Session().Viewers()` divides the connection's receive estimate among the broadcasts you watch, max-min fair by each one's ceiling. `Watch(maxBps)` returns a `Viewer` claiming up to what the largest rendition the player would switch to needs; `Rate()` reads its slice in bits per second, nil while there is no estimate, so hold the current selection. `Update(maxBps)` changes the ceiling, such as when the catalog enables or disables a rendition. Holding a viewer requests PROBE, and the session sends the viewers' summed ceilings as the PROBE target; `Close()` it once the broadcast stops being watched, from the goroutine that last used it. `SetTarget(&bps)` sends another target, zero sends none, and nil follows the viewers again. `Watch`, `Update`, and `SetTarget` return `moq.ErrOutOfRange` when the ceilings or the target would pass 2^53 - 1 bps, the most a PROBE target carries. Client viewers survive reconnects and read nil while disconnected. - **Authorization.** On `moq-lite-07-wip` (opt in with `ClientConfig.Versions`) and on moq-transport draft-18+ with the [MoQ Auth extension](/draft/moq-auth), `Session().Grants()` returns a `GrantConsumer`: `Grant()` is the `Grant` the relay sent back, nil until it answers and while a client session reconnects, and `Changed(ctx)` waits for the next one. `Close()` it when done. - **Tokens.** Set `ClientConfig.Tokens` instead of putting the credential in the URL. The first is the connection credential, sent in SETUP and, while an offered version can't carry it there (lite before 07), as the URL's `?jwt=`; each other token widens the grant over AUTH. `Session().Tokens().Set(...)` replaces a client session's tokens before they expire: the next dial presents them, a live AUTH session at once, and after an authorization refusal the session redials only if they changed. `Answered(ctx)` returns nil once the relay accepts them, or its refusal with the relay's code. A `?jwt=` already in the URL stays the connection credential, which is how to reach a relay older than this release. diff --git a/doc/lib/kt/index.md b/doc/lib/kt/index.md index fb5954f5116..d0892470e57 100644 --- a/doc/lib/kt/index.md +++ b/doc/lib/kt/index.md @@ -109,7 +109,7 @@ shows which module holds each. - **Decoded frames hold decoder buffers.** `close()` each frame from a `dev.moq.video.Consumer` promptly (or `use { }` it), or the decoder stalls. `resize` is best effort, so read each frame's `width()` and `height()`. - **Closing.** `moq.shutdown()` requests a drain immediately; suspending `moq.session.closed()` waits up to one second and reports its result. `Moq.close()`, and so `use { }`, cancels at once. Finish or abort live tracks first. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. - **Ending cleanly.** When the input ends, finish every track, then call `finish()` on its `CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.end()` takes it offline. How long to keep it up is yours to decide. A track still open at the end, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). -- **Stats.** `session.stats()` reports `rtt`, `estimatedSendRateBps`, and the byte and packet counters (`bytesSent`, `bytesReceived`, `bytesLost`, `packetsSent`, `packetsReceived`, `packetsLost`). A field is `null` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe first-dial failures and reconnects. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recvBandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rateChanged()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. +- **Stats.** `session.stats()` reports `rtt`, `estimatedSendRateBps`, and the byte and packet counters (`bytesSent`, `bytesReceived`, `bytesLost`, `packetsSent`, `packetsReceived`, `packetsLost`), and `datagramsUnrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is `null` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe first-dial failures and reconnects. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recvBandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rateChanged()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `moq.viewers()` divides the connection's receive estimate among the broadcasts you watch, max-min fair by each one's ceiling. `watch(maxBps)` returns a `Viewer` claiming up to what the largest rendition the player would switch to needs; `rate()` reads its slice in bits per second, `null` while there is no estimate, so hold the current selection. `update(maxBps)` changes the ceiling, such as when the catalog enables or disables a rendition. Holding a viewer requests PROBE, and the session sends the viewers' summed ceilings as the PROBE target; `close()` it, or leave `use { }`, once the broadcast stops being watched. `setTarget(bps)` sends another target, `0` sends none, and `null` follows the viewers again. Ceilings or a target past 2^53 - 1 bps, the most a PROBE target carries, throw `MoqException.OutOfRange`. Client viewers survive reconnects and read `null` while disconnected. - **Authorization.** On `moq-lite-07-wip` (opt in with `ClientConfig(versions = ...)`) and on moq-transport draft-18+ with the [MoQ Auth extension](/draft/moq-auth), collect `moq.grants().updates()` for the `Grant` the relay sent back and each change, `null` until it answers and while a client session reconnects. Ending the collection cancels the consumer. - **Tokens.** Pass `ClientConfig(tokens = listOf(...))` instead of putting the credential in the URL. The first is the connection credential, sent in SETUP and, while an offered version can't carry it there (lite before 07), as the URL's `?jwt=`; each other token widens the grant over AUTH. `moq.tokens().set(listOf(...))` replaces them before they expire: the next dial presents them, a live AUTH session at once, and after an authorization refusal the session redials only if they changed. Suspending `tokens().answered()` returns once the relay accepts them and throws its refusal with the relay's code. A `?jwt=` already in the URL stays the connection credential, which is how to reach a relay older than this release. diff --git a/doc/lib/py/index.md b/doc/lib/py/index.md index 156063cd121..64d40be8805 100644 --- a/doc/lib/py/index.md +++ b/doc/lib/py/index.md @@ -125,7 +125,7 @@ shows which module holds each. - **Decoded frames hold decoder buffers.** Drop each frame from a `moq.video.Consumer` promptly, or the decoder stalls. `resize` is best effort, so read each frame's `width()` and `height()`. - **Closing.** `session.shutdown()` requests a drain immediately; `await session.closed()` waits up to one second and reports its result. `cancel(code)` closes at once. Finish or abort live tracks first. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. - **Ending cleanly.** When the input ends, finish every track, then call `finish()` on its `moq.media.CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.close()` takes it offline. How long to keep it up is yours to decide. A track still open at the close, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). -- **Stats.** `session.stats()` reports `rtt`, `estimated_send_rate_bps`, and the byte and packet counters (`bytes_sent`, `bytes_received`, `bytes_lost`, `packets_sent`, `packets_received`, `packets_lost`). A field is `None` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe first-dial failures and reconnects. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recv_bandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rate_changed()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. +- **Stats.** `session.stats()` reports `rtt`, `estimated_send_rate_bps`, and the byte and packet counters (`bytes_sent`, `bytes_received`, `bytes_lost`, `packets_sent`, `packets_received`, `packets_lost`), and `datagrams_unrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is `None` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe first-dial failures and reconnects. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recv_bandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rate_changed()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `session.viewers()` divides the receive estimate among the broadcasts you watch. `watch(max_bps)` returns a `Viewer` claiming up to a ceiling, the bitrate of the largest rendition you would switch to; its `rate()` reads that viewer's fair share in bits per second, `None` while there is no estimate (keep the current rendition), and `update(max_bps)` changes the ceiling. Holding a viewer requests the estimate like `recv_bandwidth()`, and dropping it releases the claim. The session sends the summed ceilings as its PROBE target; `set_target(bps)` overrides that, `0` sends none, and `None` follows the viewers again. Ceilings or a target past 2^53 - 1 bps, the most a PROBE target carries, raise `moq.Error.OutOfRange`. - **Authorization.** On `moq-lite-07-wip` (opt in with `versions=`) and on moq-transport draft-18+ with the [MoQ Auth extension](/draft/moq-auth), `async for grant in session.grants()` yields the `Grant` the relay sent back and each change, `None` until it answers and while a client session reconnects. `grants().grant()` reads it once; cancelling or leaving an `async with` releases the wait. - **Tokens.** Pass `tokens=[...]` instead of putting the credential in the URL. The first is the connection credential, sent in SETUP and, while an offered version can't carry it there (lite before 07), as the URL's `?jwt=`; each other token widens the grant over AUTH. `session.tokens().set([...])` replaces a client session's tokens before they expire: the next dial presents them, a live AUTH session at once, and after an authorization refusal the session redials only if they changed. `await tokens.answered()` returns once the relay accepts them and raises its refusal with the relay's code. A `?jwt=` already in the URL stays the connection credential, which is how to reach a relay older than this release. diff --git a/doc/lib/swift/index.md b/doc/lib/swift/index.md index cd8fc07854c..7889493d3f4 100644 --- a/doc/lib/swift/index.md +++ b/doc/lib/swift/index.md @@ -107,7 +107,7 @@ shows which module holds each. - **Decoded frames hold decoder buffers.** Release each frame from a `Video.Consumer` promptly, or the decoder stalls. `resize` is best effort, so read each frame's `width()` and `height()`. - **Closing.** `session.shutdown()` requests a drain immediately; `try await session.closed()` waits up to one second and reports its result. `session.cancel(code:)` closes at once. Finish or abort live tracks first. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. - **Ending cleanly.** When the input ends, finish every track, then call `try finish()` on its `Media.CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.close()`, or releasing its route, takes it offline. How long to keep it up is yours to decide. A track still open at the close, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). -- **Stats.** `session.stats()` reports `rtt`, `estimatedSendRateBps`, and the byte and packet counters (`bytesSent`, `bytesReceived`, `bytesLost`, `packetsSent`, `packetsReceived`, `packetsLost`). A field is `nil` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe first-dial failures and reconnects. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recvBandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rateChanged()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. +- **Stats.** `session.stats()` reports `rtt`, `estimatedSendRateBps`, and the byte and packet counters (`bytesSent`, `bytesReceived`, `bytesLost`, `packetsSent`, `packetsReceived`, `packetsLost`), and `datagramsUnrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is `nil` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe first-dial failures and reconnects. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recvBandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rateChanged()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `session.viewers()` divides the receive estimate among the broadcasts you watch. `watch(maxBps:)` returns a `Viewer` claiming up to a ceiling, the bitrate of the largest rendition you would switch to; its `rate()` reads that viewer's fair share in bits per second, `nil` while there is no estimate (keep the current rendition), and `update(maxBps:)` changes the ceiling. Holding a viewer requests the estimate like `recvBandwidth()`, and releasing it releases the claim. The session sends the summed ceilings as its PROBE target; `setTarget(targetBps:)` overrides that, `0` sends none, and `nil` follows the viewers again. `watch`, `update`, and `setTarget` throw `MoqError.OutOfRange` when the ceilings or the target would pass 2^53 - 1 bps, the most a PROBE target carries. - **Authorization.** On `moq-lite-07-wip` (opt in with `Client(versions:)`) and on moq-transport draft-18+ with the [MoQ Auth extension](/draft/moq-auth), `for try await grant in session.grants()` yields the `Grant` the relay sent back and each change, `nil` until it answers and while a client session reconnects. Cancelling the iterating task cancels the native wait. - **Tokens.** Pass `Client(tokens: [...])` instead of putting the credential in the URL. The first is the connection credential, sent in SETUP and, while an offered version can't carry it there (lite before 07), as the URL's `?jwt=`; each other token widens the grant over AUTH. `try session.tokens()?.set([...])` replaces them before they expire: the next dial presents them, a live AUTH session at once, and after an authorization refusal the session redials only if they changed. `try await tokens.answered()` returns once the relay accepts them and throws its refusal with the relay's code. A `?jwt=` already in the URL stays the connection credential, which is how to reach a relay older than this release. diff --git a/go/wrapper/records.go b/go/wrapper/records.go index 3dbf470e4ca..30d7456af97 100644 --- a/go/wrapper/records.go +++ b/go/wrapper/records.go @@ -208,6 +208,9 @@ type ConnectionStats struct { PacketsLost *uint64 // Version is the negotiated protocol version, spelled like ClientConfig.Versions. Version *string + // DatagramsUnrouted counts datagrams received for a subscription with no datagram + // track; nil on a version without datagrams. + DatagramsUnrouted *uint64 } func connectionStatsFromFFI(s ffi.MoqConnectionStats) ConnectionStats { @@ -221,6 +224,7 @@ func connectionStatsFromFFI(s ffi.MoqConnectionStats) ConnectionStats { PacketsReceived: s.PacketsReceived, PacketsLost: s.PacketsLost, Version: s.Version, + DatagramsUnrouted: s.DatagramsUnrouted, } } diff --git a/js/net/api/index.api.md b/js/net/api/index.api.md index d121e0b669d..47d2adcf4dc 100644 --- a/js/net/api/index.api.md +++ b/js/net/api/index.api.md @@ -220,6 +220,7 @@ export namespace Connection { bytesLost?: number; bytesReceived?: number; bytesSent?: number; + datagramsUnrouted?: number; estimatedSendRate?: number; packetsLost?: number; packetsReceived?: number; diff --git a/js/net/src/connection/stats.test.ts b/js/net/src/connection/stats.test.ts index 87117c84316..3e42174d605 100644 --- a/js/net/src/connection/stats.test.ts +++ b/js/net/src/connection/stats.test.ts @@ -76,14 +76,15 @@ test("stats() snapshots the transport on demand", async () => { } }); -test("stats() is empty on a transport without getStats", async () => { +test("stats() has no transport counters on a transport without getStats", async () => { const pair = createMockTransportPair(Lite.ALPN_06); // Shadow the mock's getStats to imitate the qmux/WebSocket fallback. (pair.client as unknown as { getStats?: unknown }).getStats = undefined; const url = new URL("https://example.com/"); const [client, server] = await Promise.all([connect(url, { transport: pair.client }), accept(pair.server, url)]); try { - expect(await client.stats()).toEqual({}); + // Only the session's own counter, which needs nothing from the transport. + expect(await client.stats()).toEqual({ datagramsUnrouted: 0 }); } finally { client.abort(); server.abort(); diff --git a/js/net/src/connection/stats.ts b/js/net/src/connection/stats.ts index 25fccbe8578..556860917d3 100644 --- a/js/net/src/connection/stats.ts +++ b/js/net/src/connection/stats.ts @@ -53,6 +53,13 @@ export interface Stats { /** Packets currently declared lost. The estimate can decrease when a loss proves spurious. */ packetsLost?: number; + + /** + * Datagrams received for a subscription with no datagram track: one this session doesn't + * know, has ended, or whose answer it hasn't received. Absent on a version without datagrams: + * moq-transport and moq-lite before 05. + */ + datagramsUnrouted?: number; } /** diff --git a/js/net/src/lite/connection.ts b/js/net/src/lite/connection.ts index 9c4ae23b308..93e07323944 100644 --- a/js/net/src/lite/connection.ts +++ b/js/net/src/lite/connection.ts @@ -540,7 +540,9 @@ export class Connection implements Established { /** Snapshot the transport's counters; see {@link Established.stats}. */ async stats(): Promise { - return transportStats(this.#quic); + const stats = await transportStats(this.#quic); + if (hasDatagrams(this.#version)) stats.datagramsUnrouted = this.#subscriber.datagramsUnrouted; + return stats; } /** Settles with the peer's GOAWAY; see {@link Established.draining}. */ diff --git a/js/net/src/lite/datagram-tracks.test.ts b/js/net/src/lite/datagram-tracks.test.ts index 88d167a49fa..a79af11067f 100644 --- a/js/net/src/lite/datagram-tracks.test.ts +++ b/js/net/src/lite/datagram-tracks.test.ts @@ -1,10 +1,13 @@ -import { expect, test } from "bun:test"; +import { expect, spyOn, test } from "bun:test"; +import { yieldNow } from "@moq/clock"; +import { Signal } from "@moq/signals"; import { accept, connect } from "../connection/index.ts"; -import { createMockTransportPair } from "../mock.ts"; +import { createMockTransportPair, type MockTransport } from "../mock.ts"; import { Producer as Origin } from "../origin.ts"; import * as Path from "../path.ts"; import { sendOrder } from "../priority.ts"; import { Timescale, Timestamp } from "../time.ts"; +import { Budget } from "../util/yield.ts"; import { wireOf } from "../wire.ts"; import { ALPN_04, ALPN_05, ALPN_06, ALPN_07_WIP } from "./version.ts"; @@ -281,3 +284,311 @@ test("lite-06: a request cancelled behind the shared datagram writer sends nothi source.close(); } }); + +// A publisher's stream writes, handed to the subscriber `lag` microtasks late: the answer it +// writes lands after a datagram sent behind it, though in the same host task. +function lagAnswers(transport: MockTransport, lag: number): void { + if (lag === 0) return; + const lagging = (writable: WritableStream) => { + const writer = writable.getWriter(); + let chain = Promise.resolve(); + const later = (op: () => Promise) => { + // A write the session's close fails is lost, as it would be on the wire. + chain = chain + .then(async () => { + for (let i = 0; i < lag; i++) await Promise.resolve(); + await op(); + }) + .catch(() => {}); + return chain; + }; + return new WritableStream({ + // Copied: the write resolves now, after which the stream may reuse its buffer. + write: (chunk) => { + const copy = chunk.slice(); + void later(() => writer.write(copy)); + }, + close: () => later(() => writer.close()), + abort: (reason) => writer.abort(reason), + }); + }; + const incoming = transport.incomingBidirectionalStreams.pipeThrough( + new TransformStream({ + transform: (bidi, controller) => + controller.enqueue({ + readable: bidi.readable, + writable: lagging(bidi.writable), + } as WebTransportBidirectionalStream), + }), + ); + Object.defineProperty(transport, "incomingBidirectionalStreams", { value: incoming }); +} + +// A publisher that writes as soon as a subscription reaches it sends the datagram right behind +// its answer, which may reach the subscriber first, with the track's end too when it finishes at +// once. The subscriber decodes the answer before it routes the datagram, and routes the datagram +// before it ends the track. +for (const alpn of [ALPN_05, ALPN_06, ALPN_07_WIP]) { + for (const [lag, finish] of [ + [0, false], + [100, false], + [0, true], + [100, true], + ] as const) { + test(`${alpn}: a datagram sent right behind its answer arrives (answer ${lag} microtasks late${finish ? ", then the end" : ""})`, async () => { + const source = new Origin(); + const publication = source.route(path, { hidden: true }); + const datagrams = publication.broadcast.createDatagrams("audio"); + publication.announce(); + const pair = createMockTransportPair(alpn); + lagAnswers(pair.server, lag); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(client).consume(path); + const reader = remote.datagrams("audio"); + // Only the subscription carries this priority: a lite-06 TRACK lookup asks for none. + reader.setPriority(5); + try { + while (datagrams.priority.peek() !== 5) await datagrams.priority.changed(); + datagrams.append(undefined, Uint8Array.of(1)); + if (finish) datagrams.finish(); + await reader.info(); + if (!finish) datagrams.append(undefined, Uint8Array.of(2)); + expect((await reader.recv())?.payload).toEqual(Uint8Array.of(1)); + if (finish) expect(await reader.recv()).toBeUndefined(); + else expect((await reader.recv())?.payload).toEqual(Uint8Array.of(2)); + expect((await client.stats()).datagramsUnrouted).toBe(0); + } finally { + reader.close(); + remote.close(); + client.abort(); + server.abort(); + source.close(); + } + }); + } +} + +// A publisher that writes a last datagram and finishes at once sends the datagram right ahead +// of its track's end. The subscriber routes the datagram before it ends the track. +for (const alpn of [ALPN_05, ALPN_06, ALPN_07_WIP]) { + test(`${alpn}: a datagram sent right before its end arrives`, async () => { + const source = new Origin(); + const publication = source.route(path, { hidden: true }); + const datagrams = publication.broadcast.createDatagrams("audio"); + publication.announce(); + const pair = createMockTransportPair(alpn); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(client).consume(path); + const reader = remote.datagrams("audio"); + reader.setPriority(5); + try { + await reader.info(); + while (datagrams.priority.peek() !== 5) await datagrams.priority.changed(); + datagrams.append(undefined, Uint8Array.of(1)); + datagrams.finish(); + expect((await reader.recv())?.payload).toEqual(Uint8Array.of(1)); + expect(await reader.recv()).toBeUndefined(); + expect((await client.stats()).datagramsUnrouted).toBe(0); + } finally { + reader.close(); + remote.close(); + client.abort(); + server.abort(); + source.close(); + } + }); +} + +// A burst written right before a track's end can outlast the receive loop's budget, which yields +// with some of it still queued. The end waits for the loop to read the queue empty. +for (const alpn of [ALPN_05, ALPN_06, ALPN_07_WIP]) { + test(`${alpn}: a burst sent right before its end arrives past a budget yield`, async () => { + // As many as a datagram track buffers unread, past two receive budgets. + const BURST = 64; + const source = new Origin(); + const publication = source.route(path, { hidden: true }); + const datagrams = publication.broadcast.createDatagrams("audio"); + publication.announce(); + const pair = createMockTransportPair(alpn); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(client).consume(path); + const reader = remote.datagrams("audio"); + reader.setPriority(5); + // Every budget slice reads as spent, so the receive loop yields a host task every 32 checks. + let checks = 0; + const budget = spyOn(Budget.prototype, "poll").mockImplementation(() => + ++checks % 32 === 0 ? yieldNow() : undefined, + ); + try { + await reader.info(); + while (datagrams.priority.peek() !== 5) await datagrams.priority.changed(); + for (let i = 0; i < BURST; i++) datagrams.append(undefined, Uint8Array.of(i)); + datagrams.finish(); + let count = 0; + while ((await reader.recv()) !== undefined) count++; + expect(count).toBe(BURST); + expect((await client.stats()).datagramsUnrouted).toBe(0); + } finally { + budget.mockRestore(); + reader.close(); + remote.close(); + client.abort(); + server.abort(); + source.close(); + } + }); +} + +// A track's end waits a host task for the datagrams received with it. A session close in +// between finishes the track rather than cutting it off: the publisher already ended it. +for (const alpn of [ALPN_05, ALPN_06, ALPN_07_WIP]) { + test(`${alpn}: a session close right after a datagram track's end keeps the end`, async () => { + const source = new Origin(); + const publication = source.route(path, { hidden: true }); + const datagrams = publication.broadcast.createDatagrams("audio"); + publication.announce(); + const pair = createMockTransportPair(alpn); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(client).consume(path); + const reader = remote.datagrams("audio"); + reader.setPriority(5); + try { + await reader.info(); + while (datagrams.priority.peek() !== 5) await datagrams.priority.changed(); + datagrams.finish(); + // The mock delivers the end in microtasks, all within this host task. + for (let i = 0; i < 1000; i++) await Promise.resolve(); + client.abort(); + expect(await reader.recv()).toBeUndefined(); + } finally { + reader.close(); + remote.close(); + client.abort(); + server.abort(); + source.close(); + } + }); +} + +// Holds back what the publisher writes on the next request stream it accepts once armed, until +// released: that subscription's answer stays outstanding while everything else flows. +function holdNextAnswer(transport: MockTransport): { arm(): void; release(): void } { + let armed = false; + const release = Promise.withResolvers(); + const holding = (writable: WritableStream) => { + const writer = writable.getWriter(); + let chain = release.promise; + // A write the session's close fails is lost, as it would be on the wire. + const later = (op: () => Promise) => { + chain = chain.then(op).catch(() => {}); + return chain; + }; + return new WritableStream({ + // Copied: the write resolves now, after which the stream may reuse its buffer. + write: (chunk) => { + const copy = chunk.slice(); + void later(() => writer.write(copy)); + }, + close: () => void later(() => writer.close()), + abort: (reason) => writer.abort(reason), + }); + }; + const incoming = transport.incomingBidirectionalStreams.pipeThrough( + new TransformStream({ + transform: (bidi, controller) => { + const held = armed; + armed = false; + controller.enqueue( + held + ? ({ + readable: bidi.readable, + writable: holding(bidi.writable), + } as WebTransportBidirectionalStream) + : bidi, + ); + }, + }), + ); + Object.defineProperty(transport, "incomingBidirectionalStreams", { value: incoming }); + return { + arm: () => { + armed = true; + }, + release: () => release.resolve(), + }; +} + +// Datagrams held for a track still awaiting its answer, more than are held at once, don't hold +// up an answered track's last datagram, which its end, arriving with it, would otherwise drop. +test("lite-07: an unanswered track's datagrams don't delay another track's last", async () => { + const source = new Origin(); + const publication = source.route(path, { hidden: true }); + const b = publication.broadcast.createDatagrams("b"); + // Answers "a" on demand. + const requested = wireOf(publication.broadcast).requestedDatagrams(); + publication.announce(); + const pair = createMockTransportPair(ALPN_07_WIP); + const hold = holdNextAnswer(pair.server); + // Counts the datagrams the publisher sends. + const sending = pair.server.datagrams.writable.getWriter(); + let sent = 0; + const sends = new Signal(0); + Object.defineProperty(pair.server.datagrams, "writable", { + value: new WritableStream({ + write: async (chunk) => { + await sending.write(chunk); + sends.set(++sent); + }, + }), + }); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(client).consume(path); + const readerB = remote.datagrams("b"); + readerB.setPriority(5); + let readerA: ReturnType | undefined; + try { + await readerB.info(); + while (b.priority.peek() !== 5) await b.priority.changed(); + hold.arm(); + readerA = remote.datagrams("a"); + const request = await requested; + if (!request) throw new Error("the request never reached the handler"); + const a = request.accept({}); + for (let i = 0; i < 40; i++) a.append(undefined, Uint8Array.of(i)); + // "a"'s datagrams reach the subscriber ahead of "b"'s. + while (sends.peek() < 40) await sends.changed(); + b.append(undefined, Uint8Array.of(9)); + b.finish(); + expect((await readerB.recv())?.payload).toEqual(Uint8Array.of(9)); + expect(await readerB.recv()).toBeUndefined(); + // Those past the cap were dropped at once, without stalling "b". + expect((await client.stats()).datagramsUnrouted).toBeGreaterThanOrEqual(8); + // "a" stays unanswered past the host task its held datagrams wait, so they drop too. + while (((await client.stats()).datagramsUnrouted ?? 0) < 40) await yieldNow(); + expect((await client.stats()).datagramsUnrouted).toBe(40); + hold.release(); + await readerA.info(); + } finally { + readerA?.close(); + readerB.close(); + remote.close(); + client.abort(); + server.abort(); + source.close(); + } +}); diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index b0a3061306e..6e25f0d4904 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -1,4 +1,4 @@ -import { Milli } from "@moq/clock"; +import { Milli, yieldNow } from "@moq/clock"; import { type Dispose, Effect, type GetPromise, type Getter, race, Signal } from "@moq/signals"; import * as announce from "../announced.ts"; import type { Grant } from "../auth.ts"; @@ -96,6 +96,10 @@ import { // and removals of one name never reaches the announces cap: yield so it cannot starve the host. const listingBudget = new Budget(); const datagramBudget = new Budget(); +// The datagrams held for unanswered subscriptions at once, as many as a Rust pass holds: past it, +// one is dropped and counted rather than stalling the receive queue, which would hold up other +// tracks' datagrams behind their ends. +const HELD_DATAGRAMS = 32; // Request streams refuse exhausted peer credit, with one deadline covering create and answer. export const SUBSCRIBE_SETUP_TIMEOUT_MS = REQUEST_TIMEOUT_MS; @@ -136,6 +140,9 @@ interface Consumed { interface DatagramEntry { producer: datagram.Producer; timescale?: Time.Timescale; + // Set once the publisher ended the track, while the datagrams received with the end route: + // a session close meanwhile finishes it rather than cutting it off. + ended?: boolean; } interface SubscribeEntry { @@ -247,6 +254,16 @@ export class Subscriber { #datagrams = new Map(); #legacyDatagrams = new Map(); #legacyRequests = new Map(); + // Lite-07 datagram requests sent and not yet answered. + #unanswered = new Set(); + // Datagrams for a subscription still awaiting its answer, in arrival order, routed a host task + // later while the rest go on. + #held: { subscribe: bigint; payload: Uint8Array }[] = []; + #heldRouted?: Promise; + // Set while the receive loop yields to its budget, which may leave datagrams queued. + #receiveYield?: Promise; + /** Datagrams received for a subscription with no datagram track; see `Stats.datagramsUnrouted`. */ + datagramsUnrouted = 0; // A SUBSCRIBE sent alongside its TRACK, until its TRACK_INFO lands: settles with the // entry, or `undefined` if it never does. #pending = new Map>(); @@ -895,6 +912,7 @@ export class Subscriber { { version: this.version }, async (opened) => { stream = opened; + this.#unanswered.add(id); await opened.writer.u53(StreamId.Datagrams); await new Datagrams({ id, @@ -913,12 +931,17 @@ export class Subscriber { signal.throwIfAborted(); if (!stream) throw new Error("datagram stream was not opened"); producer = request.accept(answered); - this.#datagrams.set(id, { producer, timescale: answered.timescale }); + const route: DatagramEntry = { producer, timescale: answered.timescale }; + this.#datagrams.set(id, route); + this.#unanswered.delete(id); const opened = stream; const receiving = producer; - // The only thing following DATAGRAMS_INFO is the publisher's FIN. - const ended = opened.reader.done().then((done) => { + // The only thing following DATAGRAMS_INFO is the publisher's FIN. Its last datagram may + // arrive alongside, so the datagrams already received route first. + const ended = opened.reader.done().then(async (done) => { if (!done) throw new ProtocolViolation("data after DATAGRAMS_INFO"); + route.ended = true; + await this.#datagramsRouted(id); receiving.finish(); }); void ended.catch(() => {}); @@ -946,6 +969,7 @@ export class Subscriber { if (legacyKey !== undefined && this.#legacyRequests.get(legacyKey) === request) this.#legacyRequests.delete(legacyKey); grant?.(); + this.#unanswered.delete(id); this.#datagrams.delete(id); cancel.abort(new StreamError(StreamCode.Cancel)); stream?.reader.stop(StreamCode.Cancel); @@ -1373,6 +1397,11 @@ export class Subscriber { entry.held?.close(); entry.datagramDemand?.(); if (entry.datagrams) { + // The stream track's end is its datagrams' end: those received with it route first. + if (entry.end !== undefined && !(hooks.trackReleased(producer).peek() instanceof Error)) { + entry.datagrams.ended = true; + await this.#datagramsRouted(id); + } const closed = hooks.trackReleased(producer).peek(); if (closed instanceof Error) entry.datagrams.producer.abort(closed); else if (entry.end !== undefined) entry.datagrams.producer.finish(); @@ -2386,7 +2415,11 @@ export class Subscriber { try { for (;;) { const yielded = datagramBudget.poll(); - if (yielded !== undefined) await yielded; + if (yielded !== undefined) { + this.#receiveYield = yielded; + await yielded; + this.#receiveYield = undefined; + } const { value, done } = await reader.read(); if (done) break; if (!value) continue; @@ -2410,24 +2443,92 @@ export class Subscriber { } } - // Decode one datagram body and hand it to the matching subscription's producer. Drops the - // datagram (best-effort) if the subscription is unknown or closed. + // Decode one datagram body and hand it to the datagram track it was sent for. A publisher + // sends a track's first datagram right behind its answer, which the transport may hand over + // first: one for a subscription still awaiting its answer is held a host task, so an answer + // already received decodes before it, while other tracks' datagrams go on. One that still + // finds no track is dropped and counted. async #routeDatagram(payload: Uint8Array): Promise { if (payload.byteLength > DatagramMessage.MAX_SIZE) return; + let unrouted = await this.#deliverDatagram(payload, true); + if (unrouted === undefined) return; + // Its track may have registered while the decode awaited. + if (!this.#isHeld(unrouted) && this.#datagrams.has(unrouted)) + unrouted = await this.#deliverDatagram(payload, true); + if (unrouted === undefined) return; + const awaiting = this.#isHeld(unrouted) || this.#unanswered.has(unrouted) || this.#pending.has(unrouted); + if (awaiting && this.#held.length < HELD_DATAGRAMS) { + this.#held.push({ subscribe: unrouted, payload }); + this.#heldRouted ??= this.#routeHeld(); + return; + } + this.#dropUnrouted(unrouted); + } + + // Wait for the datagrams received for `subscribe` so far to route: a host task, then until the + // receive loop reads the transport's queue empty rather than yielding to its budget, and those + // held for its answer. A peer keeping the queue full delays the end, holding nothing more. + async #datagramsRouted(subscribe: bigint): Promise { + await yieldNow(); + for (;;) { + const receiveYield = this.#receiveYield; + if (receiveYield !== undefined) { + await receiveYield; + // The loop reads on in microtasks, until the queue empties or its next yield. + await yieldNow(); + } else if (this.#isHeld(subscribe)) { + // A held datagram keeps its routing pass running. + await this.#heldRouted; + } else return; + } + } + + // Whether a datagram for `subscribe` is held, which later ones wait behind to keep its order. + #isHeld(subscribe: bigint): boolean { + return this.#held.some((held) => held.subscribe === subscribe); + } + + // A host task later, route the datagrams held until now, then any held meanwhile. + async #routeHeld(): Promise { + await yieldNow(); + for (let count = this.#held.length; count > 0; count--) { + const [{ payload }] = this.#held; + try { + const unrouted = await this.#deliverDatagram(payload, false); + if (unrouted !== undefined) this.#dropUnrouted(unrouted); + } catch (err: unknown) { + console.debug(`dropping datagram: ${reason(err)}`); + } + this.#held.shift(); + } + this.#heldRouted = this.#held.length > 0 ? this.#routeHeld() : undefined; + } + + #dropUnrouted(subscribe: bigint): void { + this.datagramsUnrouted++; + console.debug(`dropping datagram for a subscription with no datagram track: id=${subscribe}`); + } + + // Deliver one datagram body, returning its Subscribe ID if that subscription has no datagram + // track, or, `behindHeld`, has a datagram held. + async #deliverDatagram(payload: Uint8Array, behindHeld: boolean): Promise { // Read with the entry it was decoded with: one registered while the decode awaited would - // read a datagram parsed with the defaults. An unknown subscription's datagram, or one - // whose TRACK_INFO had not landed, which a datagram cannot wait for, is dropped below. + // read a datagram parsed with the defaults. let entry: DatagramEntry | undefined; const dg = await DatagramMessage.decode(payload, this.version, (id) => { entry = this.#datagrams.get(id); return { timed: entry?.timescale !== undefined }; }); - if (!dg || !entry || this.#datagrams.get(dg.subscribe) !== entry) return; + // Padding, which names no subscription. + if (!dg) return undefined; + if (!entry || this.#datagrams.get(dg.subscribe) !== entry) return dg.subscribe; + if (behindHeld && this.#isHeld(dg.subscribe)) return dg.subscribe; const timestamp = dg.timestamp === undefined || entry.timescale === undefined ? undefined : new Time.Timestamp(dg.timestamp, entry.timescale); entry.producer.insert({ sequence: dg.sequence, timestamp, payload: dg.payload }); + return undefined; } /** @@ -2557,7 +2658,7 @@ export class Subscriber { /** * Ends every subscribed track, fetch, and setup exchange with `err`, the session's end, which * defaults to a clean close: they were cut off rather than ended. A track whose publisher - * already ended it finishes instead. + * already ended it finishes instead, datagram tracks included. */ close(err: Error = new SessionError(SessionCode.Cancel)) { this.#closed.abort(err); @@ -2568,7 +2669,10 @@ export class Subscriber { } this.#subscribes.clear(); - for (const { producer } of this.#datagrams.values()) producer.abort(err); + for (const { producer, ended } of this.#datagrams.values()) { + if (ended) producer.finish(); + producer.abort(err); + } this.#datagrams.clear(); this.#legacyDatagrams.clear(); diff --git a/py/moq-rs/moq/_records.py b/py/moq-rs/moq/_records.py index 1d8b712c5a3..d445a85e38d 100644 --- a/py/moq-rs/moq/_records.py +++ b/py/moq-rs/moq/_records.py @@ -448,6 +448,8 @@ class ConnectionStats: packets_lost: int | None version: str | None = None """The negotiated protocol version, spelled like ``Client(versions=...)``.""" + datagrams_unrouted: int | None = None + """Datagrams received for a subscription with no datagram track; ``None`` on a version without datagrams.""" @staticmethod def _from_ffi(stats: MoqConnectionStats) -> ConnectionStats: @@ -461,6 +463,7 @@ def _from_ffi(stats: MoqConnectionStats) -> ConnectionStats: packets_received=stats.packets_received, packets_lost=stats.packets_lost, version=stats.version, + datagrams_unrouted=stats.datagrams_unrouted, ) diff --git a/quest/m1/datagram-tracks/README.md b/quest/m1/datagram-tracks/README.md index def636b5d50..ffde2a0114f 100644 --- a/quest/m1/datagram-tracks/README.md +++ b/quest/m1/datagram-tracks/README.md @@ -121,7 +121,6 @@ datagram track on lite-07, direct and through a relay. ## Required - [moq-transport datagram tracks](/quest/m1/datagram-tracks/ietf.md) - Rust and JS map a datagram track to SUBSCRIBE and OBJECT_DATAGRAM with object ID 0, keeping the requested kind -- [A datagram sent right behind its answer reaches the subscriber](/quest/m1/datagram-tracks/answer-race.md) - receivers decode answers before routing datagrams, and count the ones they still can't route ## Related diff --git a/quest/m1/datagram-tracks/answer-race.md b/quest/m1/datagram-tracks/answer-race.md deleted file mode 100644 index d413c2baf1b..00000000000 --- a/quest/m1/datagram-tracks/answer-race.md +++ /dev/null @@ -1,57 +0,0 @@ -# [M] A datagram sent right behind its answer reaches the subscriber - -## Goal - -On lite-05, 06, and 07, in Rust and JS, a datagram that arrives in the same -receive pass as its subscription's answer (TRACK_INFO on lite-05/06, -DATAGRAMS_INFO on lite-07) is delivered, not dropped. A datagram the -receiver still cannot route is logged at debug and counted, never dropped -silently. - -Not in scope: a datagram whose answer is lost and retransmitted. That is -ordinary datagram loss, and the receiver keeps no buffer for it. - -## Plan - -Decided 2026-11-05 (maintainer, /quest-import of -[#7882](https://github.com/moq-dev/moq/issues/7882)): - -- ✅ **Decode answers before routing datagrams.** The receiver handles its - pending stream answers before it routes datagrams, so a datagram that - shares a packet with its answer finds its route. Not chosen: buffering raw - datagrams per unanswered subscription and replaying them (state a peer - influences, needing a bound), the publisher holding datagrams until its - answer is acknowledged (about one RTT of datagrams lost per subscription - instead), or accepting the loss with only a log. -- ✅ **One quest for Rust and JS.** The JS part builds on the paths - [JS datagram tracks](https://github.com/moq-dev/moq/pull/7846) introduces. -- ✅ m1, in the Datagram tracks epic. - -Verified on `main` at 246916845a, from the issue's report: - -- The issue's second question, the `newest - budget` start rule, no longer - applies. Every version and both languages start datagrams live from the - request (`rs/moq-net/src/model/datagram.rs`, `js/net/src/track.ts`), so - this quest leaves the start rule alone. -- Rust: `route_datagram` returns `Ok(())` when `datagram_target` finds no - route (`rs/moq-net/src/lite/subscriber.rs`). Routes register only once the - answer decodes: `register` after DATAGRAMS_INFO on lite-07, and - `legacy_datagrams` after TRACK_INFO on lite-05/06. The driver drains up to - 32 datagrams before polling the source tasks that decode answers, so even a - datagram sharing its answer's packet is dropped. -- The publisher sends datagrams as soon as the answer is written - (`rs/moq-net/src/lite/publisher.rs`), so the first live datagram is the one - at risk. -- JS: `#routeDatagram` returns silently on a miss - (`js/net/src/lite/subscriber.ts`), and #7846 keeps that. -- No stats count the drop; `datagrams` in `rs/moq-net/src/stats.rs` counts - deliveries only. - -Tests: per version, a datagram written in the same packet as its answer -arrives, and an unroutable one bumps the counter. The integration tests in -`rs/moq-net/tests/integration/datagram.rs` all wait for `consumer.info()` -before appending, so add a case that doesn't. - -## Closes - -- [#7882](https://github.com/moq-dev/moq/issues/7882) - the datagram served at subscribe time is dropped by the receiver when TRACK and SUBSCRIBE are pipelined diff --git a/quest/m2/recut-held.md b/quest/m2/recut-held.md index 296a3e42c25..59b0ee3575d 100644 --- a/quest/m2/recut-held.md +++ b/quest/m2/recut-held.md @@ -30,7 +30,6 @@ padding probes in [lite07-finalize](/quest/m2/lite07-finalize.md). ## Required -- [Datagram answer race](/quest/m1/datagram-tracks/answer-race.md) - claim held datagram-tracks - [IETF datagram tracks](/quest/m1/datagram-tracks/ietf.md) - claim held datagram-tracks - [rind CPU error](/quest/m1/rind/cpu-error.md) - claim held rind - [Connect policy](/quest/m1/session-policy/connect.md) - claim held session-policy diff --git a/rs/moq-c/MIGRATE.md b/rs/moq-c/MIGRATE.md index 4ea469fb3eb..2f11ec28ba1 100644 --- a/rs/moq-c/MIGRATE.md +++ b/rs/moq-c/MIGRATE.md @@ -4,6 +4,11 @@ How to update your code for each breaking change, newest first. ## Unreleased +### `moq_connection_stats` counts unrouted datagrams ([#7951](https://github.com/moq-dev/moq/pull/7951)) + +`moq_connection_stats` gains `datagrams_unrouted` and `datagrams_unrouted_valid` at its end, which changes its ABI: recompile against the new header. +`datagrams_unrouted` counts the datagrams received for a subscription with no datagram track, and is valid on versions with datagrams (moq-lite 05 and newer). + ### Consumers and decoders refuse a disabled rendition ([#7942](https://github.com/moq-dev/moq/pull/7942)) `moq_consume_audio`, `moq_consume_video`, `moq_decode_audio`, and `moq_decode_video` return `MOQ_ERROR_NOT_FOUND` when the catalog snapshot marks the rendition disabled, instead of subscribing and waiting for a re-enable. Check `moq_consume_audio_enabled` or `moq_consume_video_enabled` on the latest snapshot first, and cancel a consumer whose rendition a newer snapshot disables. diff --git a/rs/moq-c/src/api.rs b/rs/moq-c/src/api.rs index adb4ae1186a..d8d438b7c50 100644 --- a/rs/moq-c/src/api.rs +++ b/rs/moq-c/src/api.rs @@ -1159,6 +1159,12 @@ pub struct moq_connection_stats { pub packets_lost: u64, /// Whether `packets_lost` is available in this sample. pub packets_lost_valid: bool, + + /// Total datagrams received for a subscription with no datagram track: one the session + /// doesn't know, has ended, or whose answer it hasn't received. + pub datagrams_unrouted: u64, + /// Whether `datagrams_unrouted` is available: false on a version without datagrams. + pub datagrams_unrouted_valid: bool, } impl From<&moq_net::session::Stats> for moq_connection_stats { @@ -1177,6 +1183,7 @@ impl From<&moq_net::session::Stats> for moq_connection_stats { let (packets_sent, packets_sent_valid) = split(stats.packets_sent); let (packets_received, packets_received_valid) = split(stats.packets_received); let (packets_lost, packets_lost_valid) = split(stats.packets_lost); + let (datagrams_unrouted, datagrams_unrouted_valid) = split(stats.datagrams_unrouted); Self { rtt_us, @@ -1195,6 +1202,8 @@ impl From<&moq_net::session::Stats> for moq_connection_stats { packets_received_valid, packets_lost, packets_lost_valid, + datagrams_unrouted, + datagrams_unrouted_valid, } } } diff --git a/rs/moq-ffi/src/session.rs b/rs/moq-ffi/src/session.rs index 035747bfe7a..6f9bceae7a5 100644 --- a/rs/moq-ffi/src/session.rs +++ b/rs/moq-ffi/src/session.rs @@ -679,6 +679,11 @@ pub struct MoqConnectionStats { /// connection's version with another's numbers. #[uniffi(default = None)] pub version: Option, + /// Total datagrams received for a subscription with no datagram track: one the session + /// doesn't know, has ended, or whose answer it hasn't received. `None` on a version + /// without datagrams: moq-transport and moq-lite before 05. + #[uniffi(default = None)] + pub datagrams_unrouted: Option, } impl MoqConnectionStats { @@ -693,6 +698,7 @@ impl MoqConnectionStats { packets_received: stats.packets_received, packets_lost: stats.packets_lost, version: version.map(|version| version.to_string()), + datagrams_unrouted: stats.datagrams_unrouted, } } } diff --git a/rs/moq-net/src/client.rs b/rs/moq-net/src/client.rs index 14f107bcfb3..9f694213cf6 100644 --- a/rs/moq-net/src/client.rs +++ b/rs/moq-net/src/client.rs @@ -373,6 +373,7 @@ impl Client { bandwidth: self.allocator(), viewers, peer_implementation: start.peer_implementation, + datagrams_unrouted: start.datagrams_unrouted, }, )) } @@ -463,6 +464,7 @@ impl Client { bandwidth: self.allocator(), viewers: self.viewers(), peer_implementation, + datagrams_unrouted: None, }, ))); } @@ -564,6 +566,7 @@ impl Client { bandwidth: self.allocator(), viewers, peer_implementation: start.peer_implementation, + datagrams_unrouted: start.datagrams_unrouted, }, )) } diff --git a/rs/moq-net/src/lite/session.rs b/rs/moq-net/src/lite/session.rs index fc105f68d68..192568cffc8 100644 --- a/rs/moq-net/src/lite/session.rs +++ b/rs/moq-net/src/lite/session.rs @@ -29,6 +29,9 @@ pub(crate) struct SessionStart { pub credential: Option, /// The implementation the peer's SETUP advertised, stored on the public [`crate::Session`]. pub peer_implementation: crate::setup::PeerImplementation, + /// Counts the datagrams the subscriber could route to no datagram track, on a version + /// with datagrams. + pub datagrams_unrouted: Option>, } /// Server: reads the peer's single SETUP message off its Setup Stream before starting @@ -387,6 +390,7 @@ pub fn start(config: Config) -> Result { }); subscriber.announces = announces; subscriber.viewers = viewers; + let datagrams_unrouted = version.has_datagrams().then(|| subscriber.datagrams_unrouted.clone()); // SendSetup only logs a failure, so refuse a SETUP that cannot encode (one past the // size bound, such as a long implementation or path) here rather than leave the peer waiting. @@ -431,6 +435,7 @@ pub fn start(config: Config) -> Result { auth, credential: setup_token, peer_implementation, + datagrams_unrouted, }) } diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 7a7f906b16c..0ac030e46ae 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -80,6 +80,12 @@ pub(super) struct Subscriber { /// Lite-05/06: the datagram tracks taking the datagrams of their name's stream track /// subscription, by broadcast and name. One SUBSCRIBE carries both kinds there. legacy_datagrams: Lock>, + /// The datagram tracks ended, and on lite-05/06 the subscriptions carrying them, while the + /// driver routes the datagrams received with their end; see [`Self::end_datagrams`]. + datagram_ends: Lock, + /// Datagrams received for a subscription with no datagram track, which + /// [`crate::session::Stats::datagrams_unrouted`] reports. + pub(super) datagrams_unrouted: Arc, /// Why this session ended, once it has. A track still waiting on TRACK_INFO has no /// copy in [`Self::subscribes`], so dropping its request reads this instead of /// becoming [`Error::Dropped`]. @@ -151,6 +157,21 @@ struct DatagramEntry { timescale: Option, } +/// What the sources ended, applied once the driver read the datagrams queued with it. +#[derive(Default)] +struct DatagramEnds { + /// Lite-07: the datagram tracks the publisher finished, by Subscribe ID. + tracks: Vec<(u64, datagram::Producer)>, + /// Lite-05/06: the datagram tracks whose stream track finished, by broadcast and name. + shared: Vec<((Path, String), datagram::Producer)>, + /// Lite-05/06: the subscriptions finished, by Subscribe ID, with the broadcast and name + /// whose datagram track takes their datagrams. + subscribes: Vec<(u64, (Path, String))>, + /// Closed once these ends apply. Each serve that queued one keeps its stream until then, so + /// the peer's stream credit bounds what a datagram flood can leave queued. + applied: kio::Producer<()>, +} + struct TrackEntry { /// Where the subscription's groups go, once its TRACK_INFO says how to read them. A /// SUBSCRIBE sent alongside its TRACK registers before. @@ -414,6 +435,8 @@ impl Subscriber { subscribes: Default::default(), datagrams: Default::default(), legacy_datagrams: Default::default(), + datagram_ends: Default::default(), + datagrams_unrouted: Default::default(), ended: Default::default(), next_id: Default::default(), version: config.version, @@ -860,15 +883,47 @@ impl Subscriber { self.subscribes.lock().remove(&id); } - /// Decode one datagram body and hand it to the datagram track it was sent for. + /// Remove a subscription its publisher finished. On lite-05/06 it keeps routing its + /// datagrams to its name's datagram track, which may register meanwhile, until the ends + /// apply, which the returned consumer's close reports. + fn finish_subscribe(&self, id: u64) -> Option> { + let removed = self.subscribes.lock().remove(&id); + let legacy = removed.and_then(|mut entry| entry.legacy.take())?; + let mut ends = self.datagram_ends.lock(); + ends.subscribes.push((id, legacy)); + Some(ends.applied.consume()) + } + + /// Apply the datagram ends the sources read, once the driver routed the datagrams received + /// with them: it read the transport's queue empty. A publisher sends a track's last datagram + /// right before its end, and a fresh track's answer, first datagram and end can all share a + /// receive pass. A peer keeping the queue full delays its tracks' ends, holding nothing more. + fn end_datagrams(&self) { + let ends = std::mem::take(&mut *self.datagram_ends.lock()); + for (id, producer) in ends.tracks { + self.datagrams.lock().remove(&id); + producer.finish(); + } + for (key, producer) in ends.shared { + let mut shared = self.legacy_datagrams.lock(); + if shared.get(&key).is_some_and(|entry| entry.producer.is_clone(&producer)) { + shared.remove(&key); + } + drop(shared); + producer.finish(); + } + } + + /// Decode one datagram body and hand it to the datagram track it was sent for, or return + /// the Subscribe ID it names when that subscription has no datagram track: one unknown, + /// ended, or not yet answered. /// /// A datagram track's subscription takes its datagrams. On lite-05/06 a stream track's - /// subscription carries its name's datagrams too, which go to that name's datagram track - /// while one reads them, and are dropped otherwise, as are an unknown subscription's or - /// one not yet answered. - fn route_datagram(&self, payload: bytes::Bytes) -> Result<(), Error> { + /// subscription carries its name's datagrams too, which go to that name's datagram track. + /// Padding and a body past the wire cap name no subscription. + fn route_datagram(&self, payload: bytes::Bytes) -> Result, Error> { if payload.len() > lite::Datagram::MAX_SIZE { - return Ok(()); + return Ok(None); } let mut target = None; let Some(dg) = lite::Datagram::decode(payload, self.version, |id| { @@ -876,10 +931,10 @@ impl Subscriber { target.as_ref().is_some_and(|(_, timescale)| timescale.is_some()) })? else { - return Ok(()); + return Ok(None); }; let Some((mut producer, timescale)) = target else { - return Ok(()); + return Ok(Some(dg.subscribe)); }; let timestamp = dg .timestamp @@ -891,7 +946,8 @@ impl Subscriber { sequence: dg.sequence, timestamp, payload: dg.payload, - }) + })?; + Ok(None) } /// The datagram track Subscribe ID `id` delivers to, and its datagrams' timescale. @@ -899,7 +955,13 @@ impl Subscriber { if let Some(entry) = self.datagrams.lock().get(&id) { return Some((entry.producer.clone(), entry.timescale)); } - let key = self.subscribes.lock().get(&id)?.legacy.clone()?; + let key = match self.subscribes.lock().get(&id) { + Some(entry) => entry.legacy.clone()?, + None => { + let ends = self.datagram_ends.lock(); + ends.subscribes.iter().find(|(ended, _)| *ended == id)?.1.clone() + } + }; let legacy = self.legacy_datagrams.lock(); let entry = legacy.get(&key)?; Some((entry.producer.clone(), entry.timescale)) @@ -1014,6 +1076,7 @@ impl SubscriberDriver { let parked = self.subscriber.park_waits.load(atomic::Ordering::Relaxed); if closed && parked > 0 && self.uni.children.len() == parked { let _ = self.sources.poll(waiter); + self.subscriber.end_datagrams(); return Poll::Pending; } // Once every remaining stream waits on a SETUP that never arrived, none of @@ -1027,6 +1090,7 @@ impl SubscriberDriver { // Lite03/04 that FIN is the only end: read it, so the track finishes rather than // aborts. Each source reads its streams once, then retires on the closed session. let _ = self.sources.poll(waiter); + self.subscriber.end_datagrams(); Poll::Ready(()) } @@ -1076,12 +1140,15 @@ impl SubscriberDriver { Poll::Pending => {} } } - if let Some(datagrams) = &mut self.datagrams { - match datagrams.poll(waiter) { - Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), - Poll::Ready(Ok(())) => self.datagrams = None, - Poll::Pending => {} - } + + // Datagrams go around the sources, which decode the answers that register their + // routes and the ends that remove them. A publisher sends a track's first datagram + // right behind its answer and its last right before its end, often in the same + // packet. So the datagrams of known tracks go first, those with no known track wait + // for the sources' answers, and the ends the sources read apply after both, once the + // datagrams queued with them are read. + if let Poll::Ready(Err(err)) = self.poll_datagrams(waiter, true) { + return Poll::Ready(Err(err)); } // Sources minted by the announce half; their completion never ends the @@ -1091,6 +1158,29 @@ impl SubscriberDriver { } let _ = self.sources.poll(waiter); + let routed = self.poll_datagrams(waiter, false); + // A pass the budget cut short leaves datagrams queued, which may have come with an end: + // its ends wait for a later poll's pass to read the queue empty. + if self.datagrams.as_ref().is_none_or(|recv| recv.caught_up) { + self.subscriber.end_datagrams(); + } + if let Poll::Ready(Err(err)) = routed { + return Poll::Ready(Err(err)); + } + + Poll::Pending + } + + /// Route the ready datagrams, see [`DatagramRecv::poll`]; ready only with the error that + /// ends the session. + fn poll_datagrams(&mut self, waiter: &kio::Waiter, hold: bool) -> Poll> { + if let Some(datagrams) = &mut self.datagrams { + match datagrams.poll(waiter, hold) { + Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), + Poll::Ready(Ok(())) => self.datagrams = None, + Poll::Pending => {} + } + } Poll::Pending } } @@ -1734,6 +1824,12 @@ struct DatagramRecv { // A dedicated receive handle: the poll interface takes `&mut self`. recv: Transport, enabled: bool, + /// The datagrams with no known track before the sources ran, routed once they have. At + /// most one pass's budget, and empty again by the end of each driver poll. + held: Vec, + /// Whether the last pass read the transport's queue empty, or the receive ended, rather + /// than stopping at its budget. + caught_up: bool, } impl DatagramRecv { @@ -1744,21 +1840,57 @@ impl DatagramRecv { subscriber, recv, enabled, + held: Vec::new(), + caught_up: true, } } - fn poll(&mut self, waiter: &kio::Waiter) -> Poll> { + /// Route the ready datagrams. With `hold`, before the sources run, keep each with no known + /// track: its answer may be among what they decode. The pass after them, in the same + /// driver poll, routes those first, then the rest, dropping and counting each that still + /// finds no track. + fn poll(&mut self, waiter: &kio::Waiter, hold: bool) -> Poll> { if !self.enabled { return Poll::Ready(Ok(())); } - // A peer keeping datagrams ready must not hold the session's other tasks. + if !hold { + for payload in std::mem::take(&mut self.held) { + self.route(payload, false); + } + } + // A peer keeping datagrams ready must not hold the session's other tasks, and bounds + // what the first pass holds. let mut budget = kio::coop::Budget::new(32); loop { + self.caught_up = false; ready!(budget.poll_yield(waiter)); - let payload = ready!(self.recv.poll_recv_datagram(waiter)).map_err(crate::lite::error::from_transport)?; - if let Err(err) = self.subscriber.route_datagram(payload) { - tracing::debug!(%err, "dropping datagram"); + self.caught_up = true; + let payload = match ready!(self.recv.poll_recv_datagram(waiter)) { + Ok(payload) => payload, + Err(err) => { + // The session ends without the sources' answers: what is held finds no track. + for payload in std::mem::take(&mut self.held) { + self.route(payload, false); + } + return Poll::Ready(Err(crate::lite::error::from_transport(err))); + } + }; + self.route(payload, hold); + } + } + + /// Route one datagram, holding it with `hold` when its track isn't known yet. + fn route(&mut self, payload: bytes::Bytes, hold: bool) { + match self.subscriber.route_datagram(payload.clone()) { + Ok(None) => {} + Ok(Some(_)) if hold => self.held.push(payload), + Ok(Some(subscribe)) => { + self.subscriber + .datagrams_unrouted + .fetch_add(1, atomic::Ordering::Relaxed); + tracing::debug!(subscribe, "dropping datagram for a subscription with no datagram track"); } + Err(err) => tracing::debug!(%err, "dropping datagram"), } } } @@ -2554,6 +2686,12 @@ enum DatagramState { subscriber: Box, priority: i8, }, + /// The publisher ended the track: waiting for the end to apply, see + /// [`Subscriber::end_datagrams`]. Lite-07 keeps its stream until then. + Ending { + applied: kio::Consumer<()>, + writer: Option>, + }, /// Our side finished: waiting for the publisher to read it. Finish { writer: crate::coding::Writer, @@ -2617,13 +2755,21 @@ impl DatagramServe { } } + /// Yield to the driver once the route is registered, so it routes the datagrams it held + /// for this answer before the next pass may read the track's end, which removes the route. + /// The answer, the first datagram and the end can share a receive pass. + fn poll_routed(waiter: &kio::Waiter) -> Poll> { + waiter.waker().wake_by_ref(); + Poll::Pending + } + fn name(&self) -> &str { match &self.state { DatagramState::Open { request } | DatagramState::Info { request, .. } | DatagramState::Subscribe { request, .. } => request.name(), DatagramState::Live { producer, .. } | DatagramState::Shared { producer, .. } => producer.name(), - DatagramState::Finish { .. } | DatagramState::Done => "", + DatagramState::Ending { .. } | DatagramState::Finish { .. } | DatagramState::Done => "", } } @@ -2699,6 +2845,7 @@ impl DatagramServe { stream, priority, }; + return Self::poll_routed(waiter); } DatagramState::Subscribe { subscribing, .. } => { let subscriber = ready!(subscribing.poll_ok(waiter))?; @@ -2722,6 +2869,7 @@ impl DatagramServe { subscriber: Box::new(subscriber), priority, }; + return Self::poll_routed(waiter); } DatagramState::Live { producer, @@ -2736,20 +2884,23 @@ impl DatagramServe { None => None, }; if let Some(end) = ended { - self.subscriber.datagrams.lock().remove(&self.id); - let DatagramState::Live { - producer, mut stream, .. - } = std::mem::replace(&mut self.state, DatagramState::Done) + let DatagramState::Live { producer, stream, .. } = + std::mem::replace(&mut self.state, DatagramState::Done) else { unreachable!() }; - end.inspect_err(|err| producer.clone().abort(err.clone()))?; - producer.finish(); - if stream.writer.finish().is_ok() && version.waits_for_subscriber_fin() { - self.state = DatagramState::Finish { writer: stream.writer }; - continue; + if let Err(err) = end { + self.subscriber.datagrams.lock().remove(&self.id); + producer.abort(err.clone()); + return Poll::Ready(Err(err)); } - return Poll::Ready(Ok(())); + let mut ends = self.subscriber.datagram_ends.lock(); + ends.tracks.push((self.id, producer)); + self.state = DatagramState::Ending { + applied: ends.applied.consume(), + writer: Some(stream.writer), + }; + continue; } match producer.poll_priority(waiter, Some(*priority)) { Poll::Ready(Some(next)) => { @@ -2781,9 +2932,14 @@ impl DatagramServe { else { unreachable!() }; - self.unregister_shared(&producer); - producer.finish(); - return Poll::Ready(Ok(())); + let key = (self.path.clone(), producer.name().to_string()); + let mut ends = self.subscriber.datagram_ends.lock(); + ends.shared.push((key, producer)); + self.state = DatagramState::Ending { + applied: ends.applied.consume(), + writer: None, + }; + continue; } match producer.poll_priority(waiter, Some(*priority)) { Poll::Ready(Some(next)) => { @@ -2799,6 +2955,16 @@ impl DatagramServe { Poll::Pending => return Poll::Pending, } } + DatagramState::Ending { applied, writer } => { + ready!(applied.poll_closed(waiter)); + let Some(mut writer) = writer.take() else { + return Poll::Ready(Ok(())); + }; + if writer.finish().is_err() || !version.waits_for_subscriber_fin() { + return Poll::Ready(Ok(())); + } + self.state = DatagramState::Finish { writer }; + } DatagramState::Finish { writer } => { let _ = ready!(writer.poll_close(waiter)); return Poll::Ready(Ok(())); @@ -2828,9 +2994,10 @@ impl DatagramServe { } } - /// End the track and the request with `err`, unless it already ended. + /// End the track and the request with `err`, unless it already ended. An ended track's + /// route goes with its end instead, once the datagrams received with it routed: see + /// [`Subscriber::end_datagrams`]. fn abort(&mut self, err: Error) { - self.subscriber.datagrams.lock().remove(&self.id); match std::mem::replace(&mut self.state, DatagramState::Done) { DatagramState::Open { request } | DatagramState::Subscribe { request, .. } => request.reject(err), DatagramState::Info { request, stream, .. } => { @@ -2838,6 +3005,7 @@ impl DatagramServe { request.reject(err); } DatagramState::Live { producer, stream, .. } => { + self.subscriber.datagrams.lock().remove(&self.id); stream.writer.abort(&err); producer.abort(err); } @@ -2845,7 +3013,7 @@ impl DatagramServe { self.unregister_shared(&producer); producer.abort(err); } - DatagramState::Finish { .. } | DatagramState::Done => {} + DatagramState::Ending { .. } | DatagramState::Finish { .. } | DatagramState::Done => {} } } } @@ -2868,10 +3036,12 @@ impl kio::Task for DatagramServe { return Poll::Ready(()); } ready!(self.gate.poll_ready(waiter)); - // Nothing is left to revoke once our side finished, nor on a closed session, whose - // end decides how the track ends. - if !matches!(self.state, DatagramState::Finish { .. } | DatagramState::Done) - && self.gate.poll_denied(waiter).is_ready() + // Nothing is left to revoke once the track ended, nor on a closed session, whose end + // decides how the track ends. + if !matches!( + self.state, + DatagramState::Ending { .. } | DatagramState::Finish { .. } | DatagramState::Done + ) && self.gate.poll_denied(waiter).is_ready() && self.closed.poll_closed(waiter).is_pending() { tracing::info!(broadcast = %self.subscriber.log_path(&self.path), track = %self.name(), "datagrams no longer authorized"); @@ -3880,7 +4050,7 @@ mod tests { .unwrap(); } - assert!(recv.poll(&kio::Waiter::noop()).is_pending()); + assert!(recv.poll(&kio::Waiter::noop(), false).is_pending()); let other = local.clone(); assert!( other.poll_recv_datagram(&kio::Waiter::noop()).is_ready(), @@ -3888,6 +4058,139 @@ mod tests { ); } + /// Before the sources run, a datagram whose track isn't known yet is held rather than + /// dropped, while known tracks' datagrams go on, and the pass after them routes it to a + /// track registered in between. One that still finds no track is dropped and counted. + #[kio::test] + async fn a_datagram_waits_for_the_sources_before_it_is_dropped() { + use crate::transport::poll::Session as _; + + let (local, peer) = crate::transport::memory::pair(Default::default()); + let subscriber = Subscriber::new(SubscriberConfig { + runtime: crate::time::sim(), + session: crate::driver::Transport::new(local), + origin: origin::Config::new(crate::Hop::new(1).unwrap()).produce(), + recv_bandwidth: None, + version: Version::Lite07, + peer_setup: lite::PeerSetup::new(false, crate::transport::Address::Setup), + peer_hop: None, + cost: None, + going_away: Default::default(), + auth: crate::auth::Link::new(false), + }); + let mut recv = DatagramRecv::new(subscriber.clone()); + let send = |subscribe| { + let body = lite::Datagram { + subscribe, + sequence: 0, + timestamp: None, + payload: bytes::Bytes::from_static(b"x"), + }; + peer.send_datagram(body.encode_bytes(Version::Lite07).unwrap(), Default::default()) + .unwrap(); + }; + let unrouted = || subscriber.datagrams_unrouted.load(atomic::Ordering::Relaxed); + let waiter = kio::Waiter::noop(); + + send(1); + assert!(recv.poll(&waiter, true).is_pending()); + assert_eq!(recv.held.len(), 1, "dropped before the sources ran"); + let producer = datagram::Info::default().produce("datagrams"); + let mut received = producer.consume(); + subscriber.datagrams.lock().insert( + 1, + DatagramEntry { + producer, + timescale: None, + }, + ); + assert!(recv.poll(&waiter, false).is_pending()); + assert!(matches!(received.poll_recv(&waiter), Poll::Ready(Ok(Some(_))))); + assert_eq!(unrouted(), 0); + + // A held datagram doesn't hold up a known track's behind it, which an end the sources + // read could otherwise drop. + send(2); + send(1); + assert!(recv.poll(&waiter, true).is_pending()); + assert!(matches!(received.poll_recv(&waiter), Poll::Ready(Ok(Some(_))))); + assert_eq!(recv.held.len(), 1); + assert!(recv.poll(&waiter, false).is_pending()); + assert!(recv.held.is_empty()); + assert_eq!(unrouted(), 1); + } + + /// A track's end waits for the datagrams queued with it, though they outlast a driver + /// poll's budget, and applies once a pass reads the queue empty. + #[kio::test] + async fn an_end_waits_for_the_datagrams_queued_with_it() { + use crate::transport::poll::Session as _; + + const QUEUED: u64 = 100; + let (local, peer) = crate::transport::memory::pair(Default::default()); + let subscriber = Subscriber::new(SubscriberConfig { + runtime: crate::time::sim(), + session: crate::driver::Transport::new(local), + origin: origin::Config::new(crate::Hop::new(1).unwrap()).produce(), + recv_bandwidth: None, + version: Version::Lite07, + peer_setup: lite::PeerSetup::new(false, crate::transport::Address::Setup), + peer_hop: None, + cost: None, + going_away: Default::default(), + auth: crate::auth::Link::new(false), + }); + let mut driver = SubscriberDriver::new(subscriber.clone(), Vec::new(), false); + let producer = datagram::Info::default().produce("datagrams"); + let mut received = producer.consume(); + subscriber.datagrams.lock().insert( + 1, + DatagramEntry { + producer: producer.clone(), + timescale: None, + }, + ); + for sequence in 0..QUEUED { + let body = lite::Datagram { + subscribe: 1, + sequence, + timestamp: None, + payload: bytes::Bytes::from_static(b"x"), + }; + peer.send_datagram(body.encode_bytes(Version::Lite07).unwrap(), Default::default()) + .unwrap(); + } + // As the sources queue an end they read. + subscriber.datagram_ends.lock().tracks.push((1, producer)); + + let waiter = kio::Waiter::noop(); + let mut polls = 0; + let mut count = 0; + while !subscriber.datagram_ends.lock().tracks.is_empty() { + assert!(driver.poll(&waiter).is_pending()); + polls += 1; + assert!(polls < 16, "the end never applied"); + // The track keeps only so many unread. + while let Poll::Ready(Ok(Some(_))) = received.poll_recv(&waiter) { + count += 1; + } + } + assert!(polls > 1, "one poll read the whole queue"); + assert!(matches!(received.poll_recv(&waiter), Poll::Ready(Ok(None)))); + assert_eq!(count, QUEUED); + assert_eq!(subscriber.datagrams_unrouted.load(atomic::Ordering::Relaxed), 0); + } + + /// Poll `running` as the driver does, which applies the datagram ends queued in its pass. + fn poll_run(running: &mut TrackServeRun, waiter: &kio::Waiter) -> Poll<()> { + let res = kio::Task::poll(running, waiter); + running.serve.subscriber.end_datagrams(); + match running.state { + TrackRunState::Ending { .. } => kio::Task::poll(running, waiter), + _ => res, + } + } + #[kio::test] async fn padding_is_discarded_only_on_lite07() { for version in [ @@ -4584,7 +4887,7 @@ mod tests { gate, datagrams: None, }; - kio::wait(|waiter| kio::Task::poll(&mut running, waiter)).await; + kio::wait(|waiter| poll_run(&mut running, waiter)).await; let end = loop { match reader.recv_group().await { @@ -4898,7 +5201,7 @@ mod tests { datagrams: None, }; assert!( - kio::Task::poll(&mut running, &kio::Waiter::noop()).is_ready(), + poll_run(&mut running, &kio::Waiter::noop()).is_ready(), "{version:?}: FIN must settle immediately" ); if clean { @@ -5013,11 +5316,11 @@ mod tests { datagrams: None, }; assert!( - kio::Task::poll(&mut running, &kio::Waiter::noop()).is_pending(), + poll_run(&mut running, &kio::Waiter::noop()).is_pending(), "{version:?}: the tail still owes group 0" ); crate::transport::poll::Session::close(&session, crate::SessionError::App(7).to_code(), "died"); - assert!(kio::Task::poll(&mut running, &kio::Waiter::noop()).is_ready()); + assert!(poll_run(&mut running, &kio::Waiter::noop()).is_ready()); let res = reader.recv_group().await.err(); assert!(matches!(res, Some(Error::Session(_))), "{version:?}: {res:?}"); } @@ -5168,7 +5471,7 @@ mod tests { if version.has_padding() { // These bytes would name the active subscription without the discriminator. for body in [vec![1], vec![1; 1200], vec![1; 1201]] { - subscriber.route_datagram(body.into()).unwrap(); + assert_eq!(subscriber.route_datagram(body.into()).unwrap(), None); assert!(recv().is_pending()); } } @@ -5183,7 +5486,7 @@ mod tests { .encode_bytes(version) .unwrap() }; - subscriber.route_datagram(payload(1, 1)).unwrap(); + assert_eq!(subscriber.route_datagram(payload(1, 1)).unwrap(), None); let Poll::Ready(Ok(Some(datagram))) = recv() else { panic!("{version}: the datagram was not delivered"); }; @@ -5193,17 +5496,17 @@ mod tests { let max_payload = 1200 - payload(2, 0).len(); let oversized = payload(2, max_payload + 1); assert_eq!(oversized.len(), 1201); - subscriber.route_datagram(oversized).unwrap(); + assert_eq!(subscriber.route_datagram(oversized).unwrap(), None); assert!(recv().is_pending(), "oversized body was delivered"); let boundary = payload(3, max_payload); assert_eq!(boundary.len(), 1200); - subscriber.route_datagram(boundary).unwrap(); + assert_eq!(subscriber.route_datagram(boundary).unwrap(), None); assert!(matches!(recv(), Poll::Ready(Ok(Some(datagram))) if datagram.sequence == 3)); - // An unknown subscription's datagram goes nowhere. + // An unknown subscription's datagram goes nowhere, and names it. let mut other = payload(4, 1).to_vec(); other[usize::from(version.has_padding())] = 2; - subscriber.route_datagram(other.into()).unwrap(); + assert_eq!(subscriber.route_datagram(other.into()).unwrap(), Some(2)); assert!(recv().is_pending()); } } @@ -6352,7 +6655,7 @@ mod tests { gate, datagrams: None, }; - assert!(kio::Task::poll(&mut running, &kio::Waiter::noop()).is_ready()); + assert!(poll_run(&mut running, &kio::Waiter::noop()).is_ready()); let end = reader.recv_group().now_or_never().expect("the track ended").map(|_| ()); match reset { false => assert!(matches!(end, Ok(())), "{end:?}"), @@ -9297,11 +9600,20 @@ impl SharedDatagrams { } impl SharedDatagrams { - /// Stop routing to the track and end it with `end`. + /// Stop routing to the track and end it with `end`. A clean end waits for the datagrams + /// received with it, see [`Subscriber::end_datagrams`]. fn end(&mut self, end: Result<(), Error>) { let Some(producer) = self.producer.take() else { return; }; + let Err(err) = end else { + self.subscriber + .datagram_ends + .lock() + .shared + .push((self.key.clone(), producer)); + return; + }; let mut shared = self.subscriber.legacy_datagrams.lock(); if shared .get(&self.key) @@ -9310,10 +9622,7 @@ impl SharedDatagrams { shared.remove(&self.key); } drop(shared); - match end { - Ok(()) => producer.finish(), - Err(err) => producer.abort(err), - } + producer.abort(err); } } @@ -9346,6 +9655,12 @@ enum TrackRunState { _working: crate::session::Working, }, Serve(ServeLoop), + /// Lite-05/06: the publisher finished the track, whose subscription keeps its stream until + /// the datagram ends apply, see [`Subscriber::end_datagrams`]. + Ending { + applied: kio::Consumer<()>, + active: SubStream, + }, /// Preserve the completion FIN until the publisher acknowledges it. Lite-07 also reads /// on to the publisher's FIN in reply, so dropping our receive half never refuses it. Finish { @@ -9424,12 +9739,26 @@ impl TrackServeRun { } // Each fetch in flight resets itself as it drops, seeing the same denial. } - TrackRunState::Finish { .. } | TrackRunState::Done => {} + TrackRunState::Ending { .. } | TrackRunState::Finish { .. } | TrackRunState::Done => {} } } } impl TrackServeRun { + /// Finish our side of the subscription, `true` once it waits for the publisher to read it. + fn finish_sub(&mut self, mut active: SubStream, finished: bool) -> bool { + let version = self.serve.subscriber.version; + if active.stream.writer.finish().is_err() || !finished || !version.waits_for_subscriber_fin() { + return false; + } + self.state = TrackRunState::Finish { + writer: Some(active.stream.writer), + reader: version.keeps_subscribe_open().then_some(active.stream.reader), + _working: active._working, + }; + true + } + /// Nobody wants the track anymore (the origin failed over, the reader left): stop /// waiting on a token or a peer that may never answer, which would otherwise hold /// this task, its TRACK stream, and the track for good. Dropping the fetch resets the @@ -9470,7 +9799,7 @@ impl kio::Task for TrackServeRun { // A finished track only waits for its FIN to be acknowledged: nothing is left to revoke. // Nor on a closed session, whose grants end with it: the session's end decides how // its tracks end. - if !matches!(self.state, TrackRunState::Finish { .. }) + if !matches!(self.state, TrackRunState::Ending { .. } | TrackRunState::Finish { .. }) && self.gate.poll_denied(waiter).is_ready() && self.serve.subscriber.session.clone().poll_closed(waiter).is_pending() { @@ -9610,23 +9939,37 @@ impl kio::Task for TrackServeRun { datagrams.end(Err(Error::Cancel)); } - if let Sub::Active(mut active) = serve_loop.sub { - self.serve.subscriber.remove_subscribe(active.id); - if active.stream.writer.finish().is_ok() - && finished && self.serve.subscriber.version.waits_for_subscriber_fin() - { - let keeps_open = self.serve.subscriber.version.keeps_subscribe_open(); - self.state = TrackRunState::Finish { - writer: Some(active.stream.writer), - reader: keeps_open.then_some(active.stream.reader), - _working: active._working, - }; + if let Sub::Active(active) = serve_loop.sub { + let applied = match finished { + true => self.serve.subscriber.finish_subscribe(active.id), + false => { + self.serve.subscriber.remove_subscribe(active.id); + None + } + }; + // The subscription carries its name's datagrams, which end with it. + if let Some(applied) = applied { + self.state = TrackRunState::Ending { applied, active }; + continue; + } + if self.finish_sub(active, finished) { continue; } } return Poll::Ready(()); } + TrackRunState::Ending { applied, .. } => { + ready!(applied.poll_closed(waiter)); + let TrackRunState::Ending { active, .. } = std::mem::replace(&mut self.state, TrackRunState::Done) + else { + unreachable!() + }; + if self.finish_sub(active, true) { + continue; + } + return Poll::Ready(()); + } TrackRunState::Finish { writer, reader, .. } => { // Responses after the tail settled change nothing: read past them, yielding // to the session's other tasks while a peer streams them. diff --git a/rs/moq-net/src/server.rs b/rs/moq-net/src/server.rs index 07343c53954..fea9d832b0a 100644 --- a/rs/moq-net/src/server.rs +++ b/rs/moq-net/src/server.rs @@ -240,6 +240,7 @@ impl Server { bandwidth: crate::bandwidth::Allocator::default(), viewers, peer_implementation: start.peer_implementation, + datagrams_unrouted: start.datagrams_unrouted, }, )) } @@ -805,6 +806,7 @@ impl PausedIetf { bandwidth: crate::bandwidth::Allocator::default(), viewers: Default::default(), peer_implementation, + datagrams_unrouted: None, }, )) } @@ -935,6 +937,7 @@ impl NegotiatedStart { bandwidth: crate::bandwidth::Allocator::default(), viewers, peer_implementation: start.peer_implementation, + datagrams_unrouted: start.datagrams_unrouted, }, )) } diff --git a/rs/moq-net/src/session.rs b/rs/moq-net/src/session.rs index b5042245e90..d86c9f46f7d 100644 --- a/rs/moq-net/src/session.rs +++ b/rs/moq-net/src/session.rs @@ -1,7 +1,10 @@ //! A MoQ session handle and a snapshot of its connection statistics. use std::{ - sync::Arc, + sync::{ + Arc, + atomic::{AtomicU64, Ordering}, + }, task::{Poll, ready}, time::Duration, }; @@ -127,6 +130,11 @@ pub struct Stats { /// Total receive streams evicted because incomplete messages exhausted the receive window. /// A count above zero means the receive window is too small for what the peer sends. pub streams_evicted: Option, + + /// Total datagrams received for a subscription with no datagram track: one this session + /// doesn't know, has ended, or whose answer it hasn't received. `None` on a version without + /// datagrams: moq-transport and moq-lite before 05. + pub datagrams_unrouted: Option, } /// The congestion controller's view of the connection, as PROBE reports it. @@ -194,6 +202,8 @@ pub struct Session { credential: Option, /// The subscriptions and fetches the protocol tasks carry. work: Work, + /// What [`Stats::datagrams_unrouted`] reads. + datagrams_unrouted: Option>, } impl Session { @@ -275,8 +285,8 @@ impl Session { /// /// Cheap and non-blocking: this reads the latest sample the session's /// driver took, and schedules a refresh, so periodic polling observes - /// fresh counters (100ms cadence). See [`Stats`] for which - /// metrics each backend reports. + /// fresh counters (100ms cadence), and reads the session's own counters + /// live. See [`Stats`] for which metrics each backend reports. pub fn stats(&self) -> Stats { let mut state = self.stats.lock(); // A read is demand: wake the sampler, but only mutate (and so wake) @@ -284,7 +294,13 @@ impl Session { if !state.demanded { state.demanded = true; } - state.sample + Stats { + datagrams_unrouted: self + .datagrams_unrouted + .as_ref() + .map(|count| count.load(Ordering::Relaxed)), + ..state.sample + } } /// Close the transport with an explicit error, instead of waiting for the last @@ -440,6 +456,9 @@ pub(super) struct Handles { pub peer_target: Option>, /// What the peer's SETUP advertised as its implementation. pub peer_implementation: crate::setup::PeerImplementation, + /// The driver's count of datagrams it routed to no datagram track, for a version that + /// carries datagrams. + pub datagrams_unrouted: Option>, } impl Session { @@ -461,6 +480,7 @@ impl Session { viewers, peer_target, peer_implementation, + datagrams_unrouted, } = handles; let goaway = Arc::new(goaway); auth.drain_with(goaway.clone()); @@ -534,6 +554,7 @@ impl Session { auth: auth.handle().clone(), credential, work, + datagrams_unrouted, }; let driver = crate::Driver::new( runtime.clone(), @@ -934,6 +955,8 @@ fn snapshot(session: &S) -> Stats { packets_received: metrics.packets_received(), packets_lost: metrics.packets_lost(), streams_evicted: metrics.streams_evicted(), + // Counted by the protocol, not the transport: filled in by `Session::stats`. + datagrams_unrouted: None, } } diff --git a/rs/moq-net/tests/integration/datagram.rs b/rs/moq-net/tests/integration/datagram.rs index b13ca495a41..5d5451f68e6 100644 --- a/rs/moq-net/tests/integration/datagram.rs +++ b/rs/moq-net/tests/integration/datagram.rs @@ -566,3 +566,151 @@ async fn moq_transport_has_no_datagram_tracks() { b"frame" ); } + +/// A publisher that answers a datagram request and writes at once sends the datagram right +/// behind its answer, so both reach the subscriber in one receive pass, with the track's end +/// too when it finishes at once. The subscriber decodes the answer before it routes the +/// datagram, and routes the datagram before it ends the track. +#[kio::test] +async fn a_datagram_sent_right_behind_its_answer_arrives() { + for version in ["moq-lite-05", "moq-lite-06", "moq-lite-07-wip"] { + for finish in [false, true] { + let chain = Chain::new(&[version]).await; + let requests = chain.publication.broadcast().requests(); + let mut handler = requests.datagrams(); + let mut consumer = chain.remote().await.datagrams(TRACK).unwrap(); + let request = kio::clock::timeout(TIMEOUT, handler.requested()) + .await + .unwrap_or_else(|_| panic!("{version}: the request never reached the handler")) + .unwrap(); + let mut producer = request.accept(timed()); + assert_eq!(producer.append(millis(1), &b"first"[..]).unwrap(), 0, "{version}"); + if finish { + producer.finish(); + } + let datagram = recv(&mut consumer) + .await + .unwrap_or_else(|err| panic!("{version} finish={finish}: {err:?}")) + .unwrap_or_else(|| panic!("{version} finish={finish}: the datagram was dropped")); + assert_eq!(datagram.sequence, 0, "{version}"); + assert_eq!(&datagram.payload[..], b"first", "{version}"); + if finish { + assert!(recv(&mut consumer).await.unwrap().is_none(), "{version}"); + } + // The second node subscribes through the session it accepted. + assert_eq!( + chain.pairs[0].server.stats().datagrams_unrouted, + Some(0), + "{version} finish={finish}" + ); + } + } +} + +/// A publisher that writes a last datagram and finishes at once sends the datagram right +/// ahead of its track's end, so both reach the subscriber in one receive pass. The +/// subscriber routes the datagram before it ends the track. +#[kio::test] +async fn a_datagram_sent_right_before_its_end_arrives() { + for version in ["moq-lite-05", "moq-lite-06", "moq-lite-07-wip"] { + let chain = Chain::new(&[version]).await; + let mut producer = chain.publication.broadcast().create_datagrams(TRACK, timed()).unwrap(); + let mut consumer = chain.consume(&producer, 1).await; + producer.append(millis(1), &b"last"[..]).unwrap(); + producer.finish(); + let datagram = kio::clock::timeout(TIMEOUT, consumer.recv()) + .await + .unwrap_or_else(|_| panic!("{version}: nothing arrived")) + .unwrap_or_else(|err| panic!("{version}: {err:?}")) + .unwrap_or_else(|| panic!("{version}: the datagram was dropped")); + assert_eq!(&datagram.payload[..], b"last", "{version}"); + assert!(recv(&mut consumer).await.unwrap().is_none(), "{version}"); + assert_eq!(chain.pairs[0].server.stats().datagrams_unrouted, Some(0), "{version}"); + } +} + +/// Lite-05/06: a stream track's subscription carries its name's datagrams, and a publisher +/// answering the track, writing a datagram and finishing at once sends all three in one +/// receive pass. The subscriber routes the datagram to the name's datagram track before the +/// stream track's end removes it. +#[kio::test] +async fn a_stream_tracks_datagram_sent_between_its_answer_and_end_arrives() { + for version in ["moq-lite-05", "moq-lite-06"] { + let chain = Chain::new(&[version]).await; + let mut requests = chain.publication.broadcast().requests(); + let mut handler = requests.datagrams(); + let remote = chain.remote().await; + let mut subscribing = kio::clock::sim::spawn(async move { remote.track(TRACK).unwrap().subscribe(None).await }); + let stream = kio::clock::timeout(TIMEOUT, requests.requested_track()) + .await + .unwrap_or_else(|_| panic!("{version}: the stream lookup never reached the publisher")) + .unwrap(); + let datagrams = kio::clock::timeout(TIMEOUT, handler.requested()) + .await + .unwrap_or_else(|_| panic!("{version}: the datagram handler was never asked")) + .unwrap(); + let track = stream.accept(moq_net::track::Info::default().with_timescale(Timescale::MILLI)); + let mut producer = datagrams.accept(timed()); + producer.append(millis(1), &b"only"[..]).unwrap(); + producer.finish(); + track.finish().unwrap(); + let mut subscriber = (&mut subscribing) + .await + .expect("subscribe panicked") + .unwrap_or_else(|err| panic!("{version}: {err:?}")); + assert!( + kio::clock::timeout(TIMEOUT, subscriber.recv_group()) + .await + .expect("end timeout") + .unwrap() + .is_none(), + "{version}" + ); + assert_eq!(chain.pairs[0].server.stats().datagrams_unrouted, Some(0), "{version}"); + } +} + +/// A track's end waits for the subscriber to read the datagrams queued with it, so a flood +/// that keeps its receive queue full delays the end. Meanwhile the track keeps its stream, and +/// the session stays busy, so the peer's stream credit bounds the ends a flood can leave queued. +#[kio::test] +async fn a_track_ended_under_a_datagram_flood_keeps_its_stream() { + use moq_net::transport::poll::Session as _; + + for version in ["moq-lite-05", "moq-lite-06", "moq-lite-07-wip"] { + let chain = Chain::new(&[version]).await; + let producer = chain.publication.broadcast().create_datagrams(TRACK, timed()).unwrap(); + let mut consumer = chain.consume(&producer, 1).await; + // Each names no subscription, so the subscriber reads and drops it. The queue holds 1024. + let flood = || { + for _ in 0..512 { + let _ = chain.pairs[0] + .client_transport + .send_datagram(b"x"[..].into(), Default::default()); + } + }; + flood(); + producer.finish(); + for _ in 0..32 { + flood(); + // One turn for the session's tasks, which read at most a budget each. + let mut turned = false; + kio::wait(|waiter| { + if std::mem::replace(&mut turned, true) { + return std::task::Poll::Ready(()); + } + waiter.waker().wake_by_ref(); + std::task::Poll::Pending + }) + .await; + assert!( + chain.pairs[0].server.poll_idle(&kio::Waiter::noop()).is_pending(), + "{version}: the track released its stream before its end applied" + ); + } + assert!(recv(&mut consumer).await.unwrap().is_none(), "{version}"); + kio::clock::timeout(TIMEOUT, chain.pairs[0].server.idle()) + .await + .unwrap_or_else(|_| panic!("{version}: the track never released its stream")); + } +}