diff --git a/cpp/moq/README.md b/cpp/moq/README.md index b2449d10704..ca184c07690 100644 --- a/cpp/moq/README.md +++ b/cpp/moq/README.md @@ -30,7 +30,7 @@ Every generated type lives in `namespace moq`. Objects are `std::shared_ptr`; re - A fallible call returns `moq::expected`, which is `std::expected` on C++23 and a bundled `tl::expected` below it. The generated sources and every includer must agree on which, so `moq/moq.hpp` references a symbol named for the one it sees and `src/moq.cpp` defines the one it was built with: a mismatch fails to link. - An async call returns `moq::Future`. Block on it with `get()` or `wait_for()`, attach a continuation with `std::move(future).then(executor, callback)`, which returns a `moq::Continuation`, or `co_await` it on C++20. -- `moq::Error` is a value holding a `std::variant` of its cases, and `to_string()` gives Rust's `Display` message. Nothing throws. A Rust panic inside a call aborts with a message on stderr, and so does reading an invalid future (see [Cancellation](#cancellation)). A panic in a session's background work closes only that session. +- `moq::Error` is a value holding a `std::variant` of its cases, and `to_string()` gives Rust's `Display` message. Nothing throws. A Rust panic inside a call aborts with a message on stderr, and so does reading an invalid future (see [Cancellation](#cancellation)). A panic in owned background work reaches the next call on that object as a task error; other objects keep running. - Futures are polled on one process-wide executor thread unless the application installs its own with `moq::set_executor` before the first async call. The executor passed to `then()` dispatches only the continuation. `moq::shutdown()` stops the moq-ffi runtime workers (`moq_ffi_shutdown`), then the executor, before unloading the code a continuation could call into. ## C++ standard diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index 81ce7c8ce33..7056c329296 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -57,7 +57,7 @@ here, from media publish and consume to raw pixels and PCM with the codec inside. The header is the reference: each function carries a doc comment. - **Handles and callbacks.** Every object is an integer handle, and every async result arrives on a `moq_status_callback` with your `void *user_data`. A status `> 0` is a live result, `0` a clean close, and `< 0` an error. The last two are terminal: moq-c never touches `user_data` again, so free it there. `*_close` and `*_cancel` only request shutdown; the terminal callback still fires. -- **Errors.** Negative return codes are named by `MOQ_ERROR_*`, and `moq_error()` gives the reason on the calling thread. Auth rejections have their own codes so you don't retry them. For the peer's protocol code use `moq_error_protocol()`; do not parse `moq_error()`. A request-cap closure has kind `MOQ_PROTOCOL_KIND_TOO_MANY_REQUESTS`; reduce the workload or adjust the relay's [session limits](/bin/relay/config#session-and-peer) before reconnecting. +- **Errors.** Negative return codes are named by `MOQ_ERROR_*`, and `moq_error()` gives the reason on the calling thread. Auth rejections have their own codes so you don't retry them. For the peer's protocol code use `moq_error_protocol()`; do not parse `moq_error()`. A panic in a binding background task reports `MOQ_ERROR_PANIC` through its terminal callback; unrelated objects keep running. A request-cap closure has kind `MOQ_PROTOCOL_KIND_TOO_MANY_REQUESTS`; reduce the workload or adjust the relay's [session limits](/bin/relay/config#session-and-peer) before reconnecting. - **Threading.** Call any function from any thread. Raw publish calls block until the codec takes the frame, which paces a publisher. - **Zeroed config is the default.** A zeroed `moq_client_config` means the default for every field, so new fields don't break callers. A field whose default isn't zero has a `has_*` flag: `backoff_timeout_us = 0` means "retry forever" only with `has_backoff_timeout = true`. - **Audio needs cuts.** Video groups at its keyframes, but audio forms a group only where you call `moq_publish_media_cut`: after every frame, or at a segment cadence to align with video. diff --git a/doc/lib/cpp/index.md b/doc/lib/cpp/index.md index be4e51991fd..2b5e1beae87 100644 --- a/doc/lib/cpp/index.md +++ b/doc/lib/cpp/index.md @@ -215,8 +215,8 @@ Error cases carrying a diagnostic expose it as `reason`; protocol failures expos rejections are their own cases, so you don't retry them. `error.to_string()` gives the message Rust's `Display` does, for logs; branch on the case, not the text. A Rust panic inside a call, or a misused future (`get()` twice), aborts -with a message on stderr instead of throwing. A panic in a session's background -work closes only that session. +with a message on stderr instead of throwing. A panic in owned background work reaches the next call on that object as a task error; +other objects keep running. A session closed for exceeding the relay's request caps reports the protocol kind `ProtocolKind::kTooManyRequests`. Reduce diff --git a/doc/lib/dart/index.md b/doc/lib/dart/index.md index 5f755bc089d..af71e8b95cd 100644 --- a/doc/lib/dart/index.md +++ b/doc/lib/dart/index.md @@ -98,6 +98,8 @@ Native I/O and background tasks run on a Tokio worker pool. Calls awaited by the host still poll in place; independent native wake callbacks can run concurrently. The generated bindings deliver native wakeups to the Dart isolate. Keep UI work on its owning isolate. +A panic in a native background task reaches the pending and later fallible calls +of every object that depends on it as a task error; other objects keep running. A session closed for exceeding the relay's request caps reports the protocol kind `ProtocolKind.tooManyRequests`. Reduce diff --git a/doc/lib/go/index.md b/doc/lib/go/index.md index 5d73a98a0ab..29dbe007763 100644 --- a/doc/lib/go/index.md +++ b/doc/lib/go/index.md @@ -129,6 +129,8 @@ Native I/O and background tasks run on a Tokio worker pool. Calls awaited by the host still poll in place; independent native wake callbacks can run concurrently. The generated bridge synchronizes wakeups before resuming the awaiting goroutine. Application goroutines remain responsible for serializing their own shared state. +A panic in a native background task reaches the pending and later fallible calls +of every object that depends on it as a task error; other objects keep running. A session closed for exceeding the relay's request caps reports the protocol kind `ProtocolKindTooManyRequests`. Reduce diff --git a/doc/lib/kt/index.md b/doc/lib/kt/index.md index 01546c6de3f..906fa968735 100644 --- a/doc/lib/kt/index.md +++ b/doc/lib/kt/index.md @@ -103,6 +103,8 @@ Native I/O and background tasks run on a Tokio worker pool. Calls awaited by the host still poll in place; independent native wake callbacks can run concurrently. The generated bindings resume Kotlin coroutines through their dispatcher. Use the UI dispatcher when updating UI state. +A panic in a native background task reaches the pending and later fallible calls +of every object that depends on it as a task error; other objects keep running. A session closed for exceeding the relay's request caps reports the protocol kind `ProtocolKind.TOO_MANY_REQUESTS`. Reduce diff --git a/doc/lib/py/index.md b/doc/lib/py/index.md index ad63a2e9856..17d8181b4eb 100644 --- a/doc/lib/py/index.md +++ b/doc/lib/py/index.md @@ -120,6 +120,8 @@ Native I/O and background tasks run on a Tokio worker pool. Calls awaited by the host still poll in place; independent native wake callbacks can run concurrently. The generated bindings wake Python futures through the asyncio event loop. The package exit hook joins native workers before interpreter finalization. +A panic in a native background task reaches the pending and later fallible calls +of every object that depends on it as a task error; other objects keep running. A session closed for exceeding the relay's request caps reports the protocol kind `ProtocolKind.TOO_MANY_REQUESTS`. Reduce diff --git a/doc/lib/swift/index.md b/doc/lib/swift/index.md index a3bcd5a4ffe..b3ac873ba01 100644 --- a/doc/lib/swift/index.md +++ b/doc/lib/swift/index.md @@ -103,6 +103,8 @@ Native I/O and background tasks run on a Tokio worker pool. Calls awaited by the host still poll in place; independent native wake callbacks can run concurrently. The generated bindings resume Swift continuations. Use the appropriate actor when updating UI state. +A panic in a native background task reaches the pending and later fallible calls +of every object that depends on it as a task error; other objects keep running. A session closed for exceeding the relay's request caps reports the protocol kind `ProtocolKind.tooManyRequests`. Reduce diff --git a/doc/setup/upgrade.md b/doc/setup/upgrade.md index dd5c605fdc7..95a03667171 100644 --- a/doc/setup/upgrade.md +++ b/doc/setup/upgrade.md @@ -39,6 +39,13 @@ These land with the next breaking release, not the 2026-09-23 train. - **`cache::Pool::gc` takes a waiter.** In Rust, `gc(now)` is `gc(now, waiter)`, like `Driver::poll`: call it again when the waiter wakes, since a parked read can arm an earlier deadline than the one returned. +- **Native background panics.** UniFFI reports `MoqError::Task` with a panic + reason, and C reports `MOQ_ERROR_PANIC`. Other objects keep running. Browser + WebAssembly still aborts on panic with the current toolchain. +- **Accepted sessions return their driver task.** `moq_tokio::server::Request::ok()` + now returns `(session, driver)`. Await the supervised driver before classifying + the session close, so a driver panic reports `moq_tokio::Error::Panic` rather + than the cancellation its teardown leaves on a channel. - **A QUIC load-balancer nonce needs its server id.** `listen.lb_nonce`, `--listen-quic-lb-nonce`, and `MOQ_LISTEN_QUIC_LB_NONCE` without `lb_id` stop startup. Setting `lb_id` (or `lb_nonce`) and `load_balancer` together also diff --git a/go/wrapper/errors.go b/go/wrapper/errors.go index 6c0e0fa64c3..71e64415a92 100644 --- a/go/wrapper/errors.go +++ b/go/wrapper/errors.go @@ -45,7 +45,7 @@ var ( ErrTimeOverflow = ffi.ErrMoqErrorTimeOverflow // ErrLogLevel matches an unparseable log level string passed to LogLevel. ErrLogLevel = ffi.ErrMoqErrorLogLevel - // ErrTask matches a panic or cancellation in a background native task. + // ErrTask matches a panic in an owned background native task; cancellation is ErrCancelled. ErrTask = ffi.ErrMoqErrorTask // ErrJSON matches malformed JSON passed through the FFI API. ErrJSON = ffi.ErrMoqErrorJson diff --git a/quest/m1/README.md b/quest/m1/README.md index 18f6a258a58..82bd2bbbb87 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -85,7 +85,6 @@ above them. - [Live largest location](/quest/m1/largest.md) - `largest()` replaces `latest()` in Rust and JS as a live stream of the track's largest location, known before the first group from the publisher's report - [Live TS clock validation](/quest/m1/ts-live-clock.md) - resolve the local PCR-rate failure with reproducible timing evidence and CI controls - [TS export start-up drift](/quest/m1/ts-drift-startup.md) - a TS export never exits on start-up latency from a source within tolerance, while a far-off source still fails within seconds (#5564) -- [Binding panic supervision](/quest/m1/ffi-panic-supervision.md) - a panic in any moq-c or moq-ffi background task reaches the host as a panic, never a lost callback or Cancel - [Kotlin destroyed arguments](/quest/m1/kt-destroyed-arg.md) - a destroyed object passed to a generated Kotlin call throws instead of aborting the process - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause - [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone diff --git a/quest/m1/ffi-panic-supervision.md b/quest/m1/ffi-panic-supervision.md deleted file mode 100644 index cc43c738c1d..00000000000 --- a/quest/m1/ffi-panic-supervision.md +++ /dev/null @@ -1,44 +0,0 @@ -# [M] A panic in a binding's background task reaches the host - -## Goal - -A panic in any task moq-c or moq-ffi spawns (consume, publish, origin, audio, -video, server, and `ffi::spawn`'s tasks, including the origin driver) -reaches the host as a panic error through that object's final callback or -the uniffi call, never as a lost callback or `Cancel`. A moq-tokio driver -panic under a binding reports `Panic`, not `Cancel`. - -## Plan - -Follow-up from [#5447](https://github.com/moq-dev/moq/pull/5447), which made -a panic close only its session and supervised moq-c's session tasks. The -other moq-c tasks still drop a panic, so the host's final callback never -fires and a C host waits forever; moq-ffi's `ffi::spawn` tasks do the same -for uniffi callers. - -Decided 2026-10-13 (maintainer): m1, one quest for both bindings. Supervise -each task's `JoinHandle` the way #5447 did for moq-c sessions, reusing -`moq_tokio::panic`. Carry a driver's panic through as a panic rather than -the `Cancel` its dropped channel produces today. Add a regression per -binding that injects a panic into one background task and asserts the -host sees `MOQ_ERROR_PANIC` (moq-c) or the binding's panic error (moq-ffi), -while another object keeps working. A binding's origin task needs its own owner for -the driver, so its panic can be reported; wrapping the helper that aborts the -process (`moq_tokio::origin::spawn`) is not enough. Also add the regression #5447 left out: -a moq-tokio accept loop that panics ends its listener. - -Folded in from #5447's review (maintainer, 2026-10-13): when the driver of a -session a server accepted panics, moq-c sees an ordinary close error, not -`MOQ_ERROR_PANIC`. Have `Request::ok` (`rs/moq-tokio/src/server.rs`) hand -the driver back to the caller, so the binding observes its `JoinHandle` and -reports `moq_tokio::Error::Panic` like the client's reconnect loop does. - -Public API: `Request::ok`'s return changes in moq-tokio; otherwise the -panic error reaching more callbacks. -That is a host-visible change even without a new error variant, so walk the -`rs/moq-ffi` row of the Cross-Package Sync table: every wrapper and -`doc/lib` page surfaces the panic error. Wire: none. - -## Related - -- [Long-lived task supervision](/quest/m2/task-supervision.md) - the remaining process-wide tasks diff --git a/quest/m2/task-supervision.md b/quest/m2/task-supervision.md index 773df4ad8e4..c1ed93a0ee9 100644 --- a/quest/m2/task-supervision.md +++ b/quest/m2/task-supervision.md @@ -26,7 +26,3 @@ relay and the CLI already do. Public API: `Listener::accept`'s signature in moq-tokio (and `Server::serve_*`'s result). Wire: none. - -## Related - -- [Binding panic supervision](/quest/m1/ffi-panic-supervision.md) - the moq-c and moq-ffi tasks, ahead of this diff --git a/rs/moq-c/src/audio.rs b/rs/moq-c/src/audio.rs index f13b3991ba0..be2ef17ba0c 100644 --- a/rs/moq-c/src/audio.rs +++ b/rs/moq-c/src/audio.rs @@ -210,15 +210,16 @@ impl Audio { pub fn consume( &mut self, - broadcast: &moq_net::broadcast::Consumer, - catalog: &hang::catalog::AudioConfig, - name: &str, + rendition: crate::consume::Rendition, config: moq_audio::decode::Options, on_frame: OnStatus, ) -> Result { - let broadcast = broadcast.clone(); - let catalog = catalog.clone(); - let name = name.to_string(); + let crate::consume::Rendition { + broadcast, + config: catalog, + name, + driver, + } = rendition; let channel = oneshot::channel(); let entry = AudioTaskEntry { @@ -230,10 +231,10 @@ impl Audio { // `decode::Consumer::new` subscribes (blocking on SUBSCRIBE_OK), so run it // inside the task to keep this entrypoint non-blocking. tokio::spawn(async move { - let res = async move { + let res = crate::ffi::run(driver.watch(async move { let consumer = moq_audio::decode::Consumer::new(&broadcast, &catalog, name, config).await?; Self::run(on_frame, consumer, channel.1).await - } + })) .await; // Deliver one final terminal callback (code <= 0), then drop the entry. @@ -520,10 +521,10 @@ pub unsafe extern "C" fn moq_decode_audio( let on_frame = unsafe { OnStatus::new(user_data, on_frame)? }; let mut state = State::lock(); - let (broadcast, audio_cfg, name) = state.consume.audio_rendition(catalog, index as usize)?; + let rendition = state.consume.audio_rendition(catalog, index as usize)?; let State { audio, .. } = &mut *state; - audio.consume(&broadcast, &audio_cfg, &name, config, on_frame) + audio.consume(rendition, config, on_frame) }) } diff --git a/rs/moq-c/src/consume.rs b/rs/moq-c/src/consume.rs index a2717454037..48e2c6acbf4 100644 --- a/rs/moq-c/src/consume.rs +++ b/rs/moq-c/src/consume.rs @@ -19,7 +19,7 @@ struct ConsumeCatalog { /// The origin `broadcast` was resolved through, if any. A rendition may name a sibling /// broadcast, and only an origin can fetch one; a broadcast handed over without one (a /// local producer) leaves such a rendition unresolvable rather than silently wrong. - origin: Option, + origin: Option, // Carries the untyped `Extra` extension so application catalog sections survive // into `moq_catalog_section_*` instead of being dropped on parse. @@ -34,6 +34,30 @@ struct ConsumeCatalog { sections: Vec<(String, String)>, } +impl ConsumeCatalog { + fn rendition(&self, name: &str, config: &C) -> Rendition { + Rendition { + broadcast: self.broadcast.clone(), + config: config.clone(), + name: name.to_string(), + driver: self + .origin + .as_ref() + .map(|origin| origin.driver.clone()) + .unwrap_or_default(), + } + } +} + +/// A catalog rendition to decode: the broadcast and track serving it, and the owner whose panic +/// ends the decode. +pub struct Rendition { + pub broadcast: moq_net::broadcast::Consumer, + pub config: C, + pub name: String, + pub driver: crate::ffi::Background, +} + /// A broadcast handed to C, plus the origin cursor that named it. /// /// The origin is what lets a catalog rendition reference a sibling broadcast: only an origin @@ -43,7 +67,7 @@ struct ConsumeCatalog { #[derive(Clone)] struct ConsumeBroadcast { broadcast: moq_net::broadcast::Consumer, - origin: Option, + origin: Option, } /// The broadcast serving a rendition, honoring its catalog `broadcast` reference. @@ -54,7 +78,7 @@ struct ConsumeBroadcast { /// task, never under the state lock, so `origin` is cloned out rather than borrowed. async fn resolve( broadcast: moq_net::broadcast::Consumer, - origin: Option, + origin: Option, reference: Option, ) -> Result { let Some(reference) = reference.filter(|reference| !reference.is_empty()) else { @@ -62,8 +86,11 @@ async fn resolve( }; let origin = origin.ok_or_else(|| Error::UnresolvableBroadcast(reference.as_str().to_string()))?; - let source = moq_mux::Source::new(origin, broadcast); - Ok(source.resolve(Some(&reference.borrow())).await?) + let source = moq_mux::Source::new(origin.inner, broadcast); + origin + .driver + .watch(async { Ok(source.resolve(Some(&reference.borrow())).await?) }) + .await } /// A spawned task entry: `close` signals shutdown, `callback` delivers status. @@ -144,7 +171,7 @@ impl Consume { pub fn start( &mut self, broadcast: moq_net::broadcast::Consumer, - origin: Option, + origin: Option, ) -> Result { self.broadcast.insert(ConsumeBroadcast { broadcast, origin }) } @@ -153,6 +180,7 @@ impl Consume { let ConsumeBroadcast { broadcast, origin } = self.broadcast.get(broadcast).ok_or(Error::BroadcastNotFound)?.clone(); + let driver = origin.as_ref().map(|origin| origin.driver.clone()).unwrap_or_default(); let channel = oneshot::channel(); let entry = TaskEntry { close: Some(channel.0), @@ -163,13 +191,13 @@ impl Consume { // `subscribe` blocks on SUBSCRIBE_OK, so run it inside the task // to keep this entrypoint non-blocking. tokio::spawn(async move { - let res = async move { + let res = crate::ffi::run(driver.watch(async move { let catalog = broadcast .track(hang::catalog::Catalog::DEFAULT_NAME)? .subscribe(hang::catalog::Catalog::default_subscription()) .await?; Self::run_catalog(on_catalog, broadcast.clone(), origin, catalog.into(), channel.1).await - } + })) .await; // Deliver one final terminal callback (code <= 0), then drop the entry. @@ -186,7 +214,7 @@ impl Consume { async fn run_catalog( callback: OnStatus, broadcast: moq_net::broadcast::Consumer, - origin: Option, + origin: Option, mut catalog: moq_mux::catalog::hang::Consumer, mut close: oneshot::Receiver<()>, ) -> Result<(), Error> { @@ -434,6 +462,7 @@ impl Consume { let reference = config.broadcast.clone(); let broadcast = consume.broadcast.clone(); let origin = consume.origin.clone(); + let driver = origin.as_ref().map(|origin| origin.driver.clone()).unwrap_or_default(); let channel = oneshot::channel(); let entry = TaskEntry { @@ -444,7 +473,7 @@ impl Consume { // `subscribe` blocks on SUBSCRIBE_OK, so run it inside the task. tokio::spawn(async move { - let res = async move { + let res = crate::ffi::run(driver.watch(async move { let broadcast = resolve(broadcast, origin, reference).await?; let track = broadcast .track(&name)? @@ -456,7 +485,7 @@ impl Consume { .await?; let track = moq_mux::container::Consumer::new(track, container); Self::run_track(on_frame, track, channel.1).await - } + })) .await; // Deliver one final terminal callback (code <= 0), then drop the entry. @@ -493,6 +522,7 @@ impl Consume { let reference = config.broadcast.clone(); let broadcast = consume.broadcast.clone(); let origin = consume.origin.clone(); + let driver = origin.as_ref().map(|origin| origin.driver.clone()).unwrap_or_default(); let channel = oneshot::channel(); let entry = TaskEntry { @@ -503,7 +533,7 @@ impl Consume { // `subscribe` blocks on SUBSCRIBE_OK, so run it inside the task. tokio::spawn(async move { - let res = async move { + let res = crate::ffi::run(driver.watch(async move { let broadcast = resolve(broadcast, origin, reference).await?; let track = broadcast .track(&name)? @@ -515,7 +545,7 @@ impl Consume { .await?; let track = moq_mux::container::Consumer::new(track, container); Self::run_track(on_frame, track, channel.1).await - } + })) .await; // Deliver one final terminal callback (code <= 0), then drop the entry. @@ -607,12 +637,13 @@ impl Consume { subscription: Option, on_frame: OnStatus, ) -> Result { - let broadcast = self - .broadcast - .get(broadcast) - .ok_or(Error::BroadcastNotFound)? - .broadcast - .clone(); + let context = self.broadcast.get(broadcast).ok_or(Error::BroadcastNotFound)?; + let driver = context + .origin + .as_ref() + .map(|origin| origin.driver.clone()) + .unwrap_or_default(); + let broadcast = context.broadcast.clone(); let name = name.to_string(); let channel = oneshot::channel(); @@ -626,11 +657,11 @@ impl Consume { // `subscribe` blocks on SUBSCRIBE_OK, so run it inside the task. tokio::spawn(async move { - let res = async move { + let res = crate::ffi::run(driver.watch(async move { let mut track = broadcast.track(&name)?.subscribe(subscription.clone()).await?.ordered(); Self::apply_raw_subscription(&mut track, subscription); Self::run_raw(on_frame, track, channel.1, updates).await - } + })) .await; // Deliver one final terminal callback (code <= 0), then drop the entry. @@ -795,12 +826,13 @@ impl Consume { /// `on_datagram` is called with a datagram ID for each datagram in arrival order; /// each must be released with [`Self::datagram_close`]. pub fn datagram_track(&mut self, broadcast: Id, name: &str, on_datagram: OnStatus) -> Result { - let broadcast = self - .broadcast - .get(broadcast) - .ok_or(Error::BroadcastNotFound)? - .broadcast - .clone(); + let context = self.broadcast.get(broadcast).ok_or(Error::BroadcastNotFound)?; + let driver = context + .origin + .as_ref() + .map(|origin| origin.driver.clone()) + .unwrap_or_default(); + let broadcast = context.broadcast.clone(); let name = name.to_string(); let channel = oneshot::channel(); @@ -812,10 +844,10 @@ impl Consume { // `subscribe` blocks on SUBSCRIBE_OK, so run it inside the task. tokio::spawn(async move { - let res = async move { + let res = crate::ffi::run(driver.watch(async move { let track = broadcast.track(&name)?.subscribe(None).await?; Self::run_datagrams(on_datagram, track, channel.1).await - } + })) .await; // Deliver one final terminal callback (code <= 0), then drop the entry. @@ -886,14 +918,8 @@ impl Consume { Ok(()) } - /// Look up a video rendition by catalog index, returning the - /// (broadcast, config, name) tuple needed to subscribe, mirroring - /// the index-based selection in `video`. - pub fn video_rendition( - &self, - catalog: Id, - index: usize, - ) -> Result<(moq_net::broadcast::Consumer, hang::catalog::VideoConfig, String), Error> { + /// Look up a video rendition by catalog index, mirroring the index-based selection in `video`. + pub fn video_rendition(&self, catalog: Id, index: usize) -> Result, Error> { let consume = self.catalog.get(catalog).ok_or(Error::CatalogNotFound)?; let (name, config) = consume .catalog @@ -902,7 +928,7 @@ impl Consume { .iter() .nth(index) .ok_or(Error::NoIndex)?; - Ok((consume.broadcast.clone(), config.clone(), name.clone())) + Ok(consume.rendition(name, config)) } /// Subscribe to a JSON snapshot track (lossy latest-value) by name. @@ -917,12 +943,13 @@ impl Consume { config: moq_json::snapshot::consumer::Config, on_value: OnStatus, ) -> Result { - let broadcast = self - .broadcast - .get(broadcast) - .ok_or(Error::BroadcastNotFound)? - .broadcast - .clone(); + let context = self.broadcast.get(broadcast).ok_or(Error::BroadcastNotFound)?; + let driver = context + .origin + .as_ref() + .map(|origin| origin.driver.clone()) + .unwrap_or_default(); + let broadcast = context.broadcast.clone(); let name = name.to_string(); let channel = oneshot::channel(); @@ -934,11 +961,11 @@ impl Consume { // `subscribe` blocks on SUBSCRIBE_OK, so run it inside the task. tokio::spawn(async move { - let res = async move { + let res = crate::ffi::run(driver.watch(async move { let track = broadcast.track(&name)?.subscribe(None).await?; let consumer = moq_json::snapshot::Consumer::::new(track, config); Self::run_json_snapshot(on_value, consumer, channel.1).await - } + })) .await; // Deliver one final terminal callback (code <= 0), then drop the entry. @@ -986,12 +1013,13 @@ impl Consume { config: moq_json::stream::Config, on_value: OnStatus, ) -> Result { - let broadcast = self - .broadcast - .get(broadcast) - .ok_or(Error::BroadcastNotFound)? - .broadcast - .clone(); + let context = self.broadcast.get(broadcast).ok_or(Error::BroadcastNotFound)?; + let driver = context + .origin + .as_ref() + .map(|origin| origin.driver.clone()) + .unwrap_or_default(); + let broadcast = context.broadcast.clone(); let name = name.to_string(); let channel = oneshot::channel(); @@ -1003,11 +1031,11 @@ impl Consume { // `subscribe` blocks on SUBSCRIBE_OK, so run it inside the task. tokio::spawn(async move { - let res = async move { + let res = crate::ffi::run(driver.watch(async move { let track = broadcast.track(&name)?.subscribe(None).await?; let consumer = moq_json::stream::Consumer::::new(track, config); Self::run_json_stream(on_value, consumer, channel.1).await - } + })) .await; let entry = State::lock().consume.json_task.remove(id).flatten(); @@ -1072,14 +1100,8 @@ impl Consume { self.json_value.remove(value).ok_or(Error::FrameNotFound)?; Ok(()) } - /// Look up an audio rendition by catalog index, returning the - /// (broadcast, config, name) tuple needed to subscribe, mirroring - /// the index-based selection in `audio`. - pub fn audio_rendition( - &self, - catalog: Id, - index: usize, - ) -> Result<(moq_net::broadcast::Consumer, hang::catalog::AudioConfig, String), Error> { + /// Look up an audio rendition by catalog index, mirroring the index-based selection in `audio`. + pub fn audio_rendition(&self, catalog: Id, index: usize) -> Result, Error> { let consume = self.catalog.get(catalog).ok_or(Error::CatalogNotFound)?; let (name, config) = consume .catalog @@ -1088,6 +1110,44 @@ impl Consume { .iter() .nth(index) .ok_or(Error::NoIndex)?; - Ok((consume.broadcast.clone(), config.clone(), name.clone())) + Ok(consume.rendition(name, config)) + } +} + +#[cfg(test)] +mod supervision_tests { + use super::*; + + #[tokio::test] + async fn a_decoded_rendition_watches_the_origin_that_named_it() { + let (origin, driver, trigger) = crate::ffi::panicking_origin(); + let broadcast = moq_net::broadcast::Info::new().produce(); + let mut catalog = moq_mux::catalog::hang::Catalog::::default(); + catalog.audio.renditions.insert( + "audio".into(), + hang::catalog::AudioConfig::new(hang::catalog::AudioCodec::Opus, 48_000, 2), + ); + let mut consume = Consume::default(); + let id = consume + .catalog + .insert(ConsumeCatalog { + broadcast: broadcast.consume(), + origin: Some(crate::origin::Consumer { + inner: origin.consume(), + driver: driver.clone(), + }), + catalog, + audio_codec: Vec::new(), + video_codec: Vec::new(), + sections: Vec::new(), + }) + .unwrap(); + let rendition = consume.audio_rendition(id, 0).unwrap(); + *trigger + .write() + .unwrap_or_else(|_| unreachable!("test keeps its panic trigger open")) = true; + assert!(matches!(driver.finished().await, Err(Error::Panic))); + let result = rendition.driver.check(); + assert!(matches!(result, Err(Error::Panic)), "{result:?}"); } } diff --git a/rs/moq-c/src/error.rs b/rs/moq-c/src/error.rs index 5eef58e6834..6695f00f1b3 100644 --- a/rs/moq-c/src/error.rs +++ b/rs/moq-c/src/error.rs @@ -447,6 +447,7 @@ impl ffi::ReturnCode for Error { Error::Audio(_) => MOQ_ERROR_AUDIO, Error::BufferNotConsumed => MOQ_ERROR_BUFFER_NOT_CONSUMED, Error::GroupNotFound => MOQ_ERROR_GROUP_NOT_FOUND, + Error::Native(moq_tokio::Error::Panic) => MOQ_ERROR_PANIC, Error::Native(_) => MOQ_ERROR_NATIVE, Error::Unauthorized => MOQ_ERROR_UNAUTHORIZED, Error::Forbidden => MOQ_ERROR_FORBIDDEN, diff --git a/rs/moq-c/src/ffi.rs b/rs/moq-c/src/ffi.rs index b66c3d610ee..bc90777c4b7 100644 --- a/rs/moq-c/src/ffi.rs +++ b/rs/moq-c/src/ffi.rs @@ -51,6 +51,144 @@ pub fn enter C>(f: F) -> i32 { } } +/// Run a background operation before its terminal callback and resource cleanup. +pub async fn run(future: impl std::future::Future>) -> Result { + moq_tokio::panic::isolate(future).await.unwrap_or(Err(Error::Panic)) +} + +/// Completion of an owned background task, shared by the object's host operations. +#[derive(Clone, Default)] +pub struct Background { + result: Option>>>, +} + +impl Background { + /// Spawn an owned operation and retain its completion. + pub fn new(future: impl std::future::Future> + Send + 'static) -> Self { + let result = kio::Producer::new(None); + let receiver = result.consume(); + RUNTIME.spawn(async move { + let completed = run(future).await; + *result + .write() + .unwrap_or_else(|_| unreachable!("only the background task closes its completion")) = Some(completed); + }); + Self { result: Some(receiver) } + } + + /// Own a model driver until its completion has been reported. + pub fn driver(mut driver: impl moq_net::time::Driver + Send + 'static) -> Self { + let result = kio::Producer::new(None); + let receiver = result.consume(); + RUNTIME.spawn(async move { + // Keep the driver outside the unwind, so its channels close after panic publication. + let completed = run(async { + match moq_net::time::run(Borrowed(&mut driver)).await { + moq_net::Error::Closed => Ok(()), + error => Err(error.into()), + } + }) + .await; + if matches!(&completed, Err(Error::Panic)) { + *result + .write() + .unwrap_or_else(|_| unreachable!("only the task closes its completion")) = Some(completed.clone()); + } + let completed = run(async move { + drop(driver); + completed + }) + .await; + *result + .write() + .unwrap_or_else(|_| unreachable!("only the background task closes its completion")) = Some(completed); + }); + Self { result: Some(receiver) } + } + + /// Return a completed background failure, if any. + pub fn check(&self) -> Result<(), Error> { + self.result + .as_ref() + .map(|result| result.read().clone().unwrap_or(Ok(()))) + .unwrap_or(Ok(())) + } + + fn poll_finished(&self, waiter: &kio::Waiter) -> std::task::Poll> { + let Some(result) = &self.result else { + return std::task::Poll::Ready(Ok(())); + }; + match std::task::ready!(result.poll(waiter, |state| match &**state { + Some(result) => std::task::Poll::Ready(result.clone()), + None => std::task::Poll::Pending, + })) { + Ok(result) => std::task::Poll::Ready(result), + Err(_) => std::task::Poll::Ready(Err(Error::Panic)), + } + } + + /// Wait until the background operation finishes and return its result. + pub async fn finished(&self) -> Result<(), Error> { + kio::wait(|waiter| self.poll_finished(waiter)).await + } + + /// Fail an operation when its owned background task fails. + pub async fn watch(&self, future: impl std::future::Future>) -> Result { + self.check()?; + if self.result.is_none() { + return future.await; + } + tokio::pin!(future); + let result = tokio::select! { + biased; + result = self.finished() => { result?; future.await } + result = &mut future => result, + }; + self.check()?; + result + } +} + +#[cfg(test)] +pub(crate) fn panicking_origin() -> (moq_net::origin::Producer, Background, kio::Producer) { + struct Driver { + inner: moq_net::origin::Driver, + failed: kio::Consumer, + } + impl moq_net::time::Driver for Driver { + fn poll( + &mut self, + now: moq_net::time::Instant, + waiter: &kio::Waiter, + ) -> Result, moq_net::Error> { + let _ = self.failed.poll(waiter, |state| { + assert!(!**state, "injected origin driver panic"); + std::task::Poll::<()>::Pending + }); + self.inner.poll(now, waiter) + } + } + let (origin, inner) = moq_net::origin::Producer::new(Default::default()); + let failed = kio::Producer::new(false); + let driver = Background::driver(Driver { + inner, + failed: failed.consume(), + }); + (origin, driver, failed) +} + +struct Borrowed<'a, D>(&'a mut D); + +impl moq_net::time::Driver for Borrowed<'_, D> { + fn poll( + &mut self, + now: moq_net::time::Instant, + waiter: &kio::Waiter, + ) -> Result, moq_net::Error> { + self.0.poll(now, waiter) + } +} + /// Wrapper for C callback functions with user data. /// /// Stores a function pointer and user data pointer to call C callbacks @@ -319,3 +457,89 @@ pub unsafe fn parse_slice<'a>(data: *const u8, size: usize) -> Result<&'a [u8], let data = unsafe { std::slice::from_raw_parts(data, size) }; Ok(data) } + +#[cfg(test)] +mod tests { + + #[tokio::test] + async fn a_driver_teardown_panic_reaches_its_owner() { + struct Driver; + impl moq_net::time::Driver for Driver { + fn poll( + &mut self, + _: moq_net::time::Instant, + _: &kio::Waiter, + ) -> Result, moq_net::Error> { + Err(moq_net::Error::Closed) + } + } + impl Drop for Driver { + fn drop(&mut self) { + panic!("injected driver teardown panic"); + } + } + let driver = crate::ffi::Background::driver(Driver); + assert!(matches!(driver.finished().await, Err(Error::Panic))); + } + use super::*; + + #[tokio::test] + async fn an_owned_driver_panic_wins_over_its_dropped_channels() { + let driver = Background::new(async { panic!("injected origin driver panic") }); + assert!(matches!( + driver.watch(std::future::pending::>()).await, + Err(Error::Panic) + )); + assert!(matches!( + driver + .watch(async { Err::<(), _>(moq_net::Error::Cancel.into()) }) + .await, + Err(Error::Panic) + )); + } + + struct PanickingDriver { + dropped: Option>, + observed: std::sync::mpsc::Receiver<()>, + } + + impl moq_net::time::Driver for PanickingDriver { + fn poll( + &mut self, + _: moq_net::time::Instant, + _: &kio::Waiter, + ) -> Result, moq_net::Error> { + panic!("injected driver poll panic"); + } + } + + impl Drop for PanickingDriver { + fn drop(&mut self) { + let _ = self.dropped.take().unwrap().send(()); + // Hold teardown until the foreign host has classified the dropped channel. + let _ = self.observed.recv(); + } + } + + #[tokio::test] + async fn a_driver_panic_is_recorded_before_teardown_wakes_the_host() { + let (dropped, closed) = tokio::sync::oneshot::channel(); + let (observed, ack) = std::sync::mpsc::channel(); + let driver = Background::driver(PanickingDriver { + dropped: Some(dropped), + observed: ack, + }); + let result = driver + .watch(async { + let _ = closed.await; + Err::<(), _>(moq_net::Error::Cancel.into()) + }) + .await; + let _ = observed.send(()); + assert!(matches!(driver.finished().await, Err(Error::Panic))); + assert!( + matches!(result, Err(Error::Panic)), + "host saw teardown before panic: {result:?}" + ); + } +} diff --git a/rs/moq-c/src/origin.rs b/rs/moq-c/src/origin.rs index d966c13d08d..a4109492dd7 100644 --- a/rs/moq-c/src/origin.rs +++ b/rs/moq-c/src/origin.rs @@ -23,7 +23,7 @@ struct TaskEntry { #[derive(Default)] pub struct Origin { /// Active origin producers for publishing and consuming broadcasts. - active: NonZeroSlab, + active: NonZeroSlab, /// Delivered announce events, freed by the caller. announced: NonZeroSlab, @@ -41,6 +41,41 @@ pub struct Origin { broadcast_request: NonZeroSlab>, } +#[derive(Clone)] +pub(crate) struct Producer { + pub producer: moq_net::origin::Producer, + pub driver: crate::ffi::Background, +} + +impl std::ops::Deref for Producer { + type Target = moq_net::origin::Producer; + fn deref(&self) -> &Self::Target { + &self.producer + } +} + +#[derive(Clone)] +pub(crate) struct Consumer { + pub inner: moq_net::origin::Consumer, + pub driver: crate::ffi::Background, +} + +impl std::ops::Deref for Consumer { + type Target = moq_net::origin::Consumer; + fn deref(&self) -> &Self::Target { + &self.inner + } +} + +impl Producer { + fn consume_owned(&self) -> Consumer { + Consumer { + inner: self.producer.consume(), + driver: self.driver.clone(), + } + } +} + /// One announce event and the C string views borrowed from it. struct AnnouncedRecord { prefix: String, @@ -97,11 +132,17 @@ impl Origin { pub fn create(&mut self) -> Result { // Every FFI entry point runs inside `RUNTIME.enter()`, so the driver // lands on the dedicated moq-c runtime. - self.active.insert(moq_tokio::origin::spawn()) + let (producer, driver) = moq_net::origin::Producer::new(Default::default()); + self.active.insert(Producer { + producer, + driver: crate::ffi::Background::driver(driver), + }) } - pub fn get(&self, id: Id) -> Result<&moq_net::origin::Producer, Error> { - self.active.get(id).ok_or(Error::OriginNotFound) + pub fn get(&self, id: Id) -> Result<&Producer, Error> { + let origin = self.active.get(id).ok_or(Error::OriginNotFound)?; + origin.driver.check()?; + Ok(origin) } pub fn announced( @@ -111,8 +152,11 @@ impl Origin { hidden: bool, on_announce: OnStatus, ) -> Result { - let origin = self.active.get_mut(origin).ok_or(Error::OriginNotFound)?; + let origin = self.active.get(origin).ok_or(Error::OriginNotFound)?; + origin.driver.check()?; + let driver = origin.driver.clone(); let consumer = origin + .producer .consume() .scope("", &moq_net::Patterns::from(scope))? .with_hidden(hidden) @@ -126,7 +170,7 @@ impl Origin { let id = self.announced_task.insert(Some(entry))?; tokio::spawn(async move { - let res = Self::run_announced(on_announce, consumer, channel.1).await; + let res = crate::ffi::run(driver.watch(Self::run_announced(on_announce, consumer, channel.1))).await; // Deliver one final terminal callback (code <= 0), then drop the entry. // Pull it out from under the lock so the callback never runs while held. @@ -204,8 +248,10 @@ impl Origin { /// or cancellation it fires a single terminal code (`0` on close, negative on error). Returns a /// task handle for cancellation via [`Self::consume_announced_close`]. pub fn consume_announced(&mut self, origin: Id, path: String, on_broadcast: OnStatus) -> Result { - let origin = self.active.get_mut(origin).ok_or(Error::OriginNotFound)?; - let consumer = origin.consume(); + let origin = self.active.get(origin).ok_or(Error::OriginNotFound)?; + origin.driver.check()?; + let driver = origin.driver.clone(); + let consumer = origin.consume_owned(); let channel = oneshot::channel(); let entry = TaskEntry { @@ -215,7 +261,9 @@ impl Origin { let id = self.consume_task.insert(Some(entry))?; tokio::spawn(async move { - let res = Self::run_consume_announced(on_broadcast, consumer, path, channel.1).await; + let res = + crate::ffi::run(driver.watch(Self::run_consume_announced(on_broadcast, consumer, path, channel.1))) + .await; // Deliver one final terminal callback (code <= 0), then drop the entry. // Pull it out from under the lock so the callback never runs while held. @@ -230,7 +278,7 @@ impl Origin { async fn run_consume_announced( callback: OnStatus, - consumer: moq_net::origin::Consumer, + consumer: Consumer, path: String, mut close: oneshot::Receiver<()>, ) -> Result<(), Error> { @@ -265,8 +313,10 @@ impl Origin { /// handle (> 0) once served, then a terminal `0`; or a single terminal code (`0` on close, /// negative on error) if it can't be served. Returns a task handle for cancellation. pub fn request(&mut self, origin: Id, path: String, on_broadcast: OnStatus) -> Result { - let origin = self.active.get_mut(origin).ok_or(Error::OriginNotFound)?; - let consumer = origin.consume(); + let origin = self.active.get(origin).ok_or(Error::OriginNotFound)?; + origin.driver.check()?; + let driver = origin.driver.clone(); + let consumer = origin.consume_owned(); let channel = oneshot::channel(); let entry = TaskEntry { @@ -276,7 +326,7 @@ impl Origin { let id = self.consume_task.insert(Some(entry))?; tokio::spawn(async move { - let res = Self::run_request(on_broadcast, consumer, path, channel.1).await; + let res = crate::ffi::run(driver.watch(Self::run_request(on_broadcast, consumer, path, channel.1))).await; // Deliver one final terminal callback (code <= 0), then drop the entry. // Pull it out from under the lock so the callback never runs while held. @@ -291,7 +341,7 @@ impl Origin { async fn run_request( callback: OnStatus, - consumer: moq_net::origin::Consumer, + consumer: Consumer, path: String, mut close: oneshot::Receiver<()>, ) -> Result<(), Error> { @@ -330,9 +380,11 @@ impl Origin { &self, origin: Id, path: P, - ) -> Result { - let origin = self.active.get(origin).ok_or(Error::OriginNotFound)?; - Ok(origin.create_broadcast(path)?) + ) -> Result<(moq_net::broadcast::Producer, crate::ffi::Background), Error> { + let origin = self.get(origin)?; + let result = origin.producer.create_broadcast(path); + origin.driver.check()?; + Ok((result?, origin.driver.clone())) } /// Advertise `prefix` and serve requests beneath it, delivering each as a @@ -345,7 +397,11 @@ impl Origin { on_request: OnStatus, ) -> Result { let origin = self.active.get(origin).ok_or(Error::OriginNotFound)?; - let inner = origin.dynamic(prefix, route)?; + origin.driver.check()?; + let driver = origin.driver.clone(); + let result = origin.producer.dynamic(prefix, route); + origin.driver.check()?; + let inner = result?; let channel = oneshot::channel(); let id = self.dynamic.insert(Some(DynamicEntry { inner: Some(inner), @@ -354,7 +410,7 @@ impl Origin { }))?; tokio::spawn(async move { - let res = Self::run_dynamic(id, channel.1).await; + let res = crate::ffi::run(driver.watch(Self::run_dynamic(id, channel.1))).await; let entry = State::lock().origin.dynamic.remove(id).flatten(); if let Some(entry) = entry { entry.callback.call(res); @@ -441,3 +497,40 @@ impl Origin { Ok(()) } } + +#[cfg(test)] +mod supervision_tests { + use super::*; + + #[tokio::test] + async fn a_broadcast_keeps_the_origin_failure_after_its_origin_handle_is_released() { + let (producer, completion, trigger) = crate::ffi::panicking_origin(); + let mut origins = Origin::default(); + let origin = origins + .active + .insert(Producer { + producer, + driver: completion.clone(), + }) + .unwrap(); + let mut publish = crate::publish::Publish::default(); + let broadcast = publish + .create(origins.create_broadcast(origin, "retained").unwrap()) + .unwrap(); + origins.close(origin).unwrap(); + *trigger + .write() + .unwrap_or_else(|_| unreachable!("test keeps its panic trigger open")) = true; + assert!(matches!(completion.finished().await, Err(Error::Panic))); + drop(completion); + assert!(matches!( + publish.announce(broadcast, Default::default()), + Err(Error::Panic) + )); + let healthy = origins.create().unwrap(); + let broadcast = publish + .create(origins.create_broadcast(healthy, "healthy").unwrap()) + .unwrap(); + publish.announce(broadcast, Default::default()).unwrap(); + } +} diff --git a/rs/moq-c/src/publish.rs b/rs/moq-c/src/publish.rs index 1433db00996..e0ab5e7e9a8 100644 --- a/rs/moq-c/src/publish.rs +++ b/rs/moq-c/src/publish.rs @@ -44,6 +44,7 @@ enum Dynamic { /// Caller-authored configs are tracked separately so removing one cannot retire an importer's /// rendition with the same name. struct Broadcast { + driver: crate::ffi::Background, producer: moq_net::broadcast::Producer, catalog: moq_mux::catalog::Producer, video: BTreeMap, @@ -57,6 +58,11 @@ struct Data { clock: moq_mux::Clock, } +struct Container { + inner: import::Container, + driver: crate::ffi::Background, +} + #[derive(Default)] pub struct Publish { /// Active broadcast producers for publishing. @@ -69,7 +75,7 @@ pub struct Publish { /// Container importers, fed whole chunks. A separate space from `media` because a /// container publishes several tracks and carries its own timing, so it takes no /// per-frame timestamp. - containers: NonZeroSlab>, + containers: NonZeroSlab, /// Raw track producers (no media/container/catalog framing). tracks: NonZeroSlab, @@ -105,14 +111,30 @@ pub struct Publish { } impl Publish { + fn broadcast(&self, id: Id) -> Result<&Broadcast, Error> { + let broadcast = self.broadcasts.get(id).ok_or(Error::BroadcastNotFound)?; + broadcast.driver.check()?; + Ok(broadcast) + } + + fn broadcast_mut(&mut self, id: Id) -> Result<&mut Broadcast, Error> { + let broadcast = self.broadcasts.get_mut(id).ok_or(Error::BroadcastNotFound)?; + broadcast.driver.check()?; + Ok(broadcast) + } + /// Store an origin-created broadcast producer, attaching the catalog track /// every moq-c broadcast carries. - pub fn create(&mut self, mut broadcast: moq_net::broadcast::Producer) -> Result { + pub fn create( + &mut self, + (mut broadcast, driver): (moq_net::broadcast::Producer, crate::ffi::Background), + ) -> Result { let config = moq_mux::catalog::Config::default() .with_catalog(moq_mux::catalog::hang::Catalog::::default()); let catalog = moq_mux::catalog::Producer::new(&mut broadcast, config)?; let id = self.broadcasts.insert(Broadcast { + driver, producer: broadcast, catalog, video: BTreeMap::new(), @@ -124,26 +146,30 @@ impl Publish { /// Advertise the broadcast's exact path as a route. Announcing again re-prices /// in place. Until announced, the broadcast is invisible and unroutable. pub fn announce(&mut self, broadcast: Id, route: moq_net::origin::Route) -> Result<(), Error> { - let broadcast = self.broadcasts.get_mut(broadcast).ok_or(Error::BroadcastNotFound)?; - broadcast.producer.announce(route)?; + let broadcast = self.broadcast_mut(broadcast)?; + let result = broadcast.producer.announce(route); + broadcast.driver.check()?; + result?; Ok(()) } /// Retract the broadcast's exact-path advertisement, if any. pub fn unannounce(&mut self, broadcast: Id) -> Result<(), Error> { - let broadcast = self.broadcasts.get_mut(broadcast).ok_or(Error::BroadcastNotFound)?; + let broadcast = self.broadcast_mut(broadcast)?; broadcast.producer.unannounce(); Ok(()) } /// The broadcast's track producer. pub(crate) fn producer(&mut self, id: Id) -> Result<&mut moq_net::broadcast::Producer, Error> { - Ok(&mut self.broadcasts.get_mut(id).ok_or(Error::BroadcastNotFound)?.producer) + let broadcast = self.broadcast_mut(id)?; + Ok(&mut broadcast.producer) } /// The broadcast's catalog producer. fn catalog(&mut self, id: Id) -> Result<&mut moq_mux::catalog::Producer, Error> { - Ok(&mut self.broadcasts.get_mut(id).ok_or(Error::BroadcastNotFound)?.catalog) + let broadcast = self.broadcast_mut(id)?; + Ok(&mut broadcast.catalog) } /// The broadcast's current catalog, as consumers would receive it next. @@ -165,34 +191,39 @@ impl Publish { ), Error, > { - let broadcast = self.broadcasts.get_mut(id).ok_or(Error::BroadcastNotFound)?; + let broadcast = self.broadcast_mut(id)?; Ok((&mut broadcast.producer, &mut broadcast.catalog)) } /// End the broadcast for good and release it, finalizing the catalog stream. pub fn close(&mut self, broadcast: Id) -> Result<(), Error> { let Broadcast { + driver, producer, mut catalog, video, audio, .. } = self.broadcasts.remove(broadcast).ok_or(Error::BroadcastNotFound)?; - // Retire caller-authored entries while the catalog track is still open. - { - let mut guard = catalog.modify()?; - for name in video.keys() { - guard.video.renditions.remove(name); - } - for name in audio.keys() { - guard.audio.renditions.remove(name); + let result = (|| { + // Retire caller-authored entries while the catalog track is still open. + { + let mut guard = catalog.modify()?; + for name in video.keys() { + guard.video.renditions.remove(name); + } + for name in audio.keys() { + guard.audio.renditions.remove(name); + } + guard.commit()?; } - guard.commit()?; - } - // Close the broadcast first so it ends even if finalizing the catalog fails. - producer.close(); - catalog.finish()?; - Ok(()) + // Close the broadcast first so it ends even if finalizing the catalog fails. + producer.close(); + catalog.finish()?; + Ok(()) + })(); + driver.check()?; + result } pub fn audio(&mut self, broadcast: Id, init: import::AudioInit) -> Result { @@ -200,7 +231,7 @@ impl Publish { producer: broadcast, catalog, .. - } = self.broadcasts.get(broadcast).ok_or(Error::BroadcastNotFound)?; + } = self.broadcast(broadcast)?; let broadcast = broadcast.clone(); let name = broadcast.unique_name(&format!(".{}", init.format)); let request = broadcast.reserve_track(name)?; @@ -215,7 +246,7 @@ impl Publish { producer: broadcast, catalog, .. - } = self.broadcasts.get(broadcast).ok_or(Error::BroadcastNotFound)?; + } = self.broadcast(broadcast)?; let broadcast = broadcast.clone(); let name = broadcast.unique_name(&format!(".{}", init.format)); let request = broadcast.reserve_track(name)?; @@ -229,9 +260,14 @@ impl Publish { let Broadcast { producer: broadcast, catalog, + driver, .. - } = self.broadcasts.get(broadcast).ok_or(Error::BroadcastNotFound)?; + } = self.broadcast(broadcast)?; let container = import::Container::new(broadcast.clone(), catalog.reserve(), &init)?; + let container = Container { + inner: container, + driver: driver.clone(), + }; let id = self.containers.insert(container)?; Ok(id) } @@ -291,28 +327,34 @@ impl Publish { /// No timestamp: a container carries its tracks' timing itself. pub fn container_write(&mut self, container: Id, data: &[u8]) -> Result<(), Error> { let container = self.containers.get_mut(container).ok_or(Error::MediaNotFound)?; - container.decode(data)?; - Ok(()) + container.driver.check()?; + let result = container.inner.decode(data); + container.driver.check()?; + Ok(result?) } /// Declare that the next chunk starts a new segment, rolling a group on every track. pub fn container_cut(&mut self, container: Id) -> Result<(), Error> { let container = self.containers.get_mut(container).ok_or(Error::MediaNotFound)?; - container.cut(); - Ok(()) + container.driver.check()?; + container.inner.cut(); + container.driver.check() } /// Start a new segment and number its groups `sequence`. pub fn container_seek(&mut self, container: Id, sequence: u64) -> Result<(), Error> { let container = self.containers.get_mut(container).ok_or(Error::MediaNotFound)?; - container.seek(sequence)?; - Ok(()) + container.driver.check()?; + let result = container.inner.seek(sequence); + container.driver.check()?; + Ok(result?) } pub fn container_finish(&mut self, container: Id) -> Result<(), Error> { let mut container = self.containers.remove(container).ok_or(Error::MediaNotFound)?; - container.finish()?; - Ok(()) + let result = container.inner.finish(); + container.driver.check()?; + Ok(result?) } /// Insert or replace a caller-authored video rendition in the broadcast's catalog. @@ -320,7 +362,7 @@ impl Publish { /// Errors if a media importer owns the name, since it publishes and retires its own rendition. /// The catalog is republished automatically. pub fn video_config(&mut self, broadcast: Id, name: &str, config: hang::catalog::VideoConfig) -> Result<(), Error> { - let broadcast = self.broadcasts.get_mut(broadcast).ok_or(Error::BroadcastNotFound)?; + let broadcast = self.broadcast_mut(broadcast)?; if !broadcast.video.contains_key(name) && broadcast.catalog.is_claimed::(name) { return Err(Error::Hang(hang::Error::Duplicate(name.to_string()))); } @@ -335,7 +377,7 @@ impl Publish { /// /// Same rules as [`Self::video_config`]. pub fn audio_config(&mut self, broadcast: Id, name: &str, config: hang::catalog::AudioConfig) -> Result<(), Error> { - let broadcast = self.broadcasts.get_mut(broadcast).ok_or(Error::BroadcastNotFound)?; + let broadcast = self.broadcast_mut(broadcast)?; if !broadcast.audio.contains_key(name) && broadcast.catalog.is_claimed::(name) { return Err(Error::Hang(hang::Error::Duplicate(name.to_string()))); } @@ -352,7 +394,7 @@ impl Publish { /// the handle is what retires the entry, and the importer holds its own. The catalog is /// republished automatically. pub fn video_remove(&mut self, broadcast: Id, name: &str) -> Result<(), Error> { - let broadcast = self.broadcasts.get_mut(broadcast).ok_or(Error::BroadcastNotFound)?; + let broadcast = self.broadcast_mut(broadcast)?; if broadcast.video.remove(name).is_some() { let mut catalog = broadcast.catalog.modify()?; catalog.video.renditions.remove(name); @@ -365,7 +407,7 @@ impl Publish { /// /// Same rules as [`Self::video_remove`]. pub fn audio_remove(&mut self, broadcast: Id, name: &str) -> Result<(), Error> { - let broadcast = self.broadcasts.get_mut(broadcast).ok_or(Error::BroadcastNotFound)?; + let broadcast = self.broadcast_mut(broadcast)?; if broadcast.audio.remove(name).is_some() { let mut catalog = broadcast.catalog.modify()?; catalog.audio.renditions.remove(name); @@ -442,7 +484,7 @@ impl Publish { }))?; tokio::spawn(async move { - let res = Self::run_demand(on_demand, demand, channel.1).await; + let res = crate::ffi::run(Self::run_demand(on_demand, demand, channel.1)).await; // Deliver one final terminal callback (code <= 0), then drop the entry. // Pull it out from under the lock so the callback never runs while held. @@ -525,14 +567,15 @@ impl Publish { /// Without a live handler an unknown track name is refused, as before. pub fn dynamic(&mut self, broadcast: Id, on_request: OnStatus) -> Result { let dynamic = self.producer(broadcast)?.dynamic(); - self.spawn_dynamic(Dynamic::Broadcast(dynamic, broadcast), on_request) + let driver = self.broadcast(broadcast)?.driver.clone(); + self.spawn_dynamic(Dynamic::Broadcast(dynamic, broadcast), on_request, driver) } /// Serve fetches of uncached groups on a raw track, delivering each as a group-request /// handle via `on_group`. pub fn track_dynamic(&mut self, track: Id, on_group: OnStatus) -> Result { let dynamic = self.tracks.get(track).ok_or(Error::TrackNotFound)?.dynamic(); - self.spawn_dynamic(Dynamic::Track(dynamic), on_group) + self.spawn_dynamic(Dynamic::Track(dynamic), on_group, Default::default()) } /// Serve fetches of uncached groups on a track that has not been accepted yet. @@ -546,10 +589,15 @@ impl Publish { .ok_or(Error::NotFound)? .request .dynamic(); - self.spawn_dynamic(Dynamic::Track(dynamic), on_group) + self.spawn_dynamic(Dynamic::Track(dynamic), on_group, Default::default()) } - fn spawn_dynamic(&mut self, dynamic: Dynamic, callback: OnStatus) -> Result { + fn spawn_dynamic( + &mut self, + dynamic: Dynamic, + callback: OnStatus, + driver: crate::ffi::Background, + ) -> Result { let channel = oneshot::channel(); let id = self.dynamic.insert(Some(TaskEntry { close: Some(channel.0), @@ -557,7 +605,7 @@ impl Publish { }))?; tokio::spawn(async move { - let res = Self::run_dynamic(callback, dynamic, channel.1).await; + let res = crate::ffi::run(driver.watch(Self::run_dynamic(callback, dynamic, channel.1))).await; // Deliver one final terminal callback (code <= 0), then drop the entry. // Pull it out from under the lock so the callback never runs while held. @@ -776,7 +824,7 @@ impl Publish { name: &str, publish: impl FnOnce(&moq_mux::catalog::Producer, moq_net::track::Producer) -> moq_mux::Result, ) -> Result, Error> { - let broadcast = self.broadcasts.get_mut(broadcast).ok_or(Error::BroadcastNotFound)?; + let broadcast = self.broadcast_mut(broadcast)?; let track = broadcast.producer.create_track( name, moq_net::track::Info::default().with_timescale(moq_net::Timescale::MILLI), @@ -906,3 +954,32 @@ impl Publish { Ok(()) } } + +#[cfg(test)] +mod supervision_tests { + use super::*; + + #[tokio::test] + async fn a_container_importer_keeps_the_origin_failure_after_other_owners_are_released() { + let (origin, completion, trigger) = crate::ffi::panicking_origin(); + let mut publish = Publish::default(); + let broadcast = publish + .create((origin.create_broadcast("retained").unwrap(), completion.clone())) + .unwrap(); + let container = publish + .container( + broadcast, + import::ContainerInit::new(import::ContainerFormat::Ts, bytes::Bytes::new()), + ) + .unwrap(); + publish.broadcasts.remove(broadcast).unwrap(); + drop(origin); + *trigger + .write() + .unwrap_or_else(|_| unreachable!("test keeps its panic trigger open")) = true; + assert!(matches!(completion.finished().await, Err(Error::Panic))); + drop(completion); + assert!(matches!(publish.container_cut(container), Err(Error::Panic))); + assert!(matches!(publish.container_finish(container), Err(Error::Panic))); + } +} diff --git a/rs/moq-c/src/server.rs b/rs/moq-c/src/server.rs index 83503820eba..3f621e41099 100644 --- a/rs/moq-c/src/server.rs +++ b/rs/moq-c/src/server.rs @@ -41,7 +41,7 @@ impl Server { }))?; tokio::spawn(async move { - let res = Self::run(server, callback, closed).await; + let res = crate::ffi::run(Self::run(server, callback, closed)).await; // Deliver one final terminal callback (code <= 0), then drop the entry. // Pull it out from under the lock so the callback never runs while held. diff --git a/rs/moq-c/src/session.rs b/rs/moq-c/src/session.rs index 5fa8bccbfac..e1fe70d6db5 100644 --- a/rs/moq-c/src/session.rs +++ b/rs/moq-c/src/session.rs @@ -33,6 +33,7 @@ enum Link { struct Accepted { session: moq_net::Session, + driver: ffi::Background, /// Shared by every `moq_session_bandwidth` handle, like [`Link::Dialed`]'s. bandwidth: moq_net::bandwidth::Allocator, } @@ -41,8 +42,8 @@ struct Accepted { pub(crate) struct Connect { pub config: crate::client::Config, pub url: Url, - pub publish: Option, - pub consume: Option, + pub publish: Option, + pub consume: Option, pub callback: ffi::OnStatus, } @@ -56,10 +57,10 @@ impl Connect { .init(self.config.quic.clone()) .map_err(|err| Error::InvalidConfig(err.to_string()))?; if let Some(publish) = &self.publish { - client = client.with_publisher(publish); + client = client.with_publisher(&publish.producer); } if let Some(consume) = &self.consume { - client = client.with_subscriber(consume.clone()); + client = client.with_subscriber(consume.producer.clone()); } Ok(PreparedConnect { @@ -76,8 +77,8 @@ impl Connect { pub(crate) struct PreparedConnect { client: moq_tokio::Client, url: Url, - publish: Option, - consume: Option, + publish: Option, + consume: Option, callback: ffi::OnStatus, } @@ -116,22 +117,29 @@ impl Session { tokio::spawn(async move { // Keep the origin producers alive for the lifetime of the reconnect loop: // the session reads from the publish consumer and writes into the subscribe producer. + let publish_driver = publish.as_ref().map(|origin| origin.driver.clone()).unwrap_or_default(); + let consume_driver = consume.as_ref().map(|origin| origin.driver.clone()).unwrap_or_default(); let _publish = publish; let _consume = consume; // A panic ends only this session, reported through the terminal callback. - let res = moq_tokio::panic::isolate(async move { + let res = ffi::run(publish_driver.watch(consume_driver.watch(async move { tokio::select! { // close() requested: drain what the session queued, then close it. _ = closed.1 => reconnect.clone().close().await.map_err(map_connect_error), res = Self::report(callback, reconnect.clone()) => res, } - }) - .await - .unwrap_or(Err(Error::Panic)); + }))) + .await; // The terminal callback waits for the close to reach the peer, so a caller that exits // on it doesn't leave the peer to time the connection out. - client.close().await; + let res = crate::ffi::run(async { + client.close().await; + publish_driver.check()?; + consume_driver.check()?; + res + }) + .await; // Deliver one final terminal callback (0 = closed, < 0 = error), then // drop the entry. Pull it out from under the lock so the callback never @@ -149,16 +157,16 @@ impl Session { pub fn accept( &mut self, request: moq_tokio::server::Request, - publish: Option, - consume: Option, + publish: Option, + consume: Option, callback: ffi::OnStatus, ) -> Result { let mut request = request; if let Some(publish) = &publish { - request = request.with_publisher(publish); + request = request.with_publisher(&publish.producer); } if let Some(consume) = &consume { - request = request.with_subscriber(consume.clone()); + request = request.with_subscriber(consume.producer.clone()); } let closed = oneshot::channel(); @@ -171,25 +179,33 @@ impl Session { tokio::spawn(async move { // Keep the origin producers alive for the lifetime of the session. + let publish_driver = publish.as_ref().map(|origin| origin.driver.clone()).unwrap_or_default(); + let consume_driver = consume.as_ref().map(|origin| origin.driver.clone()).unwrap_or_default(); let _publish = publish; let _consume = consume; // A panic ends only this session, reported through the terminal callback. - let res = moq_tokio::panic::isolate(async move { + let mut res = ffi::run(publish_driver.watch(consume_driver.watch(async move { tokio::select! { _ = closed.1 => Ok(()), res = Self::serve(id, callback, request) => res, } - }) - .await - .unwrap_or(Err(Error::Panic)); + }))) + .await; let entry = State::lock().session.task.remove(id).flatten(); if let Some(entry) = entry { // Close the transport now rather than whenever the last clone drops. - if let Link::Accepted(Some(accepted)) = &entry.link { - accepted.session.abort(moq_net::Error::Cancel); + if let Link::Accepted(Some(accepted)) = &entry.link + && let Err(error) = ffi::run(async { + accepted.session.abort(moq_net::Error::Cancel); + accepted.driver.finished().await + }) + .await + { + res = Err(error); } + let res = publish_driver.check().and_then(|()| consume_driver.check()).and(res); entry.callback.call(res); } }); @@ -199,7 +215,14 @@ impl Session { /// Establish an accepted session, publish it to the entry, and wait for it to close. async fn serve(id: Id, callback: ffi::OnStatus, request: moq_tokio::server::Request) -> Result<(), Error> { - let session = request.ok().await.map_err(map_connect_error)?; + let (session, driver) = request.ok().await.map_err(map_connect_error)?; + let driver = ffi::Background::new(async move { + match driver.await { + Ok(result) => result.map_err(map_connect_error), + Err(error) if error.is_panic() => Err(Error::Panic), + Err(_) => Err(moq_net::Error::Cancel.into()), + } + }); let bandwidth = session .send_bandwidth() .map(moq_net::bandwidth::Allocator::new) @@ -208,12 +231,14 @@ impl Session { if let Some(entry) = State::lock().session.task.get_mut(id).and_then(|entry| entry.as_mut()) { entry.link = Link::Accepted(Some(Accepted { session: session.clone(), + driver: driver.clone(), bandwidth, })); } // A server-accepted session is a single transport, so it only ever connects once. callback.call(1); + driver.finished().await?; Err(session.closed().await.into()) } diff --git a/rs/moq-c/src/test.rs b/rs/moq-c/src/test.rs index f24fc912a39..b4ce355b71a 100644 --- a/rs/moq-c/src/test.rs +++ b/rs/moq-c/src/test.rs @@ -5180,3 +5180,24 @@ fn data_track_names_cannot_collide() { assert_eq!(moq_publish_close(broadcast), 0); assert_eq!(moq_origin_close(origin), 0); } + +#[tokio::test] +async fn a_background_panic_reaches_the_terminal_callback_and_keeps_another_origin_working() { + let callback = Callback::new(); + let status = unsafe { crate::ffi::OnStatus::new(callback.ptr, Some(channel_callback)) }.unwrap(); + let origin = id(moq_origin_create()); + let task = crate::ffi::RUNTIME.spawn(async move { + let result = crate::ffi::run(async { + panic!("injected background panic"); + #[allow(unreachable_code)] + Ok::<(), Error>(()) + }) + .await; + status.call(result); + }); + assert!(task.await.is_ok(), "background panic lost the terminal callback"); + assert_eq!(callback.rx.try_recv().unwrap(), MOQ_ERROR_PANIC); + let broadcast = publish_broadcast(origin, b"survivor"); + assert_eq!(moq_publish_close(broadcast), 0); + assert_eq!(moq_origin_close(origin), 0); +} diff --git a/rs/moq-c/src/video.rs b/rs/moq-c/src/video.rs index bccaf89b9f1..04e784137a9 100644 --- a/rs/moq-c/src/video.rs +++ b/rs/moq-c/src/video.rs @@ -271,6 +271,7 @@ pub(crate) struct VideoEncoder { size: moq_video::Size, reservation: Option>, follow: Option>, + followed: Option, ceiling: Option>, } @@ -368,6 +369,9 @@ struct VideoTaskEntry { impl VideoEncoder { fn publish_frame(&mut self, timestamp_us: u64, data: &[u8]) -> Result<(), Error> { + if let Some(followed) = &self.followed { + followed.check()?; + } // A buffer that isn't one picture at the configured size is rejected here, // by the surface constructors, rather than reinterpreted. let size = self.size; @@ -387,12 +391,18 @@ impl VideoEncoder { } fn publish_cut(&mut self) -> Result<(), Error> { + if let Some(followed) = &self.followed { + followed.check()?; + } // A keyframe is what a cut is on the wire: the importer closes the open // group and starts a new one at it. Ok(block_on(self.encoder.cut())?) } fn publish_bitrate(&mut self, bitrate: u64) -> Result<(), Error> { + if let Some(followed) = &self.followed { + followed.check()?; + } block_on(self.encoder.set_bitrate(moq_net::bandwidth::Rate::from_bps(bitrate)))?; // Ceiling first: `update` wakes the follower, which must not read the old // floor and retune above this cap before parking on an unchanged grant. @@ -407,12 +417,18 @@ impl VideoEncoder { fn publish_finish(self) -> Result<(), Error> { let VideoEncoder { - encoder, mut producer, .. + encoder, + mut producer, + followed, + .. } = self; // Drain the codec into the track before ending it, so the last frames land // in it rather than being dropped with the encoder. let drained = block_on(encoder.finish()).and_then(|encoded| producer.publish(&encoded)); - finalize(producer, drained) + finalize(producer, drained)?; + // A follower panic only stops rate adaptation, so the track still ends + // cleanly before the panic is reported. + followed.map_or(Ok(()), |followed| followed.check()) } } @@ -439,6 +455,7 @@ impl Video { size: config.size(), reservation: None, follow: None, + followed: None, ceiling: None, })) } @@ -463,13 +480,15 @@ impl Video { let (close, closed) = oneshot::channel(); shared.lock().as_mut().ok_or(Error::MediaNotFound)?.follow = Some(close); let follower = shared.clone(); - tokio::spawn(async move { + let background = crate::ffi::Background::new(async move { tokio::select! { biased; _ = closed => {} _ = follow_reservation(follower, reservation.consumer(), ceiling) => {} } + Ok(()) }); + shared.lock().as_mut().ok_or(Error::MediaNotFound)?.followed = Some(background); Ok(()) } @@ -511,16 +530,17 @@ impl Video { pub fn consume( &mut self, - broadcast: &moq_net::broadcast::Consumer, - catalog: &hang::catalog::VideoConfig, - name: &str, + rendition: crate::consume::Rendition, options: moq_video::decode::Options, output: DecoderOutput, on_frame: OnStatus, ) -> Result { - let broadcast = broadcast.clone(); - let catalog = catalog.clone(); - let name = name.to_string(); + let crate::consume::Rendition { + broadcast, + config: catalog, + name, + driver, + } = rendition; let channel = oneshot::channel(); let entry = VideoTaskEntry { @@ -532,10 +552,10 @@ impl Video { // `Consumer::new` subscribes (blocking on SUBSCRIBE_OK), so run it inside // the task to keep this entrypoint non-blocking. tokio::spawn(async move { - let res = async move { + let res = crate::ffi::run(driver.watch(async move { let consumer = moq_video::decode::Consumer::new(&broadcast, &catalog, name, options).await?; Self::run(on_frame, consumer, channel.1, output).await - } + })) .await; // Deliver one final terminal callback (code <= 0), then drop the entry. @@ -924,10 +944,10 @@ pub unsafe extern "C" fn moq_decode_video( let on_frame = unsafe { OnStatus::new(user_data, on_frame)? }; let mut state = State::lock(); - let (broadcast, video_cfg, name) = state.consume.video_rendition(catalog, index as usize)?; + let rendition = state.consume.video_rendition(catalog, index as usize)?; let State { video, .. } = &mut *state; - video.consume(&broadcast, &video_cfg, &name, options, output, on_frame) + video.consume(rendition, options, output, on_frame) }) } @@ -1045,4 +1065,32 @@ mod tests { "the abort should carry the drain failure: {err}" ); } + + /// A rate follower panic stops adaptation, not the track: finishing still ends + /// it cleanly, then reports the panic. + #[tokio::test] + async fn finish_ends_the_track_after_a_rate_follower_panic() { + let (producer, mut track) = track_under_test().await; + let mut config = moq_video::encode::Config::new(320, 240, moq_video::Rate::new(30, 1).unwrap()); + config.kind = moq_video::encode::Kind::Software; + let encoder = moq_video::encode::Sink::open(&config).await.unwrap(); + let followed = crate::ffi::Background::new(async { panic!("injected rate follower panic") }); + assert!(matches!(followed.finished().await, Err(Error::Panic))); + let video = VideoEncoder { + encoder, + producer, + format: moq_video_pixel_format::MOQ_VIDEO_PIXEL_FORMAT_I420, + size: config.size(), + reservation: None, + follow: None, + followed: Some(followed), + ceiling: None, + }; + + let result = tokio::task::spawn_blocking(move || video.publish_finish()) + .await + .unwrap(); + assert!(matches!(result, Err(Error::Panic)), "{result:?}"); + assert!(matches!(track.recv_group().await, Ok(None)), "expected a clean end"); + } } diff --git a/rs/moq-cli/src/main.rs b/rs/moq-cli/src/main.rs index 2805b345892..e6d53e2f90f 100644 --- a/rs/moq-cli/src/main.rs +++ b/rs/moq-cli/src/main.rs @@ -225,8 +225,10 @@ impl Serving { if let Some(subscribe) = subscribe { request = request.with_subscriber(subscribe); } - let session = request.ok().await?; - moq_relay::supervise(session, lease, self.shutdown.clone(), None).await + let (session, driver) = request.ok().await?; + let result = moq_relay::supervise(session, lease, self.shutdown.clone(), None).await; + driver.await??; + result } } diff --git a/rs/moq-ffi/src/audio.rs b/rs/moq-ffi/src/audio.rs index f3edca887e1..15f993f7ae7 100644 --- a/rs/moq-ffi/src/audio.rs +++ b/rs/moq-ffi/src/audio.rs @@ -418,10 +418,13 @@ impl MoqBroadcastConsumer { config.output.layout = output.channels.map(moq_audio::Layout::from_channels).transpose()?; config.max_delay = output.max_delay_us.map(Duration::from_micros).unwrap_or_default(); - let consumer = moq_audio::decode::Consumer::new(&broadcast, &cfg, name, config).await?; + let consumer = self + .driver + .watch(async { Ok(moq_audio::decode::Consumer::new(&broadcast, &cfg, name, config).await?) }) + .await?; Ok(Arc::new(MoqAudioConsumer { - task: Task::new(ConsumerInner { consumer }), + task: Task::new(ConsumerInner { consumer }).supervised(self.driver.clone()), })) } } diff --git a/rs/moq-ffi/src/consumer.rs b/rs/moq-ffi/src/consumer.rs index b9e93db1137..cca2ad2cd15 100644 --- a/rs/moq-ffi/src/consumer.rs +++ b/rs/moq-ffi/src/consumer.rs @@ -3,7 +3,7 @@ use std::sync::{Arc, Mutex}; use std::task::{Poll, ready}; use crate::error::MoqError; -use crate::ffi::{Guard, Task}; +use crate::ffi::{Background, Guard, Task}; use crate::media::*; use crate::producer::MoqTrackInfo; @@ -77,13 +77,19 @@ pub struct MoqBroadcastConsumer { /// The origin this broadcast was resolved through, if any. A catalog rendition may name a /// sibling broadcast, and only an origin can fetch one; a standalone broadcast (a local /// producer's `consume`) has none, so such a rendition is unresolvable rather than wrong. - origin: Option, + pub(crate) origin: Option, + pub(crate) driver: Background, } impl MoqBroadcastConsumer { /// Wrap a standalone broadcast, with no origin to resolve cross-broadcast references against. + #[cfg(test)] pub(crate) fn new(inner: moq_net::broadcast::Consumer) -> Self { - Self { inner, origin: None } + Self { + inner, + origin: None, + driver: Background::default(), + } } /// Wrap a broadcast the origin handed out, so a rendition referencing a sibling broadcast @@ -92,10 +98,15 @@ impl MoqBroadcastConsumer { /// `origin` must be the cursor that named `inner`: the broadcast's path is stamped per /// cursor, and a reference resolves against that path, so an origin rooted elsewhere would /// read a legal reference as escaping (or worse, resolve it to the wrong broadcast). - pub(crate) fn routed(inner: moq_net::broadcast::Consumer, origin: moq_net::origin::Consumer) -> Self { + pub(crate) fn routed( + inner: moq_net::broadcast::Consumer, + origin: moq_net::origin::Consumer, + driver: Background, + ) -> Self { Self { inner, origin: Some(origin), + driver, } } @@ -109,22 +120,26 @@ impl MoqBroadcastConsumer { &self, reference: Option<&str>, ) -> Result { - // Normalize before testing emptiness: this is a caller-supplied string, and one made only - // of slashes normalizes to the empty reference, which names this broadcast. - let reference = reference.map(moq_net::path::Relative::new); - - // An absent or empty reference names the catalog's own broadcast, which we already hold. - // Short-circuiting also keeps a standalone broadcast usable: the common case needs no origin. - let Some(reference) = reference.filter(|reference| !reference.is_empty()) else { - return Ok(self.inner.clone()); - }; - - let origin = self.origin.clone().ok_or_else(|| MoqError::UnresolvableBroadcast { - reason: reference.as_str().to_string(), - })?; - - let source = moq_mux::Source::new(origin, self.inner.clone()); - Ok(source.resolve(Some(&reference)).await?) + self.driver + .watch(async { + // Normalize before testing emptiness: this is a caller-supplied string, and one made only + // of slashes normalizes to the empty reference, which names this broadcast. + let reference = reference.map(moq_net::path::Relative::new); + + // An absent or empty reference names the catalog's own broadcast, which we already hold. + // Short-circuiting also keeps a standalone broadcast usable: the common case needs no origin. + let Some(reference) = reference.filter(|reference| !reference.is_empty()) else { + return Ok(self.inner.clone()); + }; + + let origin = self.origin.clone().ok_or_else(|| MoqError::UnresolvableBroadcast { + reason: reference.as_str().to_string(), + })?; + + let source = moq_mux::Source::new(origin, self.inner.clone()); + Ok(source.resolve(Some(&reference)).await?) + }) + .await } } @@ -147,8 +162,12 @@ impl MoqBroadcastConsumer { pub async fn resolve(&self, reference: Option) -> Result, MoqError> { let broadcast = self.resolve_inner(reference.as_deref()).await?; Ok(Arc::new(match self.origin.clone() { - Some(origin) => Self::routed(broadcast, origin), - None => Self::new(broadcast), + Some(origin) => Self::routed(broadcast, origin, self.driver.clone()), + None => Self { + inner: broadcast, + origin: None, + driver: self.driver.clone(), + }, })) } @@ -161,9 +180,13 @@ impl MoqBroadcastConsumer { name: String, subscription: Option, ) -> Result, MoqError> { - let subscription = subscription.map(moq_net::track::Subscription::from); - let track = self.inner.track(&name)?.subscribe(subscription).await?; - Ok(Arc::new(MoqTrackConsumer::new(track))) + self.driver + .watch(async { + let subscription = subscription.map(moq_net::track::Subscription::from); + let track = self.inner.track(&name)?.subscribe(subscription).await?; + Ok(Arc::new(MoqTrackConsumer::new(track, self.driver.clone()))) + }) + .await } /// Fetch one complete group by track name and group sequence. @@ -177,10 +200,14 @@ impl MoqBroadcastConsumer { sequence: u64, options: Option, ) -> Result, MoqError> { - let options = options.map(moq_net::group::Fetch::from); - let track = self.inner.track(&name).map_err(map_fetch_error)?; - let group = track.fetch_group(sequence, options).await.map_err(map_fetch_error)?; - Ok(Arc::new(MoqGroupConsumer::new(group))) + self.driver + .watch(async { + let options = options.map(moq_net::group::Fetch::from); + let track = self.inner.track(&name).map_err(map_fetch_error)?; + let group = track.fetch_group(sequence, options).await.map_err(map_fetch_error)?; + Ok(Arc::new(MoqGroupConsumer::new(group, self.driver.clone()))) + }) + .await } } @@ -409,10 +436,12 @@ pub struct MoqTrackConsumer { datagrams: Task, control: moq_net::track::Control, info: moq_net::track::Info, + /// The owner whose panic ends this track's reads, handed to every group and reader taken from it. + driver: Background, } impl MoqTrackConsumer { - pub(crate) fn new(track: moq_net::track::Subscriber) -> Self { + pub(crate) fn new(track: moq_net::track::Subscriber, driver: Background) -> Self { let control = track.control(); let info = track.info().clone(); Self { @@ -421,10 +450,11 @@ impl MoqTrackConsumer { track: Cursor::Uncommitted(track), })), }), - groups: Task::new(ReadLane), - datagrams: Task::new(ReadLane), + groups: Task::new(ReadLane).supervised(driver.clone()), + datagrams: Task::new(ReadLane).supervised(driver.clone()), control, info, + driver, } } @@ -432,7 +462,8 @@ impl MoqTrackConsumer { /// /// Errors with [`MoqError::AlreadyCommitted`] once a group read has started, since the reader /// would miss whatever that read consumed. - pub(crate) fn take(&self) -> Result { + pub(crate) fn take(&self) -> Result<(moq_net::track::Subscriber, Background), MoqError> { + self.driver.check()?; let mut guard = self.shared.inner.lock().expect("track reader"); match guard.as_ref().map(|inner| &inner.track) { Some(Cursor::Uncommitted(_)) => {} @@ -448,7 +479,7 @@ impl MoqTrackConsumer { drop(guard); // Wake any datagram read still parked on the subscriber this handle no longer owns. self.cancel_lanes(); - Ok(track) + Ok((track, self.driver.clone())) } fn cancel_lanes(&self) { @@ -484,15 +515,14 @@ impl MoqTrackConsumer { /// [`MoqError::AlreadyCommitted`] afterwards. pub async fn recv_group(&self) -> Result>, MoqError> { let shared = self.shared.clone(); + let driver = self.driver.clone(); self.groups .run(move |lane| { with_lane(lane, async move { - Ok(shared.recv_group().await?.map(|group| { - Arc::new(MoqGroupConsumer { - sequence: group.sequence, - task: Task::new(GroupInner { group }), - }) - })) + Ok(shared + .recv_group() + .await? + .map(|group| Arc::new(MoqGroupConsumer::new(group, driver)))) }) }) .await @@ -509,15 +539,14 @@ impl MoqTrackConsumer { /// ([`Self::recv_group`]) fail with [`MoqError::AlreadyCommitted`] afterwards. pub async fn next_group(&self) -> Result>, MoqError> { let shared = self.shared.clone(); + let driver = self.driver.clone(); self.groups .run(move |lane| { with_lane(lane, async move { - Ok(shared.next_group().await?.map(|group| { - Arc::new(MoqGroupConsumer { - sequence: group.sequence, - task: Task::new(GroupInner { group }), - }) - })) + Ok(shared + .next_group() + .await? + .map(|group| Arc::new(MoqGroupConsumer::new(group, driver)))) }) }) .await @@ -588,13 +617,15 @@ impl GroupInner { pub struct MoqGroupConsumer { sequence: u64, task: Task, + driver: Background, } impl MoqGroupConsumer { - pub(crate) fn new(group: moq_net::group::Consumer) -> Self { + pub(crate) fn new(group: moq_net::group::Consumer, driver: Background) -> Self { Self { sequence: group.sequence, - task: Task::new(GroupInner { group }), + task: Task::new(GroupInner { group }).supervised(driver.clone()), + driver, } } @@ -602,12 +633,12 @@ impl MoqGroupConsumer { /// /// Errors with [`MoqError::AlreadyCommitted`] once a frame has been read, since the reader /// would start mid-group, and [`MoqError::Busy`] while a read is in flight. - pub(crate) fn take(&self) -> Result { + pub(crate) fn take(&self) -> Result<(moq_net::group::Consumer, Background), MoqError> { let inner = self.task.take(|inner| match inner.group.index() { 0 => Ok(()), _ => Err(MoqError::AlreadyCommitted), })?; - Ok(inner.group) + Ok((inner.group, self.driver.clone())) } } diff --git a/rs/moq-ffi/src/error.rs b/rs/moq-ffi/src/error.rs index f2d6ab00291..7eae810040f 100644 --- a/rs/moq-ffi/src/error.rs +++ b/rs/moq-ffi/src/error.rs @@ -201,7 +201,7 @@ pub enum MoqError { #[error("log level: {reason}")] LogLevel { reason: String }, - // Only the native path spawns onto a runtime, so only it can fail to join. + /// An owned native task panicked; cancellation is reported separately. #[cfg(not(target_arch = "wasm32"))] #[error("task: {reason}")] Task { reason: String }, diff --git a/rs/moq-ffi/src/ffi.rs b/rs/moq-ffi/src/ffi.rs index 8d2d08dcfb7..02ff7389bc5 100644 --- a/rs/moq-ffi/src/ffi.rs +++ b/rs/moq-ffi/src/ffi.rs @@ -197,19 +197,167 @@ pub(crate) fn enter() -> EnterGuard { EnterGuard } -/// Spawn a background future, detached from the caller. -/// -/// Uses the runtime handle directly (rather than `tokio::spawn`) because a -/// constructor is called from a foreign thread with no runtime entered. +/// Completion of an owned operation, retained by the object's host calls. +#[derive(Clone, Default)] +pub(crate) struct Background { + #[cfg(not(target_arch = "wasm32"))] + result: Option>>>, +} + +#[cfg(not(target_arch = "wasm32"))] +impl Background { + pub fn new(future: impl Future> + Send + 'static) -> Self { + let result = kio::Producer::new(None); + let receiver = result.consume(); + runtime().spawn(async move { + let completed = moq_tokio::panic::isolate(future) + .await + .unwrap_or(Err(moq_tokio::Error::Panic)); + *result + .write() + .unwrap_or_else(|_| unreachable!("only the task closes its completion")) = Some(completed); + }); + Self { result: Some(receiver) } + } + + pub fn driver(mut driver: impl moq_net::time::Driver + Send + 'static) -> Self { + let result = kio::Producer::new(None); + let receiver = result.consume(); + runtime().spawn(async move { + // Keep channel teardown outside the unwind and after panic publication. + let completed = moq_tokio::panic::isolate(async { + match moq_net::time::run(Borrowed(&mut driver)).await { + moq_net::Error::Closed => Ok(()), + error => Err(error.into()), + } + }) + .await + .unwrap_or(Err(moq_tokio::Error::Panic)); + if matches!(&completed, Err(moq_tokio::Error::Panic)) { + *result + .write() + .unwrap_or_else(|_| unreachable!("only the task closes its completion")) = Some(completed.clone()); + } + let completed = moq_tokio::panic::isolate(async move { + drop(driver); + completed + }) + .await + .unwrap_or(Err(moq_tokio::Error::Panic)); + *result + .write() + .unwrap_or_else(|_| unreachable!("only the task closes its completion")) = Some(completed); + }); + Self { result: Some(receiver) } + } + + pub async fn finished(&self) -> Result<(), MoqError> { + let Some(result) = &self.result else { return Ok(()) }; + let completed = kio::wait(|waiter| { + match std::task::ready!(result.poll(waiter, |state| { + match &**state { + Some(completed) => std::task::Poll::Ready(completed.clone()), + None => std::task::Poll::Pending, + } + })) { + Ok(completed) => std::task::Poll::Ready(completed), + Err(_) => std::task::Poll::Ready(Err(moq_net::Error::Cancel.into())), + } + }) + .await; + completed.map_err(crate::session::map_closed_error) + } +} + +impl Background { + pub fn check(&self) -> Result<(), MoqError> { + #[cfg(not(target_arch = "wasm32"))] + if let Some(result) = &self.result { + return result + .read() + .clone() + .unwrap_or(Ok(())) + .map_err(crate::session::map_closed_error); + } + Ok(()) + } + + pub async fn watch(&self, future: impl Future>) -> Result { + self.check()?; + #[cfg(not(target_arch = "wasm32"))] + let result = if self.result.is_some() { + tokio::pin!(future); + tokio::select! { + biased; + result = self.finished() => { result?; future.await } + result = &mut future => result, + } + } else { + future.await + }; + #[cfg(target_arch = "wasm32")] + let result = future.await; + self.check()?; + result + } +} + +#[cfg(all(test, not(target_arch = "wasm32")))] +pub(crate) fn panicking_origin() -> (moq_net::origin::Producer, Background, kio::Producer) { + struct Driver { + inner: moq_net::origin::Driver, + failed: kio::Consumer, + } + impl moq_net::time::Driver for Driver { + fn poll( + &mut self, + now: moq_net::time::Instant, + waiter: &kio::Waiter, + ) -> Result, moq_net::Error> { + let _ = self.failed.poll(waiter, |state| { + assert!(!**state, "injected origin driver panic"); + std::task::Poll::<()>::Pending + }); + self.inner.poll(now, waiter) + } + } + let (origin, inner) = moq_net::origin::Producer::new(Default::default()); + let failed = kio::Producer::new(false); + let driver = Background::driver(Driver { + inner, + failed: failed.consume(), + }); + (origin, driver, failed) +} + +#[cfg(not(target_arch = "wasm32"))] +struct Borrowed<'a, D>(&'a mut D); + +#[cfg(not(target_arch = "wasm32"))] +impl moq_net::time::Driver for Borrowed<'_, D> { + fn poll( + &mut self, + now: moq_net::time::Instant, + waiter: &kio::Waiter, + ) -> Result, moq_net::Error> { + self.0.poll(now, waiter) + } +} + +/// Spawn an owned background future on the native runtime. #[cfg(not(target_arch = "wasm32"))] -pub(crate) fn spawn + Send + 'static>(future: F) { - runtime().spawn(future); +pub(crate) fn spawn + Send + 'static>(future: F) -> Background { + Background::new(async move { + future.await; + Ok(()) + }) } -/// Spawn onto the JS event loop: wasm32 has no other thread to hand it to. +/// Spawn onto the JS event loop, whose current toolchain aborts on panic. #[cfg(target_arch = "wasm32")] -pub(crate) fn spawn + 'static>(future: F) { +pub(crate) fn spawn + 'static>(future: F) -> Background { web_async::spawn(future); + Background::default() } /// Aborts a spawned task when the future awaiting it is dropped. @@ -234,10 +382,14 @@ where T: Send + 'static, E: Into + Send + 'static, { - let task = runtime().spawn(future); + let task = runtime().spawn(async move { + moq_tokio::panic::isolate(async move { future.await.map_err(Into::into) }) + .await + .unwrap_or_else(|| Err(crate::session::map_closed_error(moq_tokio::Error::Panic))) + }); let _abort = AbortOnDrop(task.abort_handle()); match task.await { - Ok(result) => result.map_err(Into::into), + Ok(result) => result, Err(e) if e.is_cancelled() => Err(MoqError::Cancelled), Err(e) => Err(e.into()), } @@ -264,6 +416,8 @@ pub(crate) struct Task { /// unrepresentable rather than merely documented. state: Arc>>, cancel: tokio::sync::watch::Sender, + cleanup: std::sync::Mutex>, + driver: Background, } /// Exclusive access to a [Task]'s state, held for as long as the guard lives. @@ -274,9 +428,16 @@ impl Task { Self { state: Arc::new(tokio::sync::Mutex::new(Some(inner))), cancel: tokio::sync::watch::Sender::new(false), + cleanup: Default::default(), + driver: Background::default(), } } + pub fn supervised(mut self, driver: Background) -> Self { + self.driver = driver; + self + } + /// Try to lock the state synchronously. Returns `None` if a task is running or it was cancelled. // Only the native server and its requests read state synchronously. #[cfg(not(target_arch = "wasm32"))] @@ -309,6 +470,10 @@ impl Task { /// [`Self::cancel`]. #[cfg(not(target_arch = "wasm32"))] pub fn configure(&self) -> Result, MoqError> { + self.driver.check()?; + if let Some(cleanup) = self.cleanup.lock().unwrap().as_ref() { + cleanup.check()?; + } match self.lock() { Some(guard) => Ok(guard), None if self.is_cancelled() => Err(MoqError::Cancelled), @@ -330,22 +495,37 @@ impl Task { F: FnOnce(Guard) -> Fut, Fut: Future>, { - let drive = Self::drive(self.cancel.subscribe(), self.state.clone(), f); - let mut run = std::pin::pin!(async { - tokio::select! { - biased; - _ = stopped() => Err(MoqError::Cancelled), - result = drive => result, - } - }); - std::future::poll_fn(|cx| { - let Some(_polling) = Polling::start() else { - return std::task::Poll::Ready(Err(MoqError::Cancelled)); - }; - let _runtime = enter(); - run.as_mut().poll(cx) - }) - .await + self.driver.check()?; + if let Some(cleanup) = self.cleanup.lock().unwrap().as_ref() { + cleanup.check()?; + } + let result = { + let drive = self + .driver + .watch(Self::drive(self.cancel.subscribe(), self.state.clone(), f)); + let mut run = std::pin::pin!(async { + tokio::select! { + biased; + _ = stopped() => Err(MoqError::Cancelled), + result = drive => result, + } + }); + std::future::poll_fn(|cx| { + let Some(_polling) = Polling::start() else { + return std::task::Poll::Ready(Err(MoqError::Cancelled)); + }; + let _runtime = enter(); + run.as_mut().poll(cx) + }) + .await + }; + // A foreign future may stop polling while holding the state, so cancellation cannot + // wait for cleanup. Its next call still reports a completed cleanup panic. + self.driver.check()?; + if let Some(cleanup) = self.cleanup.lock().unwrap().as_ref() { + cleanup.check()?; + } + result } /// Wait for the lock, then run `f`, with [Self::cancel] able to interrupt either. @@ -380,6 +560,10 @@ impl Task { /// [`MoqError::Busy`] while a [Self::run] owns the state, [`MoqError::Cancelled`] after /// [Self::cancel]. pub fn take(&self, check: impl FnOnce(&T) -> Result<(), MoqError>) -> Result { + self.driver.check()?; + if let Some(cleanup) = self.cleanup.lock().unwrap().as_ref() { + cleanup.check()?; + } if *self.cancel.borrow() { return Err(MoqError::Cancelled); } @@ -395,8 +579,9 @@ impl Task { Ok(inner) } - /// Cancel all current and future [Self::run] calls, causing them to return [MoqError::Cancelled], - /// and drop the state. + /// Cancel all current and future [Self::run] calls and drop the state. + /// + /// Calls return [MoqError::Cancelled]; later calls report a completed cleanup panic. /// /// Terminal: there is no way back from here, and the state is gone rather than idle. /// The drop is not synchronous with this call, because an in-flight [Self::run] holds the @@ -406,15 +591,16 @@ impl Task { // send_replace, not send: `send` refuses to store the value while no // receiver exists, so a cancel BEFORE the first `run` would silently // no-op and the run would proceed. + let mut cleanup = self.cleanup.lock().unwrap(); if self.cancel.send_replace(true) { // Already cancelled, so the first call is the one dropping the state. return; } let state = self.state.clone(); - spawn(async move { + *cleanup = Some(spawn(async move { state.lock().await.take(); - }); + })); } } @@ -426,6 +612,45 @@ impl Drop for Task { #[cfg(all(test, not(target_arch = "wasm32")))] mod tests { + + #[tokio::test] + async fn a_driver_teardown_panic_reaches_its_owner() { + struct Driver; + impl moq_net::time::Driver for Driver { + fn poll( + &mut self, + _: moq_net::time::Instant, + _: &kio::Waiter, + ) -> Result, moq_net::Error> { + Err(moq_net::Error::Closed) + } + } + impl Drop for Driver { + fn drop(&mut self) { + panic!("injected driver teardown panic"); + } + } + let driver = super::Background::driver(Driver); + assert!(matches!(driver.finished().await, Err(super::MoqError::Task { .. }))); + } + #[tokio::test] + async fn a_cleanup_panic_reaches_the_host_while_another_task_keeps_working() { + struct Panics; + impl Drop for Panics { + fn drop(&mut self) { + panic!("injected cleanup panic"); + } + } + let failed = super::Task::new(Panics); + failed.cancel(); + let cleanup = failed.cleanup.lock().unwrap().clone().unwrap(); + assert!(matches!(cleanup.finished().await, Err(super::MoqError::Task { .. }))); + let result = failed.run(|_| async { Ok(()) }).await; + assert!(matches!(result, Err(super::MoqError::Task { .. })), "{result:?}"); + let healthy = super::Task::new(7); + assert_eq!(healthy.run(|state| async move { Ok(*state) }).await.unwrap(), 7); + } + use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; use std::time::Duration; @@ -547,6 +772,30 @@ mod tests { soon(dropped).await.expect("state should be dropped"); } + #[tokio::test] + async fn a_queued_cancel_does_not_wait_for_an_unpolled_owner() { + let task = Task::new(()); + let first = task.run(|state| async move { + let _state = state; + std::future::pending::>().await + }); + tokio::pin!(first); + assert!( + std::future::poll_fn(|cx| std::task::Poll::Ready(first.as_mut().poll(cx))) + .await + .is_pending() + ); + task.cancel(); + let queued = task.run(|_| async { panic!("cancelled calls never touch the state") }); + tokio::pin!(queued); + let result: std::task::Poll> = + std::future::poll_fn(|cx| std::task::Poll::Ready(queued.as_mut().poll(cx))).await; + assert!( + matches!(result, std::task::Poll::Ready(Err(MoqError::Cancelled))), + "{result:?}" + ); + } + #[tokio::test] async fn run_after_cancel_is_cancelled_without_touching_the_state() { let (task, dropped) = tracked(); diff --git a/rs/moq-ffi/src/json.rs b/rs/moq-ffi/src/json.rs index 12577040dab..429c7963361 100644 --- a/rs/moq-ffi/src/json.rs +++ b/rs/moq-ffi/src/json.rs @@ -194,9 +194,10 @@ impl MoqJsonSnapshotConsumer { #[uniffi::constructor] pub fn new(track: &MoqTrackConsumer, config: MoqJsonSnapshotConfig) -> Result, MoqError> { let _guard = crate::ffi::enter(); - let consumer = moq_json::snapshot::Consumer::::new(track.take()?, config.into()); + let (track, driver) = track.take()?; + let consumer = moq_json::snapshot::Consumer::::new(track, config.into()); Ok(Arc::new(Self { - task: Task::new(SnapshotConsumer { inner: consumer }), + task: Task::new(SnapshotConsumer { inner: consumer }).supervised(driver), })) } @@ -306,9 +307,10 @@ impl MoqJsonStreamConsumer { #[uniffi::constructor] pub fn new(track: &MoqTrackConsumer, config: MoqJsonStreamConfig) -> Result, MoqError> { let _guard = crate::ffi::enter(); - let consumer = moq_json::stream::Consumer::::new(track.take()?, config.into()); + let (track, driver) = track.take()?; + let consumer = moq_json::stream::Consumer::::new(track, config.into()); Ok(Arc::new(Self { - task: Task::new(StreamConsumer { inner: consumer }), + task: Task::new(StreamConsumer { inner: consumer }).supervised(driver), })) } diff --git a/rs/moq-ffi/src/media.rs b/rs/moq-ffi/src/media.rs index 23c748352d7..76c6f425b8b 100644 --- a/rs/moq-ffi/src/media.rs +++ b/rs/moq-ffi/src/media.rs @@ -344,16 +344,25 @@ impl Drop for MoqMediaTimebase { /// Reserve a container's catalog slot under the owner lock, from `timebase` when shared, so the /// importer is constructed after the lock is released. +/// +/// Also returns the broadcast's driver, which the importer checks so a driver panic reaches it. fn reserve_container( broadcast: &MoqBroadcastProducer, timebase: Option<&MoqMediaTimebase>, -) -> Result<(moq_net::broadcast::Producer, moq_mux::catalog::Reserved), MoqError> { +) -> Result< + ( + moq_net::broadcast::Producer, + moq_mux::catalog::Reserved, + crate::ffi::Background, + ), + MoqError, +> { broadcast.with_state(|state| { let reserved = match timebase { Some(timebase) => timebase.reserve(broadcast, state)?, None => state.catalog.reserve(), }; - Ok((state.broadcast.clone(), reserved)) + Ok((state.broadcast.clone(), reserved, state.driver.clone())) }) } @@ -721,12 +730,14 @@ struct ContainerStreamProducer { /// Imports complete container chunks into a broadcast. #[derive(uniffi::Object)] pub struct MoqMediaContainerProducer { + driver: crate::ffi::Background, inner: std::sync::Mutex>, } /// Imports a container byte stream into a broadcast. #[derive(uniffi::Object)] pub struct MoqMediaContainerStreamProducer { + driver: crate::ffi::Background, inner: std::sync::Mutex>, } @@ -754,17 +765,17 @@ impl MoqMediaTrackProducer { init: MoqAudioInit, ) -> Result, MoqError> { let _guard = crate::ffi::enter(); - let guard = broadcast.state.lock().unwrap(); - let state = guard.as_ref().ok_or(MoqError::Closed)?; - - let init: moq_mux::import::AudioInit = init.into(); - let request = target.resolve(&state.broadcast, &init.format)?; + broadcast.with_state(|state| { + let init: moq_mux::import::AudioInit = init.into(); + let request = target.resolve(&state.broadcast, &init.format)?; - let import = - moq_mux::import::Track::audio(request, state.catalog.reserve(), init).map_err(|err| MoqError::Codec { - reason: format!("init failed: {err}"), + let import = moq_mux::import::Track::audio(request, state.catalog.reserve(), init).map_err(|err| { + MoqError::Codec { + reason: format!("init failed: {err}"), + } })?; - Ok(MoqMediaTrackProducer::from_import(import)) + Ok(MoqMediaTrackProducer::from_import(import)) + }) } } @@ -781,17 +792,17 @@ impl MoqMediaTrackProducer { init: MoqVideoInit, ) -> Result, MoqError> { let _guard = crate::ffi::enter(); - let guard = broadcast.state.lock().unwrap(); - let state = guard.as_ref().ok_or(MoqError::Closed)?; - - let init: moq_mux::import::VideoInit = init.into(); - let request = target.resolve(&state.broadcast, &init.format)?; + broadcast.with_state(|state| { + let init: moq_mux::import::VideoInit = init.into(); + let request = target.resolve(&state.broadcast, &init.format)?; - let import = - moq_mux::import::Track::video(request, state.catalog.reserve(), init).map_err(|err| MoqError::Codec { - reason: format!("init failed: {err}"), + let import = moq_mux::import::Track::video(request, state.catalog.reserve(), init).map_err(|err| { + MoqError::Codec { + reason: format!("init failed: {err}"), + } })?; - Ok(MoqMediaTrackProducer::from_import(import)) + Ok(MoqMediaTrackProducer::from_import(import)) + }) } } @@ -807,12 +818,13 @@ impl MoqMediaContainerProducer { ) -> Result, MoqError> { let _guard = crate::ffi::enter(); let native = moq_mux::import::ContainerInit::new(init.format.into(), init.data); - let (target, reserved) = reserve_container(broadcast, init.timebase.as_deref())?; + let (target, reserved, driver) = reserve_container(broadcast, init.timebase.as_deref())?; let import = moq_mux::import::Container::new(target, reserved, &native).map_err(|err| MoqError::Codec { reason: format!("init failed: {err}"), })?; Ok(Arc::new(MoqMediaContainerProducer { + driver, inner: std::sync::Mutex::new(Some(ContainerProducer { import })), })) } @@ -831,23 +843,23 @@ impl MoqMediaTrackStreamProducer { init: MoqVideoInit, ) -> Result, MoqError> { let _guard = crate::ffi::enter(); - let guard = broadcast.state.lock().unwrap(); - let state = guard.as_ref().ok_or(MoqError::Closed)?; - - let init: moq_mux::import::VideoInit = init.into(); - let request = target.resolve(&state.broadcast, &init.format)?; + broadcast.with_state(|state| { + let init: moq_mux::import::VideoInit = init.into(); + let request = target.resolve(&state.broadcast, &init.format)?; - let import = moq_mux::import::TrackStream::video(request, state.catalog.reserve(), init).map_err(|err| { - MoqError::Codec { - reason: format!("init failed: {err}"), - } - })?; + let import = + moq_mux::import::TrackStream::video(request, state.catalog.reserve(), init).map_err(|err| { + MoqError::Codec { + reason: format!("init failed: {err}"), + } + })?; - Ok(Arc::new(MoqMediaTrackStreamProducer { - inner: std::sync::Mutex::new(Some(MediaStreamProducer { - import: Box::new(import), - })), - })) + Ok(Arc::new(MoqMediaTrackStreamProducer { + inner: std::sync::Mutex::new(Some(MediaStreamProducer { + import: Box::new(import), + })), + })) + }) } } @@ -862,7 +874,7 @@ impl MoqMediaContainerStreamProducer { init: MoqContainerStreamInit, ) -> Result, MoqError> { let _guard = crate::ffi::enter(); - let (target, reserved) = reserve_container(broadcast, init.timebase.as_deref())?; + let (target, reserved, driver) = reserve_container(broadcast, init.timebase.as_deref())?; let import = moq_mux::import::ContainerStream::new(target, reserved, init.format.into()).map_err(|err| { MoqError::Codec { reason: format!("init failed: {err}"), @@ -870,6 +882,7 @@ impl MoqMediaContainerStreamProducer { })?; Ok(Arc::new(MoqMediaContainerStreamProducer { + driver, inner: std::sync::Mutex::new(Some(ContainerStreamProducer { import })), })) } @@ -995,13 +1008,18 @@ impl MoqMediaContainerProducer { /// No timestamp: a container carries its tracks' timing itself, and the importer reads it out /// rather than taking the caller's word for it. pub fn write(&self, payload: Vec) -> Result<(), MoqError> { - let _guard = crate::ffi::enter(); - let mut guard = self.inner.lock().unwrap(); - let container = guard.as_mut().ok_or(MoqError::Closed)?; - container.import.decode(&payload).map_err(|err| MoqError::Codec { - reason: format!("decode failed: {err}"), - })?; - Ok(()) + self.driver.check()?; + let result = (|| { + let _guard = crate::ffi::enter(); + let mut guard = self.inner.lock().unwrap(); + let container = guard.as_mut().ok_or(MoqError::Closed)?; + container.import.decode(&payload).map_err(|err| MoqError::Codec { + reason: format!("decode failed: {err}"), + })?; + Ok(()) + })(); + self.driver.check()?; + result } /// Declare that the next chunk starts a new segment, rolling a group on every track this @@ -1011,33 +1029,47 @@ impl MoqMediaContainerProducer { /// `styp` atoms declares its own, so this is only needed when it doesn't, and formats with no /// segment concept (MKV, TS, FLV) ignore it. pub fn cut(&self) -> Result<(), MoqError> { - let _guard = crate::ffi::enter(); - let mut guard = self.inner.lock().unwrap(); - let container = guard.as_mut().ok_or(MoqError::Closed)?; - container.import.cut(); - Ok(()) + self.driver.check()?; + let result = (|| { + let _guard = crate::ffi::enter(); + let mut guard = self.inner.lock().unwrap(); + let container = guard.as_mut().ok_or(MoqError::Closed)?; + container.import.cut(); + Ok(()) + })(); + self.driver.check()?; + result } /// Start a new segment and number its groups `sequence`. pub fn seek(&self, sequence: u64) -> Result<(), MoqError> { - let _guard = crate::ffi::enter(); - let mut guard = self.inner.lock().unwrap(); - let container = guard.as_mut().ok_or(MoqError::Closed)?; - container.import.seek(sequence).map_err(|err| MoqError::Codec { - reason: format!("seek failed: {err}"), - })?; - Ok(()) + self.driver.check()?; + let result = (|| { + let _guard = crate::ffi::enter(); + let mut guard = self.inner.lock().unwrap(); + let container = guard.as_mut().ok_or(MoqError::Closed)?; + container.import.seek(sequence).map_err(|err| MoqError::Codec { + reason: format!("seek failed: {err}"), + })?; + Ok(()) + })(); + self.driver.check()?; + result } /// Finish every track this container publishes. pub fn finish(&self) -> Result<(), MoqError> { - let _guard = crate::ffi::enter(); - let mut guard = self.inner.lock().unwrap(); - let mut container = guard.take().ok_or(MoqError::Closed)?; - container.import.finish().map_err(|err| MoqError::Codec { - reason: format!("finish failed: {err}"), - })?; - Ok(()) + let result = (|| { + let _guard = crate::ffi::enter(); + let mut guard = self.inner.lock().unwrap(); + let mut container = guard.take().ok_or(MoqError::Closed)?; + container.import.finish().map_err(|err| MoqError::Codec { + reason: format!("finish failed: {err}"), + })?; + Ok(()) + })(); + self.driver.check()?; + result } } @@ -1085,24 +1117,33 @@ impl MoqMediaContainerStreamProducer { /// Push raw container bytes. The importer recovers its own framing, so callers can write /// arbitrary chunks. pub fn write(&self, payload: Vec) -> Result<(), MoqError> { - let _guard = crate::ffi::enter(); - let mut guard = self.inner.lock().unwrap(); - let container = guard.as_mut().ok_or(MoqError::Closed)?; - container.import.decode(&payload).map_err(|err| MoqError::Codec { - reason: format!("decode failed: {err}"), - })?; - Ok(()) + self.driver.check()?; + let result = (|| { + let _guard = crate::ffi::enter(); + let mut guard = self.inner.lock().unwrap(); + let container = guard.as_mut().ok_or(MoqError::Closed)?; + container.import.decode(&payload).map_err(|err| MoqError::Codec { + reason: format!("decode failed: {err}"), + })?; + Ok(()) + })(); + self.driver.check()?; + result } /// Finish every track this container publishes. pub fn finish(&self) -> Result<(), MoqError> { - let _guard = crate::ffi::enter(); - let mut guard = self.inner.lock().unwrap(); - let mut container = guard.take().ok_or(MoqError::Closed)?; - container.import.finish().map_err(|err| MoqError::Codec { - reason: format!("finish failed: {err}"), - })?; - Ok(()) + let result = (|| { + let _guard = crate::ffi::enter(); + let mut guard = self.inner.lock().unwrap(); + let mut container = guard.take().ok_or(MoqError::Closed)?; + container.import.finish().map_err(|err| MoqError::Codec { + reason: format!("finish failed: {err}"), + })?; + Ok(()) + })(); + self.driver.check()?; + result } } @@ -1226,11 +1267,15 @@ pub struct MoqMediaGroupConsumer { } impl MoqMediaGroupConsumer { - fn from_inner(group: moq_net::group::Consumer, container: moq_mux::catalog::hang::Container) -> Self { + fn from_inner( + group: moq_net::group::Consumer, + container: moq_mux::catalog::hang::Container, + driver: crate::ffi::Background, + ) -> Self { let inner = moq_mux::container::GroupConsumer::new(group, container); Self { sequence: inner.sequence(), - task: Task::new(MediaGroupInner { inner }), + task: Task::new(MediaGroupInner { inner }).supervised(driver), } } } @@ -1295,13 +1340,18 @@ impl MoqMediaCatalogConsumer { #[uniffi::constructor] pub async fn subscribe(broadcast: &MoqBroadcastConsumer) -> Result, MoqError> { let track = broadcast - .inner - .track(hang::catalog::Catalog::DEFAULT_NAME)? - .subscribe(hang::catalog::Catalog::default_subscription()) + .driver + .watch(async { + Ok(broadcast + .inner + .track(hang::catalog::Catalog::DEFAULT_NAME)? + .subscribe(hang::catalog::Catalog::default_subscription()) + .await?) + }) .await?; let consumer = moq_mux::catalog::hang::Consumer::from(track); Ok(Arc::new(MoqMediaCatalogConsumer { - task: Task::new(Catalog { inner: consumer }), + task: Task::new(Catalog { inner: consumer }).supervised(broadcast.driver.clone()), })) } } @@ -1323,10 +1373,13 @@ impl MoqMediaTrackConsumer { // subscription if init parsing fails. let media = media_container(container)?; let subscription = subscription.map(moq_net::track::Subscription::from).unwrap_or_default(); - let track = broadcast.inner.track(&name)?.subscribe(subscription).await?; + let track = broadcast + .driver + .watch(async { Ok(broadcast.inner.track(&name)?.subscribe(subscription).await?) }) + .await?; let consumer = moq_mux::container::Consumer::new(track, media); Ok(Arc::new(MoqMediaTrackConsumer { - task: Task::new(Media { inner: consumer }), + task: Task::new(Media { inner: consumer }).supervised(broadcast.driver.clone()), })) } } @@ -1344,7 +1397,8 @@ impl MoqMediaGroupConsumer { pub fn new(group: &MoqGroupConsumer, container: MoqContainer) -> Result, MoqError> { // Parse the container first, so a bad init leaves the raw group readable. let media = media_container(container)?; - Ok(Arc::new(Self::from_inner(group.take()?, media))) + let (group, driver) = group.take()?; + Ok(Arc::new(Self::from_inner(group, media, driver))) } /// Fetch and decode exactly one media group. @@ -1363,9 +1417,18 @@ impl MoqMediaGroupConsumer { // dynamic group request waiting for a consumer that can never read it. let media = media_container(container)?; let options = options.map(moq_net::group::Fetch::from); - let track = broadcast.inner.track(&name).map_err(map_fetch_error)?; - let group = track.fetch_group(sequence, options).await.map_err(map_fetch_error)?; - Ok(Arc::new(MoqMediaGroupConsumer::from_inner(group, media))) + let group = broadcast + .driver + .watch(async { + let track = broadcast.inner.track(&name).map_err(map_fetch_error)?; + track.fetch_group(sequence, options).await.map_err(map_fetch_error) + }) + .await?; + Ok(Arc::new(MoqMediaGroupConsumer::from_inner( + group, + media, + broadcast.driver.clone(), + ))) } } @@ -1447,8 +1510,13 @@ mod timebase_tests { async fn containers_of_one_source_keep_their_relative_media_time() { for shared in [false, true] { // Construct under the paused runtime, before exported entry points enter their worker. - let broadcast = - Arc::new(MoqBroadcastProducer::from_inner(moq_net::broadcast::Info::new().produce()).unwrap()); + let broadcast = Arc::new( + MoqBroadcastProducer::from_inner( + moq_net::broadcast::Info::new().produce(), + crate::ffi::Background::default(), + ) + .unwrap(), + ); let catalog = broadcast.with_state(|state| Ok(state.catalog.clone())).unwrap(); let clock = catalog.clock(); let consumer = broadcast.consume_inner().unwrap(); diff --git a/rs/moq-ffi/src/origin.rs b/rs/moq-ffi/src/origin.rs index 47eb40da433..191813ac8e6 100644 --- a/rs/moq-ffi/src/origin.rs +++ b/rs/moq-ffi/src/origin.rs @@ -2,7 +2,7 @@ use std::sync::Arc; use crate::consumer::MoqBroadcastConsumer; use crate::error::MoqError; -use crate::ffi::Task; +use crate::ffi::{Background, Task}; use crate::producer::MoqBroadcastProducer; /// Config used when creating an origin. @@ -114,12 +114,26 @@ pub fn moq_epoch_time_ms(epoch: String) -> Result { #[derive(uniffi::Object)] pub struct MoqOriginProducer { - inner: moq_net::origin::Producer, + inner: Origin, +} + +#[derive(Clone)] +pub(crate) struct Origin { + pub inner: moq_net::origin::Producer, + pub driver: Background, +} + +impl std::ops::Deref for Origin { + type Target = moq_net::origin::Producer; + fn deref(&self) -> &Self::Target { + &self.inner + } } #[derive(uniffi::Object)] pub struct MoqOriginConsumer { inner: moq_net::origin::Consumer, + pub(crate) driver: Background, } #[derive(uniffi::Object)] @@ -135,6 +149,7 @@ pub struct MoqAnnounceConsumer { /// `MoqOriginProducer` is gone. pub struct MoqOriginDynamic { slot: Slot, + driver: Background, task: std::sync::Mutex>>>, } @@ -146,6 +161,7 @@ pub struct MoqBroadcastRequest { struct Announced { inner: moq_net::announce::Consumer, + driver: Background, } /// The served route behind one `MoqOriginProducer::dynamic` handle, shared @@ -155,26 +171,34 @@ type Slot = Arc>>; struct OriginDynamic { slot: Slot, + driver: Background, } impl OriginDynamic { async fn requested_broadcast(&mut self) -> Result, MoqError> { - let request = kio::wait(|waiter| match self.slot.lock().unwrap().as_ref() { - Some(dynamic) => dynamic.poll_requested_broadcast(waiter), - None => std::task::Poll::Ready(Err(moq_net::Error::Closed)), - }) - .await - .map_err(|err| match err { - moq_net::Error::Closed => MoqError::Closed, - other => other.into(), - })?; + let request = self + .driver + .watch(async { + kio::wait(|waiter| match self.slot.lock().unwrap().as_ref() { + Some(dynamic) => dynamic.poll_requested_broadcast(waiter), + None => std::task::Poll::Ready(Err(moq_net::Error::Closed)), + }) + .await + .map_err(|err| match err { + moq_net::Error::Closed => MoqError::Closed, + other => other.into(), + }) + }) + .await?; Ok(Arc::new(MoqBroadcastRequest::new(request))) } } impl Announced { async fn next(&mut self) -> Result, MoqError> { - Ok(self.inner.next().await.map(Into::into)) + self.driver + .watch(async { Ok(self.inner.next().await.map(Into::into)) }) + .await } } @@ -182,6 +206,7 @@ impl Announced { struct AnnouncedBroadcast { origin: moq_net::origin::Consumer, path: moq_net::PathOwned, + driver: Background, } impl AnnouncedBroadcast { @@ -189,8 +214,15 @@ impl AnnouncedBroadcast { // `routed_broadcast` rides out the churn between something covering the path // and the path actually resolving (failover, an advertise-only announce // racing its handler). - let broadcast = self.origin.routed_broadcast(&self.path).await?; - Ok(Arc::new(MoqBroadcastConsumer::routed(broadcast, self.origin.clone()))) + let broadcast = self + .driver + .watch(async { Ok(self.origin.routed_broadcast(&self.path).await?) }) + .await?; + Ok(Arc::new(MoqBroadcastConsumer::routed( + broadcast, + self.origin.clone(), + self.driver.clone(), + ))) } } @@ -262,13 +294,12 @@ pub struct MoqAnnouncedBroadcast { } impl MoqOriginProducer { - pub(crate) fn inner(&self) -> &moq_net::origin::Producer { + pub(crate) fn inner(&self) -> &Origin { &self.inner } - /// Wrap an existing `moq_net::origin::Producer` (e.g. one auto-created - /// during `MoqClient::connect`) so it can cross the FFI boundary. - pub(crate) fn from_inner(inner: moq_net::origin::Producer) -> Self { + /// Wrap an owned origin so it can cross the FFI boundary. + pub(crate) fn from_inner(inner: Origin) -> Self { Self { inner } } @@ -285,18 +316,24 @@ impl MoqOriginProducer { } } -/// Build an origin producer, spawning its driver on the FFI runtime. -pub(crate) fn spawn(config: moq_net::origin::Config) -> moq_net::origin::Producer { - let (producer, driver) = moq_net::origin::Producer::new(config); - crate::ffi::spawn(async move { +/// Build an origin whose host handles retain its driver's completion. +pub(crate) fn spawn(config: moq_net::origin::Config) -> Origin { + let (inner, driver) = moq_net::origin::Producer::new(config); + #[cfg(not(target_arch = "wasm32"))] + let driver = Background::driver(driver); + #[cfg(target_arch = "wasm32")] + let driver = crate::ffi::spawn(async move { moq_net::time::run(driver).await; }); - producer + Origin { inner, driver } } impl MoqOriginConsumer { - pub(crate) fn from_inner(inner: moq_net::origin::Consumer) -> Self { - Self { inner } + pub(crate) fn from_inner(origin: Origin) -> Self { + Self { + inner: origin.inner.consume(), + driver: origin.driver, + } } } @@ -308,7 +345,7 @@ impl MoqOriginConsumer { pub(crate) fn resolve_pair( publish: Option<&Arc>, consume: Option<&Arc>, -) -> (moq_net::origin::Producer, moq_net::origin::Producer) { +) -> (Origin, Origin) { if publish.is_none() && consume.is_none() { // Clones of a Producer share the underlying origin, so this is one origin, not two. let shared = spawn(moq_net::origin::Config::default()); @@ -335,9 +372,7 @@ impl MoqOriginProducer { /// Create a consumer for this origin. pub fn consume(&self) -> Arc { let _guard = crate::ffi::enter(); - Arc::new(MoqOriginConsumer { - inner: self.inner.consume(), - }) + Arc::new(MoqOriginConsumer::from_inner(self.inner.clone())) } /// Advertise `prefix` and serve the requests beneath it. @@ -350,12 +385,19 @@ impl MoqOriginProducer { /// demand, populate, then announce. pub fn dynamic(&self, prefix: String, route: MoqRoute) -> Result, MoqError> { let _guard = crate::ffi::enter(); + self.inner.driver.check()?; let route: moq_net::origin::Route = route.try_into()?; - let dynamic = self.inner.dynamic(prefix, route)?; + let result = self.inner.dynamic(prefix, route); + self.inner.driver.check()?; + let dynamic = result?; let slot = Arc::new(std::sync::Mutex::new(Some(dynamic))); Ok(Arc::new(MoqOriginDynamic { slot: slot.clone(), - task: std::sync::Mutex::new(Some(Arc::new(Task::new(OriginDynamic { slot })))), + driver: self.inner.driver.clone(), + task: std::sync::Mutex::new(Some(Arc::new(Task::new(OriginDynamic { + slot, + driver: self.inner.driver.clone(), + })))), })) } @@ -371,9 +413,15 @@ impl MoqOriginProducer { /// `dynamic()` included, does the same. pub fn create_broadcast(&self, path: String) -> Result, MoqError> { let _guard = crate::ffi::enter(); + self.inner.driver.check()?; // Surfaces Error::Unauthorized (out of scope) via the MoqError::Protocol conversion. - let broadcast = self.inner.create_broadcast(path.as_str())?; - Ok(Arc::new(MoqBroadcastProducer::from_inner(broadcast)?)) + let result = self.inner.create_broadcast(path.as_str()); + self.inner.driver.check()?; + let broadcast = result?; + Ok(Arc::new(MoqBroadcastProducer::from_inner( + broadcast, + self.inner.driver.clone(), + )?)) } } @@ -382,6 +430,7 @@ impl MoqOriginConsumer { /// Subscribe to routes matching a pattern scope; updates stay relative to the origin. pub fn announced(&self, config: MoqAnnounceConfig) -> Result, MoqError> { let _guard = crate::ffi::enter(); + self.driver.check()?; let filter = match config.filter { Some(filter) => filter.parse::()?, None => moq_net::Pattern::all(), @@ -394,6 +443,7 @@ impl MoqOriginConsumer { Ok(Arc::new(MoqAnnounceConsumer { task: Task::new(Announced { inner: origin.announced(), + driver: self.driver.clone(), }), })) } @@ -406,6 +456,7 @@ impl MoqOriginConsumer { /// remote one. pub fn announced_broadcast(&self, path: String) -> Result, MoqError> { let _guard = crate::ffi::enter(); + self.driver.check()?; let path = moq_net::Path::new(&path).to_owned(); // Probe the permission eagerly so an unreachable path fails here, rather than @@ -421,6 +472,7 @@ impl MoqOriginConsumer { // it its own root, and a catalog's `../sibling` reference would read as escaping. origin: self.inner.clone(), path, + driver: self.driver.clone(), }), })) } @@ -437,8 +489,15 @@ impl MoqOriginConsumer { /// Calling this straight after connecting therefore races the session's announcements /// and can report a live broadcast as unroutable. Await `announced_broadcast` first. pub async fn request_broadcast(&self, path: String) -> Result, MoqError> { - let broadcast = self.inner.request_broadcast(path.as_str(), None).await?; - Ok(Arc::new(MoqBroadcastConsumer::routed(broadcast, self.inner.clone()))) + let broadcast = self + .driver + .watch(async { Ok(self.inner.request_broadcast(path.as_str(), None).await?) }) + .await?; + Ok(Arc::new(MoqBroadcastConsumer::routed( + broadcast, + self.inner.clone(), + self.driver.clone(), + ))) } } @@ -459,11 +518,14 @@ impl MoqOriginDynamic { /// Replace the route in place: its epoch, hops, and a static cost. The prefix cannot /// change; call `dynamic` again instead. pub fn update(&self, route: MoqRoute) -> Result<(), MoqError> { + self.driver.check()?; let _guard = crate::ffi::enter(); let route: moq_net::origin::Route = route.try_into()?; let slot = self.slot.lock().unwrap(); let dynamic = slot.as_ref().ok_or(MoqError::Closed)?; - Ok(dynamic.update(route)?) + let result = dynamic.update(route); + self.driver.check()?; + Ok(result?) } /// Stop serving and retract the route. Terminal: this handler is released @@ -555,3 +617,171 @@ impl MoqAnnouncedBroadcast { self.task.cancel(); } } + +#[cfg(all(test, not(target_arch = "wasm32")))] +mod supervision_tests { + use super::*; + + #[tokio::test] + async fn a_panicked_origin_task_reaches_the_host_while_another_origin_keeps_working() { + let (inner, driver, trigger) = crate::ffi::panicking_origin(); + let failed = MoqOriginProducer::from_inner(Origin { inner, driver }); + let failed = failed.consume().announced(Default::default()).unwrap(); + *trigger + .write() + .unwrap_or_else(|_| unreachable!("test keeps its panic trigger open")) = true; + assert!(matches!(failed.next().await, Err(MoqError::Task { .. }))); + + let healthy = MoqOriginProducer::new(Default::default()); + let broadcast = healthy.create_broadcast("healthy".into()).unwrap(); + broadcast.announce(Default::default()).unwrap(); + healthy.consume().request_broadcast("healthy".into()).await.unwrap(); + } + + #[tokio::test] + async fn a_broadcast_keeps_the_origin_failure_after_its_origin_handle_is_released() { + let (inner, completion, trigger) = crate::ffi::panicking_origin(); + let origin = MoqOriginProducer::from_inner(Origin { + inner, + driver: completion.clone(), + }); + let broadcast = origin.create_broadcast("retained".into()).unwrap(); + let dynamic = broadcast.dynamic().unwrap(); + drop(origin); + *trigger + .write() + .unwrap_or_else(|_| unreachable!("test keeps its panic trigger open")) = true; + assert!(matches!(completion.finished().await, Err(MoqError::Task { .. }))); + drop(completion); + assert!(matches!( + broadcast.announce(Default::default()), + Err(MoqError::Task { .. }) + )); + assert!(matches!(dynamic.requested_track().await, Err(MoqError::Task { .. }))); + let healthy = MoqOriginProducer::new(Default::default()); + healthy + .create_broadcast("healthy".into()) + .unwrap() + .announce(Default::default()) + .unwrap(); + } + + #[tokio::test] + async fn container_importers_keep_the_origin_failure_after_other_owners_are_released() { + use crate::media::{ + MoqContainerFormat, MoqContainerInit, MoqContainerStreamInit, MoqMediaContainerProducer, + MoqMediaContainerStreamProducer, MoqMediaTimebase, + }; + let (inner, completion, trigger) = crate::ffi::panicking_origin(); + let origin = MoqOriginProducer::from_inner(Origin { + inner, + driver: completion.clone(), + }); + let broadcast = origin.create_broadcast("retained".into()).unwrap(); + // One importer shares a timebase and one doesn't, so both reservation paths carry the driver. + let timebase = MoqMediaTimebase::new(broadcast.clone()).unwrap(); + let container = MoqMediaContainerProducer::new( + &broadcast, + MoqContainerInit { + format: MoqContainerFormat::Ts, + data: Vec::new(), + timebase: Some(timebase.clone()), + }, + ) + .unwrap(); + let stream = MoqMediaContainerStreamProducer::new( + &broadcast, + MoqContainerStreamInit { + format: MoqContainerFormat::Ts, + timebase: None, + }, + ) + .unwrap(); + drop(timebase); + drop(broadcast); + drop(origin); + *trigger + .write() + .unwrap_or_else(|_| unreachable!("test keeps its panic trigger open")) = true; + assert!(matches!(completion.finished().await, Err(MoqError::Task { .. }))); + drop(completion); + assert!(matches!(container.cut(), Err(MoqError::Task { .. }))); + assert!(matches!(stream.write(Vec::new()), Err(MoqError::Task { .. }))); + assert!(matches!(container.finish(), Err(MoqError::Task { .. }))); + assert!(matches!(stream.finish(), Err(MoqError::Task { .. }))); + } + + #[tokio::test] + async fn readers_derived_from_a_routed_broadcast_report_the_origin_panic() { + let (inner, completion, trigger) = crate::ffi::panicking_origin(); + let origin = MoqOriginProducer::from_inner(Origin { + inner, + driver: completion.clone(), + }); + let broadcast = origin.create_broadcast("routed".into()).unwrap(); + broadcast.announce(Default::default()).unwrap(); + let track = broadcast.publish_track("data".into(), None).unwrap(); + let open = track.append_group().unwrap(); + let consumer = origin.consume().request_broadcast("routed".into()).await.unwrap(); + let subscriber = consumer.subscribe_track("data".into(), None).await.unwrap(); + let group = subscriber.recv_group().await.unwrap().unwrap(); + let catalog = crate::media::MoqMediaCatalogConsumer::subscribe(&consumer) + .await + .unwrap(); + drop(consumer); + drop(origin); + *trigger + .write() + .unwrap_or_else(|_| unreachable!("test keeps its panic trigger open")) = true; + assert!(matches!(completion.finished().await, Err(MoqError::Task { .. }))); + drop(completion); + // Teardown drops the producers too, which the readers would otherwise report as `dropped`. + drop(open); + drop(track); + drop(broadcast); + let result = subscriber.recv_group().await.map(|group| group.is_some()); + assert!(matches!(result, Err(MoqError::Task { .. })), "{result:?}"); + let result = group.read_frame().await.map(|frame| frame.is_some()); + assert!(matches!(result, Err(MoqError::Task { .. })), "{result:?}"); + let result = catalog.next().await.map(|catalog| catalog.is_some()); + assert!(matches!(result, Err(MoqError::Task { .. })), "{result:?}"); + } + + #[tokio::test] + async fn a_driver_panic_is_recorded_before_teardown_wakes_the_host() { + struct Driver { + dropped: Option>, + ack: std::sync::mpsc::Receiver<()>, + } + impl moq_net::time::Driver for Driver { + fn poll( + &mut self, + _: moq_net::time::Instant, + _: &kio::Waiter, + ) -> Result, moq_net::Error> { + panic!("injected driver panic"); + } + } + impl Drop for Driver { + fn drop(&mut self) { + let _ = self.dropped.take().unwrap().send(()); + self.ack.recv().unwrap(); + } + } + let (dropped, teardown) = tokio::sync::oneshot::channel(); + let (ack, released) = std::sync::mpsc::channel(); + let driver = Background::driver(Driver { + dropped: Some(dropped), + ack: released, + }); + let result: Result<(), MoqError> = driver + .watch(async { + let _ = teardown.await; + Err(MoqError::Cancelled) + }) + .await; + ack.send(()).unwrap(); + assert!(matches!(driver.finished().await, Err(MoqError::Task { .. }))); + assert!(matches!(result, Err(MoqError::Task { .. })), "{result:?}"); + } +} diff --git a/rs/moq-ffi/src/producer.rs b/rs/moq-ffi/src/producer.rs index d43ed780ebf..9e30534e81a 100644 --- a/rs/moq-ffi/src/producer.rs +++ b/rs/moq-ffi/src/producer.rs @@ -5,7 +5,7 @@ use moq_mux::catalog::hang::Extra; use crate::consumer::{MoqBroadcastConsumer, MoqGroupConsumer, MoqSubscription, MoqTrackConsumer}; use crate::demand::{MoqGroupDemand, MoqTrackDemand}; use crate::error::MoqError; -use crate::ffi::Task; +use crate::ffi::{Background, Task}; use crate::media::MoqFrame; /// Publisher-side track properties, mirroring [`moq_net::track::Info`]. @@ -80,6 +80,7 @@ pub(crate) struct BroadcastProducer { pub(crate) catalog: moq_mux::catalog::Producer, // Core timebases own catalog outputs, so the broadcast holds them and binding handles stay weak. pub(crate) timebases: std::collections::HashMap>, + pub(crate) driver: Background, } #[derive(uniffi::Object)] @@ -134,7 +135,10 @@ impl TrackDynamicProducer { impl MoqBroadcastProducer { /// Wrap a `moq_net::broadcast::Producer` (standalone or origin-created), attaching /// the catalog track every FFI broadcast carries. - pub(crate) fn from_inner(mut broadcast: moq_net::broadcast::Producer) -> Result { + pub(crate) fn from_inner( + mut broadcast: moq_net::broadcast::Producer, + driver: Background, + ) -> Result { let config = moq_mux::catalog::Config::default().with_catalog(moq_mux::catalog::hang::Catalog::::default()); let catalog = moq_mux::catalog::Producer::new(&mut broadcast, config)?; @@ -143,14 +147,13 @@ impl MoqBroadcastProducer { broadcast, catalog, timebases: Default::default(), + driver, })), }) } pub(crate) fn consume_inner(&self) -> Result { - let guard = self.state.lock().unwrap(); - let state = guard.as_ref().ok_or(MoqError::Closed)?; - Ok(state.broadcast.consume()) + self.with_state(|state| Ok(state.broadcast.consume())) } /// Run `f` against the open broadcast and catalog. Errors with @@ -162,7 +165,10 @@ impl MoqBroadcastProducer { ) -> Result { let mut guard = self.state.lock().unwrap(); let state = guard.as_mut().ok_or(MoqError::Closed)?; - f(state) + state.driver.check()?; + let result = f(state); + state.driver.check()?; + result } } @@ -171,7 +177,13 @@ impl MoqBroadcastProducer { /// Create a consumer that reads from this broadcast's tracks. pub fn consume(&self) -> Result, MoqError> { let _guard = crate::ffi::enter(); - Ok(Arc::new(MoqBroadcastConsumer::new(self.consume_inner()?))) + self.with_state(|state| { + Ok(Arc::new(MoqBroadcastConsumer { + inner: state.broadcast.consume(), + origin: None, + driver: state.driver.clone(), + })) + }) } /// Create a dynamic producer that yields tracks requested by subscribers. @@ -180,13 +192,14 @@ impl MoqBroadcastProducer { /// accepted. Dropping it makes future subscriptions to unknown tracks fail. pub fn dynamic(&self) -> Result, MoqError> { let _guard = crate::ffi::enter(); - let guard = self.state.lock().unwrap(); - let state = guard.as_ref().ok_or(MoqError::Closed)?; - Ok(Arc::new(MoqBroadcastDynamic { - task: Task::new(DynamicProducer { - inner: state.broadcast.dynamic(), - }), - })) + self.with_state(|state| { + Ok(Arc::new(MoqBroadcastDynamic { + task: Task::new(DynamicProducer { + inner: state.broadcast.dynamic(), + }) + .supervised(state.driver.clone()), + })) + }) } /// Create a standalone broadcast, not attached to any origin. @@ -197,7 +210,10 @@ impl MoqBroadcastProducer { #[uniffi::constructor] pub fn new() -> Result, MoqError> { let _guard = crate::ffi::enter(); - Ok(Arc::new(Self::from_inner(moq_net::broadcast::Info::new().produce())?)) + Ok(Arc::new(Self::from_inner( + moq_net::broadcast::Info::new().produce(), + Background::default(), + )?)) } /// Advertise this broadcast's exact path as a route. @@ -234,15 +250,15 @@ impl MoqBroadcastProducer { /// track properties (priority, max age, timescale); omit for defaults. pub fn publish_track(&self, name: String, info: Option) -> Result, MoqError> { let _guard = crate::ffi::enter(); - let guard = self.state.lock().unwrap(); - let state = guard.as_ref().ok_or(MoqError::Closed)?; - let info = raw_track_info(info)?; - // Clone the broadcast handle (shared Arc internally) to get &mut access. - let broadcast = state.broadcast.clone(); - let producer = broadcast.create_track(name, Some(info))?; - Ok(Arc::new(MoqTrackProducer { - inner: std::sync::Mutex::new(Some(producer)), - })) + self.with_state(|state| { + let info = raw_track_info(info)?; + // Clone the broadcast handle (shared Arc internally) to get &mut access. + let broadcast = state.broadcast.clone(); + let producer = broadcast.create_track(name, Some(info))?; + Ok(Arc::new(MoqTrackProducer { + inner: std::sync::Mutex::new(Some(producer)), + })) + }) } /// End the broadcast for good: retract it, serve no new tracks, and finalize the catalog. @@ -258,8 +274,9 @@ impl MoqBroadcastProducer { }; // Close the broadcast first so it ends even if finalizing the catalog fails. state.broadcast.close(); - state.catalog.finish()?; - Ok(()) + let result = state.catalog.finish().map_err(Into::into); + state.driver.check()?; + result } } @@ -501,7 +518,10 @@ impl MoqTrackProducer { let guard = self.inner.lock().unwrap(); let track = guard.as_ref().ok_or(MoqError::Closed)?; let subscription = subscription.map(moq_net::track::Subscription::from); - Ok(Arc::new(MoqTrackConsumer::new(track.subscribe(subscription)))) + Ok(Arc::new(MoqTrackConsumer::new( + track.subscribe(subscription), + Background::default(), + ))) } /// Append a new group to the track, returning a producer for writing frames into it. @@ -613,7 +633,7 @@ impl MoqGroupProducer { let _guard = crate::ffi::enter(); let guard = self.inner.lock().unwrap(); let group = guard.as_ref().ok_or(MoqError::Closed)?; - Ok(Arc::new(MoqGroupConsumer::new(group.consume()))) + Ok(Arc::new(MoqGroupConsumer::new(group.consume(), Background::default()))) } /// Write `frame` into this group. diff --git a/rs/moq-ffi/src/server.rs b/rs/moq-ffi/src/server.rs index 70650664a6b..56828748b6c 100644 --- a/rs/moq-ffi/src/server.rs +++ b/rs/moq-ffi/src/server.rs @@ -63,6 +63,7 @@ struct ServerState { } struct Requests { + driver: crate::ffi::Background, queue: tokio::sync::mpsc::Receiver, certificates: moq_tokio::tls::Certificates, } @@ -176,11 +177,7 @@ impl ServerState { } /// Bind, and start the accept loop that `listening` stops. - async fn listen( - &mut self, - task: &Task, - listening: &std::sync::Mutex>, - ) -> Result { + async fn listen(&mut self, owner: &MoqServer) -> Result { if self.requests.is_some() { return Err(MoqError::Bind { reason: "already listening".into(), @@ -188,12 +185,12 @@ impl ServerState { } let (bound, addr) = tokio::sync::oneshot::channel(); let (queue, requests) = tokio::sync::mpsc::channel(1); - let certificates = { + let (certificates, driver) = { // `cancel` flags the task before it takes `listening`, so checking the flag under // that lock means either `cancel` finds the loop or this finds the cancel. `init` // binds the QUIC socket, so it goes after the check and straight to the loop. - let mut slot = listening.lock().unwrap(); - if task.is_cancelled() { + let mut slot = owner.listening.lock().unwrap(); + if owner.task.is_cancelled() { return Err(MoqError::Cancelled); } let server = self @@ -206,14 +203,22 @@ impl ServerState { let certificates = server.certificates(); let (stop, stopped) = tokio::sync::oneshot::channel(); let (closed, wait) = std::sync::mpsc::channel(); - crate::ffi::spawn(serve(server, bound, queue, stopped, closed)); + let driver = crate::ffi::spawn(serve(server, bound, queue, stopped, closed)); + *owner.driver.lock().unwrap() = driver.clone(); *slot = Some(Listening { stop, closed: wait }); - certificates + (certificates, driver) }; // Dropped only when the runtime is gone. - let addr = addr.await.map_err(|_| MoqError::Cancelled)??; + let addr = match addr.await { + Ok(addr) => addr?, + Err(_) => { + driver.finished().await?; + return Err(MoqError::Cancelled); + } + }; self.requests = Some(Requests { + driver, queue: requests, certificates, }); @@ -230,7 +235,10 @@ impl ServerState { self.publish.clone(), self.consume.clone(), )?)), - None => Ok(None), + None => { + requests.driver.finished().await?; + Ok(None) + } } } } @@ -239,6 +247,8 @@ impl ServerState { #[derive(uniffi::Object)] pub struct MoqServer { task: Task, + /// Outside cancellable state so a terminal call can still report the listener's panic. + driver: std::sync::Mutex, /// Outside `task`, so `cancel` closes the listener without waiting for a parked `accept`. listening: std::sync::Mutex>, } @@ -253,6 +263,7 @@ impl MoqServer { let _guard = crate::ffi::runtime().enter(); Ok(Arc::new(Self { task: Task::new(ServerState::new(config)?), + driver: Default::default(), listening: Default::default(), })) } @@ -260,9 +271,13 @@ impl MoqServer { /// Bind the listening socket. Returns the bound local address as a string, /// which is useful when binding to an ephemeral port (`:0`). pub async fn listen(&self) -> Result { - self.task - .run(|mut state| async move { state.listen(&self.task, &self.listening).await }) - .await + let result = self.task.run(|mut state| async move { state.listen(self).await }).await; + let driver = self.driver.lock().unwrap().clone(); + if self.task.is_cancelled() { + driver.finished().await?; + } + driver.check()?; + result } /// Accept the next incoming session. Returns `None` when the server has closed. @@ -270,7 +285,14 @@ impl MoqServer { /// `listen()` must be called first. Dropping the returned future aborts this /// call alone and leaves the server listening. pub async fn accept(&self) -> Result>, MoqError> { - self.task.run(|mut state| async move { state.accept().await }).await + let driver = self.driver.lock().unwrap().clone(); + let result = driver + .watch(self.task.run(|mut state| async move { state.accept().await })) + .await; + if self.task.is_cancelled() { + driver.finished().await?; + } + result } /// SHA-256 fingerprints of the configured TLS certificates, hex-encoded. @@ -279,6 +301,7 @@ impl MoqServer { /// WebTransport's `serverCertificateHashes`. Returns an error if called /// before `listen()`, and [`MoqError::Busy`] while `listen()` or `accept()` is in flight. pub fn cert_fingerprints(&self) -> Result, MoqError> { + self.driver.lock().unwrap().check()?; let state = self.task.configure()?; let requests = state.requests.as_ref().ok_or_else(|| MoqError::Bind { reason: "not listening; call listen() first".into(), @@ -463,15 +486,15 @@ impl MoqRequest { publish.as_ref().or(state.publish.as_ref()), consume.as_ref().or(state.consume.as_ref()), ); - let session = request - .with_publisher(&publish) - .with_subscriber(subscribe.clone()) + let (session, driver) = request + .with_publisher(&publish.inner) + .with_subscriber(subscribe.inner.clone()) .ok() .await .map_err(|err| MoqError::Connect { reason: format!("{err}"), })?; - Ok(Arc::new(MoqSession::accepted(session, publish, subscribe))) + Ok(Arc::new(MoqSession::accepted(session, publish, subscribe, driver))) }) .await } diff --git a/rs/moq-ffi/src/session.rs b/rs/moq-ffi/src/session.rs index a0fc995fac4..6ab517025c7 100644 --- a/rs/moq-ffi/src/session.rs +++ b/rs/moq-ffi/src/session.rs @@ -208,14 +208,21 @@ impl Client { let connection = self .client .clone() - .with_publisher(&publish) - .with_subscriber(subscribe.clone()) + .with_publisher(&publish.inner) + .with_subscriber(subscribe.inner.clone()) .connect(url); // Wait for the first session so auth errors surface here; later drops are the // connection's to ride out. `MoqClient::cancel` unblocks a dial stuck retrying. // `established` hands the connection back, which is what keeps it reconnecting. - let connection = connection.established().await.map_err(map_connect_error)?; + let connection = publish + .driver + .watch( + subscribe + .driver + .watch(async { connection.established().await.map_err(map_connect_error) }), + ) + .await?; Ok(Arc::new(MoqSession::connected(connection, publish, subscribe))) } @@ -225,6 +232,9 @@ impl Client { fn map_connect_error(err: moq_tokio::Error) -> MoqError { match err { moq_tokio::Error::MoqNet(err) => err.into(), + moq_tokio::Error::Panic => MoqError::Task { + reason: "background task panicked".into(), + }, err => match err.connect_error() { Some(moq_tokio::ConnectError::Unauthorized) => MoqError::Unauthorized, Some(moq_tokio::ConnectError::Forbidden) => MoqError::Forbidden, @@ -248,8 +258,11 @@ fn map_connect_error(err: moq_tokio::Error) -> MoqError { /// connect error would make every status watcher treat an expected teardown as a /// broken connection. #[cfg(not(target_arch = "wasm32"))] -fn map_closed_error(err: moq_tokio::Error) -> MoqError { +pub(crate) fn map_closed_error(err: moq_tokio::Error) -> MoqError { match err { + moq_tokio::Error::Panic => MoqError::Task { + reason: "background task panicked".into(), + }, moq_tokio::Error::Stopped => MoqError::Closed, // A peer's session/stream code stays structured. HTTP 401/403 are a // connect-time rejection, not a protocol code, and land below. @@ -487,8 +500,8 @@ impl Client { // transport and ends that task. let (session, driver) = moq_net::Client::new() .with_versions(self.versions.clone()) - .with_publisher(&publish) - .with_subscriber(subscribe.clone()) + .with_publisher(&publish.inner) + .with_subscriber(subscribe.inner.clone()) .connect(web_async::time::Instant::now(), transport) .await?; @@ -662,7 +675,37 @@ impl From for MoqConnectionStatus { enum Inner { #[cfg(not(target_arch = "wasm32"))] Connection(moq_tokio::Connection), - Session(moq_net::Session), + Session(Accepted), +} + +#[derive(Clone)] +struct Accepted { + session: moq_net::Session, + driver: crate::ffi::Background, +} + +impl std::ops::Deref for Accepted { + type Target = moq_net::Session; + fn deref(&self) -> &Self::Target { + &self.session + } +} + +impl Accepted { + async fn closed(&self) -> Result { + self.driver.check()?; + #[cfg(not(target_arch = "wasm32"))] + self.driver.finished().await?; + Err(self.session.closed().await.into()) + } + + async fn close(self) -> Result<(), MoqError> { + self.driver.check()?; + let result = self.session.close().await; + #[cfg(not(target_arch = "wasm32"))] + self.driver.finished().await?; + result.map_err(Into::into) + } } #[derive(uniffi::Object)] @@ -685,8 +728,8 @@ impl MoqSession { #[cfg(not(target_arch = "wasm32"))] pub(crate) fn connected( connection: moq_tokio::Connection, - publish: moq_net::origin::Producer, - subscribe: moq_net::origin::Producer, + publish: crate::origin::Origin, + subscribe: crate::origin::Origin, ) -> Self { Self::build(Inner::Connection(connection), publish, subscribe) } @@ -694,10 +737,21 @@ impl MoqSession { /// Wrap a server-accepted session (see `MoqServer::accept`). pub(crate) fn accepted( session: moq_net::Session, - publish: moq_net::origin::Producer, - subscribe: moq_net::origin::Producer, + publish: crate::origin::Origin, + subscribe: crate::origin::Origin, + #[cfg(not(target_arch = "wasm32"))] driver: tokio::task::JoinHandle>, ) -> Self { - Self::build(Inner::Session(session), publish, subscribe) + #[cfg(not(target_arch = "wasm32"))] + let driver = crate::ffi::Background::new(async move { + match driver.await { + Ok(result) => result, + Err(error) if error.is_cancelled() => Err(moq_net::Error::Cancel.into()), + Err(_) => Err(moq_tokio::Error::Panic), + } + }); + #[cfg(target_arch = "wasm32")] + let driver = crate::ffi::Background::default(); + Self::build(Inner::Session(Accepted { session, driver }), publish, subscribe) } fn mint_allocator(inner: &Inner) -> moq_net::bandwidth::Allocator { @@ -713,12 +767,12 @@ impl MoqSession { } } - fn build(inner: Inner, publish: moq_net::origin::Producer, subscribe: moq_net::origin::Producer) -> Self { + fn build(inner: Inner, publish: crate::origin::Origin, subscribe: crate::origin::Origin) -> Self { // Eagerly wrap the wired origin sides so each publish()/consume() // call hands back the same Arc. `publish` is published into; `subscribe` // is where the remote's broadcasts land (read via its consumer view). let publisher = Arc::new(MoqOriginProducer::from_inner(publish)); - let consumer = Arc::new(MoqOriginConsumer::from_inner(subscribe.consume())); + let consumer = Arc::new(MoqOriginConsumer::from_inner(subscribe)); let bandwidth = Arc::new(MoqBandwidth::new(Self::mint_allocator(&inner))); Self { inner: inner.clone(), @@ -730,6 +784,14 @@ impl MoqSession { } } + async fn watch(&self, future: impl std::future::Future>) -> Result { + self.publisher + .inner() + .driver + .watch(self.consumer.driver.watch(future)) + .await + } + /// Abort the live transport (if any) with `err` and stop any reconnect loop. fn teardown(&self, err: moq_net::Error) { match &self.inner { @@ -763,15 +825,14 @@ impl MoqSession { /// session resolves with the session's close reason. pub async fn closed(&self) -> Result<(), MoqError> { // We have a task to run all of the closed calls juuuuust so they use the same tokio runtime. - self.closed - .run(|inner| async move { - match &*inner { - #[cfg(not(target_arch = "wasm32"))] - Inner::Connection(connection) => connection.closed().await.map_err(map_closed_error), - Inner::Session(session) => Err(session.closed().await.into()), - } - }) - .await + self.watch(self.closed.run(|inner| async move { + match &*inner { + #[cfg(not(target_arch = "wasm32"))] + Inner::Connection(connection) => connection.closed().await.map_err(map_closed_error), + Inner::Session(session) => session.closed().await, + } + })) + .await } /// Wait for the connection status to differ from the one this handle last reported. @@ -787,15 +848,14 @@ impl MoqSession { /// before you ask again is coalesced away, so the outages it hides are the ones /// that already healed. Don't count outages with it. pub async fn status(&self) -> Result { - self.status - .run(|mut inner| async move { - match &mut *inner { - #[cfg(not(target_arch = "wasm32"))] - Inner::Connection(connection) => Ok(connection.status().await.map_err(map_closed_error)?.into()), - Inner::Session(session) => Err(session.closed().await.into()), - } - }) - .await + self.watch(self.status.run(|mut inner| async move { + match &mut *inner { + #[cfg(not(target_arch = "wasm32"))] + Inner::Connection(connection) => Ok(connection.status().await.map_err(map_closed_error)?.into()), + Inner::Session(session) => session.closed().await, + } + })) + .await } /// How many times this session has connected: 1 for the connect it was built from, one more @@ -829,14 +889,17 @@ impl MoqSession { /// IETF also waits for transport ACKs. pub async fn shutdown(&self) -> Result<(), MoqError> { let inner = self.inner.clone(); - crate::ffi::detached(async move { + let result = crate::ffi::detached(async move { match inner { #[cfg(not(target_arch = "wasm32"))] Inner::Connection(connection) => connection.close().await.map_err(map_closed_error), - Inner::Session(session) => session.close().await.map_err(MoqError::from), + Inner::Session(session) => session.close().await, } }) - .await + .await; + self.publisher.inner().driver.check()?; + self.consumer.driver.check()?; + result } /// The publish-side origin: where local broadcasts get advertised diff --git a/rs/moq-ffi/src/video.rs b/rs/moq-ffi/src/video.rs index 94c5c86e61b..ef525fc2fa7 100644 --- a/rs/moq-ffi/src/video.rs +++ b/rs/moq-ffi/src/video.rs @@ -11,7 +11,7 @@ //! timestamp. use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::time::Instant; use tokio::sync::oneshot; @@ -191,14 +191,33 @@ struct Follow { close: Option>, thread: Option>, ceiling: Arc, + failed: Arc, } -impl Drop for Follow { - fn drop(&mut self) { +impl Follow { + fn check(&self) -> Result<(), MoqError> { + if self.failed.load(Ordering::Acquire) { + return Err(panic_error()); + } + Ok(()) + } + + fn stop(&mut self) -> Result<(), MoqError> { drop(self.close.take()); if let Some(thread) = self.thread.take() { - let _ = thread.join(); + thread.join().map_err(|_| panic_error())?; } + self.check() + } +} + +fn panic_error() -> MoqError { + crate::session::map_closed_error(moq_tokio::Error::Panic) +} + +impl Drop for Follow { + fn drop(&mut self) { + let _ = self.stop(); } } @@ -247,8 +266,16 @@ pub struct MoqVideoProducer { } impl MoqVideoProducer { + fn check(&self) -> Result<(), MoqError> { + if let Some(follow) = self.follow.lock().unwrap().as_ref() { + follow.check()?; + } + Ok(()) + } + fn track_demand(&self) -> Result { - let guard = self.inner.lock().unwrap(); + self.check()?; + let guard = self.inner.lock().map_err(|_| panic_error())?; let producer = guard.as_ref().ok_or(MoqError::Closed)?; Ok(producer.producer.demand()) } @@ -293,8 +320,9 @@ impl MoqVideoProducer { /// A backend that pipelines publishes an earlier frame's output here, so a /// call that emits nothing on the wire is normal rather than an error. pub fn write(&self, frame: MoqVideoFrame) -> Result<(), MoqError> { + self.check()?; let _guard = crate::ffi::runtime().enter(); - let mut guard = self.inner.lock().unwrap(); + let mut guard = self.inner.lock().map_err(|_| panic_error())?; let producer = guard.as_mut().ok_or(MoqError::Closed)?; producer.write(frame) } @@ -316,7 +344,8 @@ impl MoqVideoProducer { /// without the control): nothing is queued, and groups keep falling at the /// configured interval. pub fn cut(&self) -> Result<(), MoqError> { - let mut guard = self.inner.lock().unwrap(); + self.check()?; + let mut guard = self.inner.lock().map_err(|_| panic_error())?; let producer = guard.as_mut().ok_or(MoqError::Closed)?; // A keyframe is what a cut is on the wire: the importer closes the open // group and starts a new one at it. @@ -341,9 +370,10 @@ impl MoqVideoProducer { /// encoder keeps running at its current rate, so stop adapting rather than /// stop publishing. pub fn set_bitrate(&self, bitrate: u64) -> Result<(), MoqError> { + self.check()?; let _guard = crate::ffi::runtime().enter(); { - let mut guard = self.inner.lock().unwrap(); + let mut guard = self.inner.lock().map_err(|_| panic_error())?; let producer = guard.as_mut().ok_or(MoqError::Closed)?; block_on( producer @@ -373,12 +403,25 @@ impl MoqVideoProducer { /// Flush any frames the codec is still holding and finalize the track. pub fn finish(&self) -> Result<(), MoqError> { let _guard = crate::ffi::runtime().enter(); - let producer = self.inner.lock().unwrap().take().ok_or(MoqError::Closed)?; + let producer = self + .inner + .lock() + .map_err(|_| panic_error())? + .take() + .ok_or(MoqError::Closed)?; // Stop following and release the share before draining, so siblings can // take the room while the last frames go out. - self.follow.lock().unwrap().take(); + let followed = self + .follow + .lock() + .unwrap() + .take() + .map_or(Ok(()), |mut follow| follow.stop()); self.reservation.lock().unwrap().take(); - producer.finish() + // A follower panic only stops rate adaptation, so the encoded frames still + // end the track cleanly before the panic is reported. + producer.finish()?; + followed } } @@ -497,28 +540,36 @@ fn spawn_follow( let ceiling = Arc::new(AtomicU64::new(ceiling.as_bps())); let (close, closed) = oneshot::channel(); let shared_ceiling = ceiling.clone(); + let failed = Arc::new(AtomicBool::new(false)); + let failure = failed.clone(); // A dedicated thread rather than the FFI current-thread runtime: this loop // `pollster::block_on`s `set_bitrate`, which would stall that runtime. let thread = std::thread::Builder::new() .name("moq-ffi-rate".into()) .spawn(move || { - tokio::runtime::Builder::new_current_thread() - .enable_all() - .build() - .expect("rate-control runtime") - .block_on(async move { - tokio::select! { - biased; - _ = closed => {} - _ = follow_reservation(inner, consumer, shared_ceiling, applied) => {} - } - }) + let completed = pollster::block_on(moq_tokio::panic::isolate(async move { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("rate-control runtime") + .block_on(async move { + tokio::select! { + biased; + _ = closed => {} + _ = follow_reservation(inner, consumer, shared_ceiling, applied) => {} + } + }) + })); + if completed.is_none() { + failure.store(true, Ordering::Release); + } }) .expect("failed to spawn rate-control thread"); Follow { close: Some(close), thread: Some(thread), ceiling, + failed, } } @@ -787,10 +838,13 @@ impl MoqBroadcastConsumer { .map(std::time::Duration::from_micros) .unwrap_or_default(); - let consumer = moq_video::decode::Consumer::new(&broadcast, &cfg, name, options).await?; + let consumer = self + .driver + .watch(async { Ok(moq_video::decode::Consumer::new(&broadcast, &cfg, name, options).await?) }) + .await?; Ok(Arc::new(MoqVideoConsumer { - task: crate::ffi::Task::new(VideoConsumerInner { consumer }), + task: crate::ffi::Task::new(VideoConsumerInner { consumer }).supervised(self.driver.clone()), })) } } @@ -851,3 +905,52 @@ mod decode_tests { assert!(matches!(error, MoqError::Unsupported)); } } + +#[cfg(test)] +mod finish_tests { + use super::*; + + #[tokio::test] + async fn finish_ends_the_track_after_a_rate_follower_panic() { + let estimate = moq_net::bandwidth::Producer::new(); + let bandwidth = Arc::new(MoqBandwidth::new(moq_net::bandwidth::Allocator::new( + estimate.consume(), + ))); + let broadcast = MoqBroadcastProducer::new().unwrap(); + let input = MoqVideoEncoderInput { + format: MoqVideoPixelFormat::Rgba, + width: 320, + height: 240, + framerate: 30, + }; + let output = MoqVideoEncoderOutput { + codec: MoqVideoCodec::H264, + track: Some("camera".into()), + bitrate: Some(4_000_000), + gop: None, + kind: MoqVideoEncoderKind::Software, + }; + let video = broadcast.encode_video(input, output, Some(bandwidth)).unwrap(); + let track = broadcast + .consume() + .unwrap() + .subscribe_track("camera".into(), None) + .await + .unwrap(); + // Stand in for a follower thread that panicked. + video + .follow + .lock() + .unwrap() + .as_ref() + .expect("published against an allocator") + .failed + .store(true, Ordering::Release); + + let result = video.finish(); + assert!(matches!(result, Err(MoqError::Task { .. })), "{result:?}"); + drop(video); + let result = track.recv_group().await.map(|group| group.is_some()); + assert!(matches!(result, Ok(false)), "the track should end cleanly: {result:?}"); + } +} diff --git a/rs/moq-gst/src/source/imp.rs b/rs/moq-gst/src/source/imp.rs index 6458bf4f5fc..231bcd7f2eb 100644 --- a/rs/moq-gst/src/source/imp.rs +++ b/rs/moq-gst/src/source/imp.rs @@ -2962,7 +2962,7 @@ mod session_tests { let Some(request) = request else { break }; if refuse { request.reject(moq_tokio::server::Reject::Forbidden).await.unwrap(); - } else if let Ok(session) = request.with_publisher(&serving).with_subscriber(serving.clone()).ok().await { + } else if let Ok((session, _driver)) = request.with_publisher(&serving).with_subscriber(serving.clone()).ok().await { sessions.push(session); } } diff --git a/rs/moq-hls/src/server/routes.rs b/rs/moq-hls/src/server/routes.rs index 3396b01be1d..320e9af88dd 100644 --- a/rs/moq-hls/src/server/routes.rs +++ b/rs/moq-hls/src/server/routes.rs @@ -567,11 +567,12 @@ mod tests { let pub_for_accept = pub_origin.clone(); let accept = tokio::spawn(async move { let request = server.accept().await.expect("accept"); - let session = request + let (session, driver) = request .with_publisher(&pub_for_accept) .ok() .await .expect("publisher session"); + let _ = driver.await; let _ = session.closed().await; }); diff --git a/rs/moq-relay/src/connection.rs b/rs/moq-relay/src/connection.rs index f944205f2c5..6c8425e773e 100644 --- a/rs/moq-relay/src/connection.rs +++ b/rs/moq-relay/src/connection.rs @@ -138,7 +138,7 @@ impl Connection { } let lease = lease.preparing(request.auth(), request.role())?; // Bounded too: a peer that stalls the SETUP reply must not outlive the relay. - let session = self + let (session, driver) = self .shutdown .until_deadline(request.ok()) .await @@ -146,7 +146,9 @@ impl Connection { tracing::info!(version = %session.version(), %transport, "negotiated"); - supervise(session, lease, self.shutdown.clone(), registration).await + let result = supervise(session, lease, self.shutdown.clone(), registration).await; + driver.await??; + result } /// Admit this connection. Any failure is returned as a [`StatusError`] so diff --git a/rs/moq-relay/tests/cluster_idle.rs b/rs/moq-relay/tests/cluster_idle.rs index fc90ee1b7eb..7ae673014e8 100644 --- a/rs/moq-relay/tests/cluster_idle.rs +++ b/rs/moq-relay/tests/cluster_idle.rs @@ -62,7 +62,7 @@ async fn spawn_peer() -> (u16, tokio::runtime::Runtime) { while let Some(request) = listener.accept().await { let origin = origin.clone(); tokio::spawn(async move { - if let Ok(session) = request.with_publisher(&origin).ok().await { + if let Ok((session, _driver)) = request.with_publisher(&origin).ok().await { let _ = session.closed().await; } }); diff --git a/rs/moq-relay/tests/goaway_cluster.rs b/rs/moq-relay/tests/goaway_cluster.rs index 726479103b0..0128f3e0cf6 100644 --- a/rs/moq-relay/tests/goaway_cluster.rs +++ b/rs/moq-relay/tests/goaway_cluster.rs @@ -259,7 +259,7 @@ async fn spawn_upstream( while let Some(request) = server.accept().await { // Serve the shared origin bidirectionally, like a relay peer would. let scratch = moq_tokio::origin::spawn(); - let session = match request.with_publisher(&origin).with_subscriber(scratch).ok().await { + let (session, _driver) = match request.with_publisher(&origin).with_subscriber(scratch).ok().await { Ok(session) => session, Err(err) => { tracing::warn!(%err, "upstream accept failed"); diff --git a/rs/moq-relay/tests/hidden_cluster.rs b/rs/moq-relay/tests/hidden_cluster.rs index c19a8a34f3c..6b7159bf95d 100644 --- a/rs/moq-relay/tests/hidden_cluster.rs +++ b/rs/moq-relay/tests/hidden_cluster.rs @@ -57,7 +57,7 @@ async fn old_peer_keeps_hidden_paths(version: moq_net::Version) { let mut discovered = peer_origin.consume().with_hidden(true).announced(); let accept = tokio::spawn(async move { let request = server.accept().await.expect("cluster dial"); - let session = request.with_subscriber(peer_origin).ok().await.expect("accept"); + let (session, _driver) = request.with_subscriber(peer_origin).ok().await.expect("accept"); session.closed().await; }); diff --git a/rs/moq-tokio/src/client.rs b/rs/moq-tokio/src/client.rs index 20c71f8b585..44cec6bca74 100644 --- a/rs/moq-tokio/src/client.rs +++ b/rs/moq-tokio/src/client.rs @@ -825,7 +825,7 @@ mod tests { let request = server.accept().await.unwrap(); assert_eq!(request.path(), "/room"); assert_eq!(request.query(), Some("jwt=secret")); - let session = request.with_publisher(&server_origin).ok().await.unwrap(); + let (session, _driver) = request.with_publisher(&server_origin).ok().await.unwrap(); (server, session) }); let mut config = crate::connect::Config::default(); @@ -1519,7 +1519,7 @@ mod tests { crate::Transport::WebTransport, "expected the QUIC listener", ); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; assert_eq!(session.version(), expected_version, "server negotiated stale version"); let _broadcast = broadcast; let _track = track; diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index f98e88f8e1c..cb0d0eb4209 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -2285,7 +2285,7 @@ mod tests { tokio::spawn(async move { while let Some(request) = server.accept().await { let transport = request.transport(); - if let Ok(session) = request.ok().await { + if let Ok((session, _driver)) = request.ok().await { let _ = accepted.send((transport, session)); } } @@ -2359,7 +2359,7 @@ mod tests { let (accepted, sessions) = tokio::sync::mpsc::unbounded_channel(); let task = tokio::spawn(async move { while let Some(request) = server.accept().await { - if let Ok(session) = request.with_publisher(&origin).ok().await { + if let Ok((session, _driver)) = request.with_publisher(&origin).ok().await { let _ = accepted.send(session); } } diff --git a/rs/moq-tokio/src/panic.rs b/rs/moq-tokio/src/panic.rs index f02f05af2fc..332ea1e0871 100644 --- a/rs/moq-tokio/src/panic.rs +++ b/rs/moq-tokio/src/panic.rs @@ -1,12 +1,11 @@ -//! Panic isolation for tasks that serve a single session. +//! Panic isolation for tasks with an independent owner. //! -//! A panic unwinds only the task it happens in, so a task serving one session -//! ends that session and nothing else. [`isolate`] is how such a task observes -//! it: logged as an error in the current span and counted for [`count`]. +//! A session or binding object can end without ending the process. [`isolate`] +//! reports its task panic to that owner, logging it in the current span and +//! counting it for [`count`]. //! -//! A task the process cannot serve without (an origin driver, an accept loop) is -//! never isolated. Running on without it is a process that looks healthy and -//! serves nothing, so its panic has to end the process instead. +//! Process-wide tasks that have no independent owner, such as the standalone +//! origin helper, abort instead of leaving a healthy-looking process that serves nothing. use std::{ any::Any, @@ -19,7 +18,7 @@ use futures::FutureExt; static COUNT: AtomicU64 = AtomicU64::new(0); -/// Run one session's task, ending it on a panic instead of unwinding past it. +/// Run one owned task, returning its panic to the owner instead of unwinding past it. /// /// Returns `None` once the panic is logged and counted. pub async fn isolate(task: F) -> Option { @@ -27,7 +26,7 @@ pub async fn isolate(task: F) -> Option { Ok(output) => Some(output), Err(payload) => { COUNT.fetch_add(1, Ordering::Relaxed); - tracing::error!(panic = message(payload.as_ref()), "session task panicked"); + tracing::error!(panic = message(payload.as_ref()), "owned task panicked"); None } } diff --git a/rs/moq-tokio/src/server.rs b/rs/moq-tokio/src/server.rs index 3c57e4b287a..40675a64696 100644 --- a/rs/moq-tokio/src/server.rs +++ b/rs/moq-tokio/src/server.rs @@ -708,6 +708,8 @@ impl Server { // this one, so stop accepting at all and let the owner fail loudly. Err(err) => { tracing::error!(%err, "a stream accept loop died"); + self.shutdown().await; + if err.is_panic() { std::panic::resume_unwind(err.into_panic()); } return None; } } @@ -868,8 +870,8 @@ impl Listener { /// [Request::reject] to complete the handshake. /// /// `None` means nothing can arrive again: every configured listener has stopped - /// with no handshake in flight, or one of them died (logged), which ends them all - /// rather than serving on a partial set of transports. Everything is already + /// with no handshake in flight. An accept-loop panic closes every listener before + /// resuming the panic in this call. Everything is already /// bound, so a bind failure cannot arrive here. No process signal is watched: /// race this against your own ctrl-C future and drop the listener if you want /// one to stop the loop. @@ -937,7 +939,8 @@ impl Drop for Listener { /// Complete one accepted [`Request`] and wait for the session to close. async fn serve_session(request: Request) -> crate::Result<()> { - let session = request.ok().await?; + let (session, driver) = request.ok().await?; + driver.await.map_err(|_| Error::Panic)??; Err(session.closed().await.into()) } @@ -1689,12 +1692,14 @@ impl Request { self } - /// Accept the session, starting the MoQ session loops. + /// Accept the session and return its supervised driver task. + /// + /// Await the driver before classifying the session close so a panic cannot look like cancellation. /// /// Bounded by what remains of the handshake deadline: answering SETUP can wait /// on a peer that withholds flow-control credit, which keep-alives would /// otherwise hold open. - pub async fn ok(self) -> crate::Result { + pub async fn ok(self) -> crate::Result<(Session, tokio::task::JoinHandle>)> { let Request { kind, deadline, active, .. } = self; @@ -1702,11 +1707,14 @@ impl Request { Ok(request_into!(kind, request => { let (session, driver) = request.ok().await?; use tracing::Instrument; - tokio::spawn(crate::panic::isolate(async move { - let _active = active; - moq_net::time::run(driver).await - }).instrument(tracing::Span::current())); - session + let driver = tokio::spawn(async move { + crate::panic::isolate(async move { + let _active = active; + moq_net::time::run(driver).await; + Ok(()) + }).await.unwrap_or(Err(Error::Panic)) + }.instrument(tracing::Span::current())); + (session, driver) })) }; Deadline::bound(deadline.as_ref(), ok).await @@ -2627,6 +2635,33 @@ mod tests { let _ = std::fs::remove_file(&path); } + #[cfg(feature = "tcp")] + #[tokio::test] + async fn a_panicked_accept_loop_ends_its_listener() { + let mut config = crate::listen::Config::default(); + config.tcp.bind = Some("127.0.0.1:0".parse().unwrap()); + let mut listener = Config { + listen: config, + ..Default::default() + } + .init() + .unwrap() + .listen() + .await + .unwrap(); + let address = listener.tcp_local_addr().unwrap(); + listener + .server + .streams + .tasks + .push(tokio::spawn(async { panic!("injected accept loop panic") })); + assert!(crate::panic::isolate(listener.accept()).await.is_none()); + assert!( + tokio::net::TcpListener::bind(address).await.is_ok(), + "a dead accept loop left its listener bound" + ); + } + /// Closing a listener must release its TCP socket before returning. Reusing it /// afterwards is unrepresentable: `listen` consumes the `Server` and `close` /// consumes the `Listener`. @@ -2677,7 +2712,7 @@ mod tests { let connection = client.with_reconnect(false).connect(url); let request = listener.accept().await.expect("no incoming connection"); - let session = request.with_publisher(&origin).ok().await.expect("server handshake"); + let (session, _driver) = request.with_publisher(&origin).ok().await.expect("server handshake"); listener.close().await; std::net::UdpSocket::bind(addr).expect("close must release the QUIC socket"); diff --git a/rs/moq-tokio/src/tcp.rs b/rs/moq-tokio/src/tcp.rs index 1c19e4310f5..c4a30517268 100644 --- a/rs/moq-tokio/src/tcp.rs +++ b/rs/moq-tokio/src/tcp.rs @@ -405,7 +405,7 @@ mod tests { assert_eq!(request.transport(), crate::Transport::Tcp); assert_eq!(request.path(), "/room"); assert_eq!(request.query(), Some("jwt=credential")); - let session = request.ok().await.unwrap(); + let (session, _driver) = request.ok().await.unwrap(); (server, session) }); diff --git a/rs/moq-tokio/tests/backend.rs b/rs/moq-tokio/tests/backend.rs index 60973441975..4c751b7aa19 100644 --- a/rs/moq-tokio/tests/backend.rs +++ b/rs/moq-tokio/tests/backend.rs @@ -168,7 +168,7 @@ async fn connect_test(config: ConnectTest<'_>) { if let Some(expect_authority) = expect_authority { assert_eq!(request.authority(), expect_authority.as_deref()); } - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; @@ -489,7 +489,7 @@ async fn mtls_test(scheme: &str, reject: bool) { request.reject(moq_tokio::server::Reject::Forbidden).await?; return Ok::<_, anyhow::Error>(has_cert); } - let session = request.with_publisher(pub_origin.consume()).ok().await?; + let (session, _driver) = request.with_publisher(pub_origin.consume()).ok().await?; let _ = session.closed().await; Ok::<_, anyhow::Error>(has_cert) }); @@ -682,7 +682,7 @@ async fn iroh_connect_test(version: Option<&str>) { if let Some(version) = version.first() { assert_eq!(request.alpn(), Some(version.alpn())); } - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; @@ -787,7 +787,7 @@ async fn noq_client_close_reaches_server() { .await .expect("accept timed out") .expect("no incoming connection"); - let session = request.ok().await.expect("server handshake failed"); + let (session, _driver) = request.ok().await.expect("server handshake failed"); accepted.send(()).expect("client waiting for the accepted session"); tokio::task::spawn_blocking(move || client.join()) .await @@ -868,7 +868,7 @@ async fn noq_client_close_drains_finished_track() { let origin = moq_tokio::origin::spawn(); let consumer = origin.consume(); let mut announcements = consumer.announced(); - let _session = request + let (_session, _driver) = request .with_subscriber(origin) .ok() .await @@ -989,7 +989,7 @@ async fn noq_client_close_drains_migrated_predecessor() { let origin = moq_tokio::origin::spawn(); let consumer = origin.consume(); let mut announcements = consumer.announced(); - let first = request + let (first, _driver) = request .with_subscriber(origin) .ok() .await @@ -1126,7 +1126,7 @@ async fn noq_client_close_keeps_predecessor_handover() { let origin = moq_tokio::origin::spawn(); let consumer = origin.consume(); let mut announcements = consumer.announced(); - let first = request + let (first, _driver) = request .with_subscriber(origin) .ok() .await diff --git a/rs/moq-tokio/tests/broadcast.rs b/rs/moq-tokio/tests/broadcast.rs index 7cb70653e90..bf01de50450 100644 --- a/rs/moq-tokio/tests/broadcast.rs +++ b/rs/moq-tokio/tests/broadcast.rs @@ -72,7 +72,7 @@ async fn broadcast_test(scheme: &str, client_version: Option<&str>, server_versi // ── run server and client concurrently ────────────────────────── let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; // Keep producers alive so the subscriber can read data. let _broadcast = broadcast; @@ -191,7 +191,7 @@ async fn lite05_timestamp_roundtrip(scheme: &str) { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -314,7 +314,7 @@ async fn lite05_fetch_roundtrip(scheme: &str) { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -428,7 +428,7 @@ async fn transport_fetch_roundtrip(version: &str) { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -570,7 +570,7 @@ async fn lite05_fetch_during_subscribe(scheme: &str) { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -679,7 +679,7 @@ async fn broadcast_moq_lite_05_untimed_track() { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("accept"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -783,7 +783,7 @@ async fn broadcast_moq_transport_20_current_group_join() { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -899,7 +899,7 @@ async fn announce_lifecycle_test(version: &str) { let server_origin = pub_origin.clone(); let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("accept"); - let session = request.with_publisher(&server_origin).ok().await?; + let (session, _driver) = request.with_publisher(&server_origin).ok().await?; let _ = session.closed().await; Ok::<_, anyhow::Error>(()) }); @@ -1073,7 +1073,7 @@ async fn broadcast_route_migration() { let handle_a = tokio::spawn(async move { let request = server_a.accept().await.expect("accept"); - let session = request.with_publisher(&origin_a).ok().await?; + let (session, _driver) = request.with_publisher(&origin_a).ok().await?; let _broadcast = broadcast_a; let _track = track_a; let _ = session.closed().await; @@ -1081,7 +1081,7 @@ async fn broadcast_route_migration() { }); let handle_b = tokio::spawn(async move { let request = server_b.accept().await.expect("accept"); - let session = request.with_publisher(&origin_b).ok().await?; + let (session, _driver) = request.with_publisher(&origin_b).ok().await?; let _broadcast = broadcast_b; let _track = track_b; let _ = session.closed().await; @@ -1220,7 +1220,7 @@ async fn rejoin_skips_a_stale_warm_cache(version: &str) { let addr = server.local_addr().expect("local addr"); let server = tokio::spawn(async move { let request = server.accept().await.expect("accept"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -1340,7 +1340,7 @@ async fn rejoin_replays_a_current_warm_cache(version: &str, open: bool) { let addr = server.local_addr().expect("local addr"); let server = tokio::spawn(async move { let request = server.accept().await.expect("accept"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -1467,7 +1467,7 @@ async fn route_reannounce_test(version: Option<&str>) { let server_producer = producer.clone(); let handle = tokio::spawn(async move { let request = server.accept().await.expect("accept"); - let session = request.with_publisher(&origin).ok().await?; + let (session, _driver) = request.with_publisher(&origin).ok().await?; let _producer = server_producer; let _ = session.closed().await; Ok::<_, anyhow::Error>(()) @@ -1772,7 +1772,7 @@ async fn max_age_test(version: &str, published: Option) -> Option(()) }); @@ -2688,7 +2688,7 @@ async fn idle_subscription_releases_the_viewer_count() { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("accept"); - let session = request.with_publisher(&pub_origin).with_stats(stats).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).with_stats(stats).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -2845,7 +2845,7 @@ async fn one_shot_goaway_ends_the_connection() { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.ok().await?; + let (session, _driver) = request.ok().await?; // No deadline, so nothing on the peer's side ever force-closes us. session @@ -2890,7 +2890,7 @@ async fn one_shot_connect_surfaces_the_session_close() { let server_handle = tokio::spawn(async move { while let Some(request) = server.accept().await { server_accepts.fetch_add(1, Ordering::SeqCst); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; session.abort(moq_net::Error::Cancel); } Ok::<_, anyhow::Error>(()) @@ -2949,7 +2949,7 @@ async fn a_dead_session_unannounces_while_the_reconnect_retries() { let (session_tx, session_rx) = tokio::sync::oneshot::channel(); let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("accept"); - let session = request.with_publisher(&server_pub).ok().await?; + let (session, _driver) = request.with_publisher(&server_pub).ok().await?; let _ = session_tx.send(session); std::future::pending::<()>().await; Ok::<_, anyhow::Error>(()) @@ -3044,7 +3044,7 @@ async fn announce_interest_unauthorized_keeps_session_alive() { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(publish).ok().await?; + let (session, _driver) = request.with_publisher(publish).ok().await?; let _broadcast = broadcast; let _track = track; let _ = session.closed().await; @@ -3136,7 +3136,7 @@ async fn wildcard_scope_test(version: &str, server_scope: &str) { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(publish).ok().await?; + let (session, _driver) = request.with_publisher(publish).ok().await?; let _broadcasts = broadcasts; let _ = session.closed().await; Ok::<_, anyhow::Error>(()) @@ -3251,7 +3251,7 @@ async fn publish_only_client_to_subscribe_only_server() { let url: url::Url = format!("https://localhost:{}", addr.port()).parse().unwrap(); let server_handle = tokio::spawn(async move { - let session = server + let (session, _driver) = server .accept() .await .expect("no incoming connection") @@ -3423,7 +3423,7 @@ async fn goaway_test(scheme: &str, version: &str, expect_wire_timeout: bool) { let (start_drain_tx, start_drain_rx) = tokio::sync::oneshot::channel::<()>(); let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; start_drain_rx.await.expect("drain signal"); session @@ -3557,7 +3557,7 @@ async fn goaway_timeout_force_close_moq_transport_19_quic() { let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; session .drain() @@ -3630,7 +3630,7 @@ async fn zero_initial_backoff_still_gives_up_on_a_flapping_peer() { let server_handle = tokio::spawn(async move { // Accept and immediately sever, over and over. while let Some(request) = server.accept().await { - let session = request.with_publisher(&pub_origin).ok().await?; + let (session, _driver) = request.with_publisher(&pub_origin).ok().await?; session.abort(moq_net::Error::Cancel); } Ok::<_, anyhow::Error>(()) @@ -3752,7 +3752,7 @@ async fn abort_carries_its_code_to_the_peer() { let (reason_tx, reason_rx) = tokio::sync::oneshot::channel(); let server_handle = tokio::spawn(async move { let request = server.accept().await.expect("no incoming connection"); - let session = request.ok().await?; + let (session, _driver) = request.ok().await?; let _ = reason_tx.send(session.closed().await); Ok::<_, anyhow::Error>(()) }); diff --git a/rs/moq-tokio/tests/close_code.rs b/rs/moq-tokio/tests/close_code.rs index 9d5f287e97e..d1cdcedb5b9 100644 --- a/rs/moq-tokio/tests/close_code.rs +++ b/rs/moq-tokio/tests/close_code.rs @@ -55,7 +55,9 @@ async fn serve(case: Case) -> Server { request.reject(moq_tokio::server::Reject::App(CODE)).await.ok(); } Case::Abort | Case::AbortThenDrop => { - let Ok(session) = request.ok().await else { return }; + let Ok((session, _driver)) = request.ok().await else { + return; + }; session.abort(moq_net::Error::App(CODE)); if let Case::Abort = case { // Hold the handle until the test's runtime shuts down. diff --git a/rs/moq-tokio/tests/max_age.rs b/rs/moq-tokio/tests/max_age.rs index fe1233b0aac..8d80fe0069d 100644 --- a/rs/moq-tokio/tests/max_age.rs +++ b/rs/moq-tokio/tests/max_age.rs @@ -30,7 +30,8 @@ async fn relay(version: moq_net::Version, javascript: Option) -> anyhow::R .await?, ); } - for session in sessions { + for (session, driver) in sessions { + driver.await??; let _ = session.closed().await; } Ok::<_, anyhow::Error>(()) diff --git a/rs/moq-tokio/tests/reconnect.rs b/rs/moq-tokio/tests/reconnect.rs index 1fd0877d6e9..ab60b026ed2 100644 --- a/rs/moq-tokio/tests/reconnect.rs +++ b/rs/moq-tokio/tests/reconnect.rs @@ -78,7 +78,7 @@ async fn a_handshake_slower_than_the_retry_window_still_lands() { let _server = tokio::spawn(async move { let origin = moq_tokio::origin::spawn(); while let Some(request) = server.accept().await { - if let Ok(session) = request.with_publisher(&origin).ok().await { + if let Ok((session, _driver)) = request.with_publisher(&origin).ok().await { let _ = accepted.send(session); } } @@ -187,7 +187,7 @@ async fn spawn_server() -> ( while let Some(request) = server.accept().await { let origin = moq_tokio::origin::spawn(); match request.with_publisher(&origin).ok().await { - Ok(session) => { + Ok((session, _driver)) => { let _ = accepted.send(session); } Err(err) => tracing::warn!(%err, "accept failed"), @@ -401,7 +401,7 @@ async fn rides_out_a_drain(backoff: moq_tokio::Backoff, drained: usize) { // Held, as a draining server does: each one closes when the client leaves it. let mut held = Vec::new(); while let Some(request) = server.accept().await { - let Ok(session) = request.with_publisher(&origin).ok().await else { + let Ok((session, _driver)) = request.with_publisher(&origin).ok().await else { continue; }; if held.len() < drained { diff --git a/rs/moq-tokio/tests/subscription_end_integrity.rs b/rs/moq-tokio/tests/subscription_end_integrity.rs index 2bf2d11c2e0..14e45bae069 100644 --- a/rs/moq-tokio/tests/subscription_end_integrity.rs +++ b/rs/moq-tokio/tests/subscription_end_integrity.rs @@ -120,7 +120,7 @@ async fn round(drop_session: bool) -> (usize, Option) { .await .expect("accept timeout") .expect("no incoming connection"); - let session = request + let (session, _driver) = request .with_publisher(&pub_origin) .ok() .await diff --git a/rs/moq-tokio/tests/worker.rs b/rs/moq-tokio/tests/worker.rs index c31196628b9..c3acfee58f3 100644 --- a/rs/moq-tokio/tests/worker.rs +++ b/rs/moq-tokio/tests/worker.rs @@ -741,7 +741,7 @@ async fn rotated_identity_reaches_every_worker() { let send = send.clone(); let certificates = certificates.clone(); tokio::spawn(async move { - let session = request.ok().await.unwrap(); + let (session, _driver) = request.ok().await.unwrap(); send.send((index, certificates.fingerprints())).unwrap(); let _ = session.closed().await; });