From f6e5f30a49f3086a61ca196891c079e6327d19ba Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 28 Oct 2026 22:29:47 -0700 Subject: [PATCH 1/3] chore(quest): claim quest/m1/security-audit/c-handle-wrap.md Quest-Claim: f60d50fb574f431b From 16c9fa33490f40da09982144b6f8cd717119e77c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 28 Oct 2026 22:36:27 -0700 Subject: [PATCH 2/3] fix(moq-c): wrap handle allocation and document callback safety Co-Authored-By: GPT-6.1 Sol --- doc/lib/c/index.md | 2 +- quest/m1/security-audit/README.md | 1 - quest/m1/security-audit/c-handle-wrap.md | 50 ----------- rs/moq-c/src/bandwidth.rs | 3 + rs/moq-c/src/ffi.rs | 10 +++ rs/moq-c/src/id.rs | 106 ++++++++++++++++++++--- sh/gh/nightly.sh | 1 + 7 files changed, 111 insertions(+), 62 deletions(-) delete mode 100644 quest/m1/security-audit/c-handle-wrap.md diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index cf9f889a4bd..e6a2c79ee5f 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -73,7 +73,7 @@ inside. The header is the reference: each function carries a doc comment. - **Handles and callbacks.** Every object is an integer handle, and every async result arrives on a `moq_status_callback` with your `void *user_data`. A status `> 0` is a live result, `0` a clean close, and `< 0` an error. The last two are terminal: moq-c never touches `user_data` again, so free it there. `*_close` and `*_cancel` only request shutdown; the terminal callback still fires. A session reports on a `moq_state_callback` instead: each `moq_connection_state` carries its `phase`, `connects`, the negative `failure` of the latest attempt, and the live `protocol`, and the `MOQ_CONNECTION_PHASE_CLOSED` snapshot is terminal, delivered after the handle is released. `moq_session_close()` drains finished tracks for dialed and accepted sessions; wait for that terminal snapshot to learn the result. - **Errors.** Negative return codes are named by `MOQ_ERROR_*`, and `moq_error()` gives the reason on the calling thread. Auth rejections have their own codes so you don't retry them. For the peer's protocol code use `moq_error_protocol()`; do not parse `moq_error()`. A panic in a binding background task reports `MOQ_ERROR_PANIC` through its terminal callback; unrelated objects keep running. A session closed because its credential expired has kind `MOQ_PROTOCOL_KIND_EXPIRED`, and a request refused for an expired token returns `MOQ_ERROR_EXPIRED`; refresh the token before retrying. A catalog that needs a newer moq-hang draft than this build reads returns `MOQ_ERROR_UNSUPPORTED_CATALOG`; upgrade the library to read it. A request-cap closure has kind `MOQ_PROTOCOL_KIND_TOO_MANY_REQUESTS`; reduce the workload or adjust the relay's [session limits](/bin/relay/config#session-and-peer) before reconnecting. - **Tokens.** Set `moq_client_config.tokens` instead of putting the credential in the URL. The first is the connection credential, sent in SETUP and, while an offered version can't carry it there (lite before 07), as the URL's `?jwt=`; each other token widens the grant over AUTH. A `?jwt=` already in the URL stays the connection credential, which is how to reach a relay older than this release. To replace expiring tokens, close the session and connect again. -- **Threading.** Call any function from any thread. Raw publish calls block until the codec takes the frame, which paces a publisher. +- **Threading.** Call any function from any thread. Callbacks run on moq-c's runtime thread; the callback and its `user_data` must be safe to use there. Synchronize access to data shared with other threads, and dispatch UI work to your UI thread. Raw publish calls block until the codec takes the frame, which paces a publisher. - **Zeroed config is the default.** A zeroed `moq_client_config` means the default for every field, so new fields don't break callers. A field whose default isn't zero has a `has_*` flag: `backoff_timeout_us = 0` means "retry forever" only with `has_backoff_timeout = true`. - **Compressed tracks.** `moq_track_info.compress` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames. - **fMP4 import.** `MOQ_CONTAINER_FORMAT_FMP4` splits each fragment into one frame per sample. diff --git a/quest/m1/security-audit/README.md b/quest/m1/security-audit/README.md index 8bcbb21bfa9..fce3ba70cf5 100644 --- a/quest/m1/security-audit/README.md +++ b/quest/m1/security-audit/README.md @@ -58,7 +58,6 @@ Dropped, with reasons: - [SETUP Path refused on URL bindings](/quest/m1/security-audit/setup-path.md) - a WebSocket or WebTransport client that also sends a SETUP Path is closed with PROTOCOL_VIOLATION, the same in moq-tokio and moq-uring - [fMP4 import streams mdat](/quest/m1/security-audit/fmp4-stream-mdat.md) - a box header declaring a huge size cannot pin importer memory across segments - [MKV import streams blocks](/quest/m1/security-audit/mkv-stream-blocks.md) - a declared block, metadata master, or ignored element cannot pin importer memory -- [moq-c handles wrap like PIDs](/quest/m1/security-audit/c-handle-wrap.md) - long-running hosts never exhaust handle IDs, and a wrap never reuses a live one - [Relay HTTP surface](/quest/m1/security-audit/relay-http.md) - `/fetch` cannot be sniffed as active content, and no web page can reach the ops listener - [Iroh secret written private](/quest/m1/security-audit/iroh-key.md) - a generated iroh key is created 0600 and never follows a symlink - [Lite fetch backstop](/quest/m1/security-audit/fetch-backstop.md) - the pre-lite-05 fetch backstop is bounded by design or by the session's limits, not an unrelated default diff --git a/quest/m1/security-audit/c-handle-wrap.md b/quest/m1/security-audit/c-handle-wrap.md deleted file mode 100644 index 30bfda10da8..00000000000 --- a/quest/m1/security-audit/c-handle-wrap.md +++ /dev/null @@ -1,50 +0,0 @@ -# [S] moq-c handles wrap like PIDs - -## Goal - -A long-running moq-c host never runs out of handle IDs, and no new handle -ever takes an ID that is still live. Closing a bandwidth consumer documents -that its callback still fires once afterward, like every other cancel path. -The C docs say which thread runs callbacks and that a callback and its -`user_data` must be safe to use from that thread. - -## Plan - -From the 2026-10-27 audit (FFI-2, FFI-1). `rs/moq-c/src/id.rs` allocates -every handle, including one per delivered frame, from one process-wide -`AtomicU32` and refuses IDs above `i32::MAX`. At 110 frames/s per broadcast, -eight consumed broadcasts (an OBS multiview) exhaust it in about 28 days, -after which every handle-creating call fails. Failed inserts still advance -the counter, so at 2^32 it wraps and `HashMap::insert` silently replaces a -live entry. - -Decided 2026-10-27 (maintainer): wrap like PIDs. Keep the monotonic counter, -wrap from `i32::MAX` back to 1, and skip IDs live in the target map. A stale -handle can alias only after about 2^31 intervening allocations, which -preserves the reason -[b0847860bf](https://github.com/moq-dev/moq/commit/b0847860bf) replaced the -reusing slab (parallel tests aliasing each other's freed indices) without the -exhaustion. Rejected: a generation-tagged slab (per-frame churn wraps the -generation within minutes) and removing per-frame handles (hosts may hold a -frame past the callback; breaks the C API). A unit test starts the counter -near `i32::MAX`. - -FFI-1: add the "terminal callback fires after close" line to -`moq_bandwidth_consumer_close` in `rs/moq-c/src/bandwidth.rs`, matching its -siblings in `api.rs`. - -The 2026-10-28 audit repeated the handle finding (frames, datagrams, and -decoded JSON values each take an ID, so a peer's send rate sets the clock) -and found an unwritten callback contract. `OnStatus` and `OnState` -(`rs/moq-c/src/ffi.rs`) are `unsafe impl Send` over the caller's `user_data`, -and background callbacks run on moq-c's runtime thread, but -`doc/lib/c/index.md` says only "call any function from any thread" and the -SAFETY comments require only that the pointer stay valid. Decided 2026-10-28 -(maintainer): document it here rather than in its own quest. Name the -callback thread under Threading in `doc/lib/c/index.md`, add the -thread-safety precondition to the callback typedefs' doc comments (cbindgen -copies them into the header), and state it in the `# Safety` sections of -`OnStatus::new` and `OnState::new` and in each `unsafe impl Send`'s SAFETY -comment. - -Public API: doc change only. Wire: none. diff --git a/rs/moq-c/src/bandwidth.rs b/rs/moq-c/src/bandwidth.rs index 99448ae6329..40666ceed13 100644 --- a/rs/moq-c/src/bandwidth.rs +++ b/rs/moq-c/src/bandwidth.rs @@ -248,6 +248,9 @@ pub unsafe extern "C" fn moq_bandwidth_consumer_changed( } /// Release this consumer's demand and cancel any outstanding wait. +/// +/// The outstanding `moq_bandwidth_consumer_changed` callback still fires once. +/// Release its `user_data` in that callback, not when this function returns. #[unsafe(no_mangle)] pub extern "C" fn moq_bandwidth_consumer_close(consumer: u32) -> i32 { ffi::enter(move || { diff --git a/rs/moq-c/src/ffi.rs b/rs/moq-c/src/ffi.rs index c942f85bb34..e4a03aef315 100644 --- a/rs/moq-c/src/ffi.rs +++ b/rs/moq-c/src/ffi.rs @@ -9,10 +9,14 @@ use url::Url; use crate::{Error, Id, moq_protocol_error}; /// A callback receiving a positive handle/value, zero on clean completion, or a negative error. +/// +/// Runs on moq-c's runtime thread. The callback and `user_data` must be safe to use on that thread. #[expect(non_camel_case_types, reason = "C ABI type names match moq.h")] pub type moq_status_callback = Option; /// A callback receiving each session lifecycle change; the `MOQ_CONNECTION_PHASE_CLOSED` one is terminal. +/// +/// Runs on moq-c's runtime thread. The callback and `user_data` must be safe to use on that thread. #[expect(non_camel_case_types, reason = "C ABI type names match moq.h")] pub type moq_state_callback = Option; @@ -209,6 +213,7 @@ impl OnStatus { /// # Safety /// - The caller must ensure user_data remains valid for the callback's lifetime. /// - The callback function pointer must be valid if provided. + /// - The callback and `user_data` must be safe to use on moq-c's runtime thread. pub unsafe fn new(user_data: *mut c_void, on_status: moq_status_callback) -> Result { Ok(Self { user_data, @@ -227,6 +232,8 @@ impl OnStatus { } } +// SAFETY: The constructor requires the callback and user_data to remain valid and +// safe to use on moq-c's runtime thread for the callback's lifetime. unsafe impl Send for OnStatus {} /// A [moq_state_callback] and its user data. @@ -241,6 +248,7 @@ impl OnState { /// /// # Safety /// - The caller must keep `user_data` valid until the terminal callback. + /// - The callback and `user_data` must be safe to use on moq-c's runtime thread. pub unsafe fn new(user_data: *mut c_void, on_state: moq_state_callback) -> Result { Ok(Self { user_data, @@ -258,6 +266,8 @@ impl OnState { } } +// SAFETY: The constructor requires the callback and user_data to remain valid and +// safe to use on moq-c's runtime thread until the terminal callback. unsafe impl Send for OnState {} /// Types that can be converted to C-compatible return codes. diff --git a/rs/moq-c/src/id.rs b/rs/moq-c/src/id.rs index 1887b0be1f5..5bf1927d7d6 100644 --- a/rs/moq-c/src/id.rs +++ b/rs/moq-c/src/id.rs @@ -1,4 +1,4 @@ -use std::collections::HashMap; +use std::collections::{HashMap, hash_map::Entry}; use std::num::NonZero; use std::sync::atomic::{AtomicU32, Ordering}; @@ -18,24 +18,39 @@ impl std::fmt::Display for Id { } } -/// Global monotonic counter so IDs are never reused across the process lifetime. +/// Process-wide counter wrapping from i32::MAX back to 1. static NEXT_ID: AtomicU32 = AtomicU32::new(1); -/// Map that assigns globally unique, never-reused IDs. +/// Map that assigns IDs from a process-wide counter, skipping live entries. /// -/// Unlike a slab, freed IDs are not recycled. This avoids races where -/// parallel consumers of the same global state accidentally alias each -/// other's resources. +/// Freed IDs are not recycled until the counter wraps. This avoids immediately +/// aliasing stale handles while allowing long-running hosts to keep allocating. pub(crate) struct NonZeroSlab { map: HashMap, } impl NonZeroSlab { pub fn insert(&mut self, value: T) -> Result { - let raw = NEXT_ID.fetch_add(1, Ordering::Relaxed); - let id = Id::try_from(raw)?; - self.map.insert(id, value); - Ok(id) + self.insert_next(value, &NEXT_ID) + } + + fn insert_next(&mut self, value: T, next: &AtomicU32) -> Result { + if self.map.len() == i32::MAX as usize { + return Err(Error::InvalidId); + } + + loop { + let raw = next + .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |id| { + Some(if id == i32::MAX as u32 { 1 } else { id + 1 }) + }) + .unwrap(); + let id = Id::try_from(raw)?; + if let Entry::Vacant(entry) = self.map.entry(id) { + entry.insert(value); + return Ok(id); + } + } } pub fn get(&self, id: Id) -> Option<&T> { @@ -87,3 +102,74 @@ impl Default for NonZeroSlab { Self { map: HashMap::new() } } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn wrap_uses_the_whole_positive_i32_range() { + let next = AtomicU32::new(i32::MAX as u32); + let mut slab = NonZeroSlab::default(); + let high = slab.insert_next((), &next).unwrap(); + assert_eq!(i32::from(high), i32::MAX); + let low = slab.insert_next((), &next).unwrap(); + assert_eq!(i32::from(low), 1); + assert_eq!(slab.remove(high), Some(())); + assert_eq!(i32::from(slab.insert_next((), &next).unwrap()), 2); + } + + #[test] + fn wrap_skips_live_handles_and_preserves_their_values() { + let next = AtomicU32::new(i32::MAX as u32 - 1); + let mut slab = NonZeroSlab::default(); + let low = Id::try_from(1u32).unwrap(); + let high = Id::try_from(i32::MAX).unwrap(); + slab.map.insert(low, "low"); + slab.map.insert(high, "high"); + + let before = slab.insert_next("before wrap", &next).unwrap(); + assert_eq!(i32::from(before), i32::MAX - 1); + let after = slab.insert_next("after wrap", &next).unwrap(); + assert_eq!(i32::from(after), 2); + assert_eq!(slab.get(low), Some(&"low")); + assert_eq!(slab.get(high), Some(&"high")); + assert_eq!(slab.get(before), Some(&"before wrap")); + assert_eq!(slab.get(after), Some(&"after wrap")); + + assert_eq!(slab.remove(low), Some("low")); + let following = slab.insert_next("following", &next).unwrap(); + assert_eq!(i32::from(following), 3); + } + + #[test] + #[ignore = "microbenchmark; runs in Nightly"] + fn benchmark_handles() { + use std::hint::black_box; + + for live in [0, 64, 4096] { + for wrap in [false, true] { + let next = AtomicU32::new(live + 1); + let mut slab = NonZeroSlab::default(); + for raw in 1..=live { + slab.map.insert(Id::try_from(raw).unwrap(), 0); + } + let iterations = 1000; + let start = moq_clock::Instant::now(); + for _ in 0..iterations { + if wrap { + next.store(i32::MAX as u32, Ordering::Relaxed); + let high = slab.insert_next(black_box(42), &next).unwrap(); + black_box(slab.remove(high).unwrap()); + } + let id = slab.insert_next(black_box(42), &next).unwrap(); + black_box(slab.remove(id).unwrap()); + } + eprintln!( + "handles live={live} wrap={wrap}: {:.0} ns/iteration", + start.elapsed().as_nanos() as f64 / f64::from(iterations) + ); + } + } + } +} diff --git a/sh/gh/nightly.sh b/sh/gh/nightly.sh index 89403170435..7f8bc0d4698 100755 --- a/sh/gh/nightly.sh +++ b/sh/gh/nightly.sh @@ -84,6 +84,7 @@ case "$1" in run just rs test -p moq-quic --release -E 'test(benchmark_receive_credit)' --run-ignored only --success-output immediate run just rs test -p moq-tokio --release -E 'test(benchmark_receive_windows)' --run-ignored only --success-output immediate run just rs test -p moq-relay -E 'test(benchmark_stun_sources)' --run-ignored only --success-output immediate + run just rs test -p moq-c -E 'test(benchmark_handles)' --run-ignored only --success-output immediate run just rs test -p moq-archive -p moq-mux -p moq-hls --no-fail-fast \ -E 'test(benchmark_timeline_fetch_fanout) | test(benchmark_damage_pid_fanout) | test(benchmark_pes_staging) | test(benchmark_reference_handoff_fanout)' \ --run-ignored only --success-output immediate From 015750db953c99e94ed6182e76c343b03ffcdc0f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 08:48:25 -0700 Subject: [PATCH 3/3] docs(moq-c): scope Id uniqueness to live resources of one kind Co-Authored-By: Claude Opus 5.5 --- rs/moq-c/src/id.rs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/rs/moq-c/src/id.rs b/rs/moq-c/src/id.rs index 5bf1927d7d6..420fc6796fb 100644 --- a/rs/moq-c/src/id.rs +++ b/rs/moq-c/src/id.rs @@ -6,9 +6,10 @@ use crate::Error; /// Opaque resource identifier returned to C code. /// -/// Non-zero u32 value that uniquely identifies resources like sessions, -/// origins, broadcasts, tracks, etc. Zero is reserved to indicate "none" -/// or optional parameters. +/// Non-zero u32 value that uniquely identifies a live resource of one kind, +/// such as a session, origin, broadcast, or track. After the counter wraps, +/// resources of different kinds may share a value. Zero is reserved to +/// indicate "none" or optional parameters. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub struct Id(NonZero);