diff --git a/doc/bin/hls.md b/doc/bin/hls.md index 490a0323af0..025b1117fc5 100644 --- a/doc/bin/hls.md +++ b/doc/bin/hls.md @@ -28,7 +28,9 @@ ended, such as a VOD recording, is refused with an error rather than imported. An `EXT-X-DISCONTINUITY` keeps the source's timestamps. Timestamp resets are not supported: if a rendition's timestamps go backwards under the same `EXT-X-MAP`, such as an encoder restart, the import fails with an error. Import the restarted -source as a new broadcast. +source as a new broadcast. Import fetches only from the playlist's origin +(scheme, host, and port): a reference or redirect to another origin, such as a +separate CDN host, fails the import. ## How export works diff --git a/quest/m0/hls-admission-backport.md b/quest/m0/hls-admission-backport.md index d17d6a1a939..8db37c735be 100644 --- a/quest/m0/hls-admission-backport.md +++ b/quest/m0/hls-admission-backport.md @@ -4,7 +4,7 @@ `release`, which moq.dev runs, refuses the same hostile HLS playlists `main` does: the origin-admission and redirect-hop checks from -[HLS admission](/quest/m1/release-audit/hls-admission.md) ship in a +[moq-dev/moq#6374](https://github.com/moq-dev/moq/pull/6374) ship in a non-breaking `moq-hls` patch release. ## Plan @@ -20,7 +20,3 @@ on `main`, following the shape of `release`, no breaking change, a patch release. Public API: none on `release`. Wire: none. - -## Required - -- [HLS admission](/quest/m1/release-audit/hls-admission.md) - the checks land and are validated on `main` first diff --git a/quest/m1/release-audit/README.md b/quest/m1/release-audit/README.md index 7fa6e8165ff..4fd946c1c9b 100644 --- a/quest/m1/release-audit/README.md +++ b/quest/m1/release-audit/README.md @@ -76,8 +76,7 @@ indexes, JS IETF timestamp changes, JS datagram range violations, and OBS reordered-frame timestamps and missing EOF output. Each owning quest records the reproduction's limits. JS reader and publisher datagram paths now enforce subscription bounds; the OBS quest retains ownership of its findings. -Go reservation disposal remains an investigation, and -the security-sensitive HLS admission item is deliberately a public pointer. +Go reservation disposal remains an investigation. No platform-wide binding parity, full OBS runtime, fuzzing campaign, or network-impairment performance result is implied by these checks. @@ -88,11 +87,10 @@ work, because obs-moq ships on its own schedule. ## Required -The entries comprise reproduced or source-confirmed findings, -contract/lifetime investigations, and one security-review pointer. Each +The entries comprise reproduced or source-confirmed findings +and contract/lifetime investigations. Each child records whether its behavior is new on main or inherited from release. -- [Review HLS admission](/quest/m1/release-audit/hls-admission.md) - resolve the security-sensitive resource-boundary review before the cut - [Decode order](/quest/m1/ts-reliability/decode-order.md) - the exporters' decode clock holds at most the declared reorder depth, so flat timestamps cannot bypass MKV/TS backpressure or cause quadratic draining - [Release AUTH wake listeners](/quest/m1/release-audit/auth-listeners.md) - session churn does not retain one terminal promise reaction per change - [Finish bitrate recovery](/quest/m1/release-audit/rate-recovery.md) - a quiet bandwidth grant no longer strands moq-video, moq-ffi, and moq-c encoders partway up the ramp diff --git a/quest/m1/release-audit/hls-admission.md b/quest/m1/release-audit/hls-admission.md deleted file mode 100644 index 2b417eb4d81..00000000000 --- a/quest/m1/release-audit/hls-admission.md +++ /dev/null @@ -1,33 +0,0 @@ -# [M] Review HLS resource admission boundaries - -## Goal - -HLS import preserves the caller's intended resource-access boundary across -secondary resource resolution and fetching. - -## Plan - -Security-sensitive follow-up from source review at main 1d43c096e0, also -relevant to release. Review resource admission in `rs/moq-hls/src/import.rs` -before the cut. Establish the affected configurations, address the boundary -at its owner, and add regression coverage for allowed and refused resources. -This public quest is deliberately a brief pointer at the maintainer's -request; it does not include sensitive reproduction details or claim a -demonstrated exploit. Report any caller-visible policy or API change and -consider a release backport after validation. - -Planning settled from the 2026-10-22 security audit: - -- A reference is accepted only when the URL it finally resolves to has the - playlist's scheme and authority, whatever form the reference took - (absolute, network-path, or relative). Check once where references - resolve so every resource kind shares the rule, and cover each form in - the tests. -- Every redirect hop is held to the same rule before it is followed. -- `file:` resources are read only when the playlist itself is a `file:` URL, - which keeps local fixtures working. -- A caller-supplied `Config::client` keeps its own redirect policy, which the - importer cannot override per request. It has no in-repo consumer, so - propose dropping or narrowing it on `main` (a migration note if it - changes); a release port that keeps the API documents that the caller's - client owns redirect handling. diff --git a/quest/m2/hls-variant-area.md b/quest/m2/hls-variant-area.md index ec627af1f64..16c6c9e66fe 100644 --- a/quest/m2/hls-variant-area.md +++ b/quest/m2/hls-variant-area.md @@ -11,7 +11,3 @@ Today `select_variants` in `rs/moq-hls/src/import.rs` computes `width * height` From the 2026-10-22 security audit. Compute the area as an exact widened product (for example `u128` from the two `u64` dimensions), so every resolution ranks by its true area. Saturating would collapse large areas into ties that bandwidth and map order then break arbitrarily. A test with an enormous `RESOLUTION` covers both the panic and the ordering. Public API: none. Wire: none. - -## Related - -- [HLS admission](/quest/m1/release-audit/hls-admission.md) - the other HLS import hardening from the audit diff --git a/rs/moq-hls/src/error.rs b/rs/moq-hls/src/error.rs index 4aa69209aec..77a7c02f256 100644 --- a/rs/moq-hls/src/error.rs +++ b/rs/moq-hls/src/error.rs @@ -42,6 +42,10 @@ pub enum Error { #[error("requested range is unavailable: {0}")] RangeUnavailable(String), + /// An import resource or redirect hop left the playlist's origin, or the playlist uses an unsupported scheme. + #[error("resource is outside the playlist's origin")] + ResourceRefused, + /// A `file://` URL could not be turned back into a filesystem path. #[error("invalid file URL")] InvalidFileUrl, diff --git a/rs/moq-hls/src/import.rs b/rs/moq-hls/src/import.rs index 095755eb802..4d5653076a7 100644 --- a/rs/moq-hls/src/import.rs +++ b/rs/moq-hls/src/import.rs @@ -57,26 +57,19 @@ const MAX_RENDITION_FAILURES: usize = 3; #[non_exhaustive] pub struct Config { /// The master or media playlist URL to import. Local files use a `file://` URL. + /// Every resource and redirect hop must share its scheme, host, and port. pub playlist: Url, - /// HTTP client used to fetch the playlist and segments, for example one carrying - /// credentials for an authenticated origin. Defaults to a plain client with a 30 - /// second per-request timeout. - /// - /// This is a [`reqwest::Client`]. - pub client: Option, - /// Republish each fMP4 fragment whole as CMAF instead of splitting it into one Legacy frame per /// sample. See [`fmp4::Import::with_passthrough`](moq_mux::container::fmp4::Import::with_passthrough). pub passthrough: bool, } impl Config { - /// Create an import configuration for `playlist` using the default HTTP client. + /// Create an import configuration for `playlist`. pub fn new(playlist: Url) -> Self { Self { playlist, - client: None, passthrough: false, } } @@ -180,23 +173,49 @@ fn resolve_map(url: Url, range: Option<&ByteRange>) -> Result { RangeCursor::default().resolve(url, range) } +/// Whether `url` shares `entry`'s scheme, host, and port. +/// +/// `file:` URLs have opaque origins in [`Url::origin`], so compare the parts directly. +fn same_origin(entry: &Url, url: &Url) -> bool { + entry.scheme() == url.scheme() + && entry.host() == url.host() + && entry.port_or_known_default() == url.port_or_known_default() +} + /// Fetches playlists and segments over HTTP or from the filesystem, applying HLS byte /// ranges (and validating that the server honored them). struct Fetcher { client: Client, + /// The caller's playlist URL. A playlist is untrusted, so every fetch stays on its origin. + entry: Url, } impl Fetcher { - fn new(client: Option) -> Result { - let client = match client { - Some(client) => client, - None => Client::builder() - .user_agent(concat!(env!("CARGO_PKG_NAME"), "/", env!("CARGO_PKG_VERSION"))) - // Bound playlist/segment fetches so a stuck request can't wedge `run()`. - .timeout(REQUEST_TIMEOUT) - .build()?, - }; - Ok(Self { client }) + fn new(config: &Config) -> Result { + let entry = config.playlist.clone(); + if !matches!(entry.scheme(), "file" | "http" | "https") { + return Err(Error::ResourceRefused); + } + + // An intermediate hop is already a request to its origin, so check each one before + // following it. A stopped hop reaches `fetch` as a redirect response, refused there. + let origin = entry.clone(); + let follow = reqwest::redirect::Policy::default(); + let redirect = reqwest::redirect::Policy::custom(move |attempt| { + if same_origin(&origin, attempt.url()) { + follow.redirect(attempt) + } else { + attempt.stop() + } + }); + + let client = Client::builder() + .user_agent(concat!(env!("CARGO_PKG_NAME"), "/", env!("CARGO_PKG_VERSION"))) + // Bound playlist/segment fetches so a stuck request can't wedge `run()`. + .timeout(REQUEST_TIMEOUT) + .redirect(redirect) + .build()?; + Ok(Self { client, entry }) } async fn media_playlist(&self, url: Url) -> Result { @@ -214,6 +233,9 @@ impl Fetcher { async fn fetch(&self, resource: &Resource) -> Result { let url = &resource.url; + if !same_origin(&self.entry, url) { + return Err(Error::ResourceRefused); + } if url.scheme() == "file" { let path = url.to_file_path().map_err(|_| Error::InvalidFileUrl)?; let Some(range) = resource.range else { @@ -230,6 +252,9 @@ impl Fetcher { self.validate_range_length(resource, Bytes::from(bytes)) } else { let response = self.request(resource).send().await.map_err(Error::from)?; + if response.status().is_redirection() { + return Err(Error::ResourceRefused); + } let response = response.error_for_status().map_err(Error::from)?; let partial = response.status() == reqwest::StatusCode::PARTIAL_CONTENT; if partial { @@ -662,10 +687,11 @@ pub struct Import { impl Import { /// Create a new HLS import that will write into the given broadcast. pub fn new(broadcast: moq_net::broadcast::Producer, catalog: CatalogProducer, cfg: Config) -> Result { + let fetcher = Fetcher::new(&cfg)?; let base_url = cfg.playlist; Ok(Self { sink: Sink::new(broadcast, catalog, cfg.passthrough), - fetcher: Fetcher::new(cfg.client)?, + fetcher, base_url, video: Vec::new(), audio: None, @@ -736,6 +762,7 @@ impl Import { deadline = moq_clock::now() + STEP_RETRY_BUDGET; outcome } + Err(err @ Error::ResourceRefused) => return Err(err), // A settled answer, such as a 404 playlist, is not going to become a 200 on the next // pass. Everything else rides the backoff budget. Err(err) if err.settled() => { @@ -849,6 +876,8 @@ impl Import { wrote_segments += count; ok += 1; } + // Admission is settled by the caller's input policy, not rendition health. + Err(err @ Error::ResourceRefused) => return Err(err), Err(err) => match on_error { OnError::Fail => return Err(err), // A playlist that ended before this rendition saw it live is refused input, not a @@ -1192,13 +1221,7 @@ mod tests { (init, fragments) } - async fn serve_response( - status: &str, - headers: &[(&str, &str)], - body: &[u8], - ) -> (Url, tokio::task::JoinHandle>) { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let address = listener.local_addr().unwrap(); + fn http_response(status: &str, headers: &[(&str, &str)], body: &[u8]) -> Vec { let mut response = format!( "HTTP/1.1 {status}\r\nContent-Length: {}\r\nConnection: close\r\n", body.len() @@ -1209,24 +1232,42 @@ mod tests { response.push_str("\r\n"); let mut response = response.into_bytes(); response.extend_from_slice(body); + response + } - let server = tokio::spawn(async move { - let (mut stream, _) = listener.accept().await.unwrap(); - let mut request = Vec::new(); - loop { - let mut chunk = [0; 1024]; - let read = stream.read(&mut chunk).await.unwrap(); - if read == 0 { - break; - } - request.extend_from_slice(&chunk[..read]); - if request.windows(4).any(|window| window == b"\r\n\r\n") { - break; + /// Answer one connection per canned response, in order, returning each request. + fn serve(listener: TcpListener, responses: Vec>) -> tokio::task::JoinHandle>> { + tokio::spawn(async move { + let mut requests = Vec::new(); + for response in responses { + let (mut stream, _) = listener.accept().await.unwrap(); + let mut request = Vec::new(); + loop { + let mut chunk = [0; 1024]; + let read = stream.read(&mut chunk).await.unwrap(); + if read == 0 { + break; + } + request.extend_from_slice(&chunk[..read]); + if request.windows(4).any(|window| window == b"\r\n\r\n") { + break; + } } + stream.write_all(&response).await.unwrap(); + requests.push(request); } - stream.write_all(&response).await.unwrap(); - request - }); + requests + }) + } + + async fn serve_response( + status: &str, + headers: &[(&str, &str)], + body: &[u8], + ) -> (Url, tokio::task::JoinHandle>>) { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let server = serve(listener, vec![http_response(status, headers, body)]); (Url::parse(&format!("http://{address}/media.mp4")).unwrap(), server) } @@ -1245,8 +1286,141 @@ mod tests { ) } - fn fetcher() -> Fetcher { - Fetcher::new(None).unwrap() + fn fetcher(entry: &Url) -> Fetcher { + Fetcher::new(&Config::new(entry.clone())).unwrap() + } + + #[test] + fn same_origin_references_are_admitted() { + let entry = Url::parse("https://example.com/live/master.m3u8").unwrap(); + for reference in [ + "video.m3u8", + "../a.mp4", + "/a.mp4", + "//example.com/a.mp4", + "https://example.com:443/a.mp4", + "HTTPS://EXAMPLE.COM/a.mp4", + ] { + let url = resolve_uri(&entry, reference).unwrap(); + assert!(same_origin(&entry, &url), "{reference}"); + } + } + + #[tokio::test] + async fn cross_origin_references_are_refused() { + let dir = temp_dir(); + let path = dir.join("resource"); + std::fs::write(&path, b"sentinel").unwrap(); + let local = Url::from_file_path(dir.join("media.m3u8")).unwrap(); + let remote = Url::parse("https://example.com/live/media.m3u8").unwrap(); + let file = Url::from_file_path(&path).unwrap(); + let mut downgraded = remote.clone(); + downgraded.set_scheme("http").unwrap(); + for (entry, reference) in [ + (&remote, "https://other.example/live/a.mp4"), + (&remote, "//other.example/live/a.mp4"), + (&remote, downgraded.as_str()), + (&remote, "https://example.com:8443/live/a.mp4"), + (&remote, file.as_str()), + (&local, "https://example.com/live/a.mp4"), + (&local, "//example.com/live/a.mp4"), + ] { + // Refused before any I/O: none of these hosts resolve, and the file exists. + let url = resolve_uri(entry, reference).unwrap(); + let fetcher = fetcher(entry); + assert!( + matches!(fetcher.fetch_url(url.clone()).await, Err(Error::ResourceRefused)), + "{reference} from {entry}" + ); + assert!(matches!( + fetcher.fetch(&ranged_resource(url)).await, + Err(Error::ResourceRefused) + )); + } + } + + #[tokio::test] + async fn redirect_hops_stay_on_the_playlist_origin() { + // A same-origin hop is followed. + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let entry = Url::parse(&format!("http://{}/media.m3u8", listener.local_addr().unwrap())).unwrap(); + let server = serve( + listener, + vec![ + http_response("302 Found", &[("Location", "/moved.m3u8")], b""), + http_response("200 OK", &[], b"moved"), + ], + ); + assert_eq!(fetcher(&entry).fetch_url(entry.clone()).await.unwrap(), b"moved"[..]); + server.await.unwrap(); + + // A later hop to another origin is refused before it is requested. + let other = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + other.set_nonblocking(true).unwrap(); + let location = format!("http://{}/media.m3u8", other.local_addr().unwrap()); + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let entry = Url::parse(&format!("http://{}/media.m3u8", listener.local_addr().unwrap())).unwrap(); + let server = serve( + listener, + vec![ + http_response("302 Found", &[("Location", "/hop.m3u8")], b""), + http_response("302 Found", &[("Location", &location)], b""), + ], + ); + assert!(matches!( + fetcher(&entry).fetch_url(entry.clone()).await, + Err(Error::ResourceRefused) + )); + server.await.unwrap(); + assert_eq!(other.accept().unwrap_err().kind(), std::io::ErrorKind::WouldBlock); + } + + #[tokio::test] + async fn admission_failure_ends_import_without_retrying() { + tokio::time::pause(); + let sink = sink(); + let mut import = Import::new( + sink.broadcast.clone(), + sink.catalog.clone(), + Config::new(Url::parse("https://example.com/master.m3u8").unwrap()), + ) + .unwrap(); + import.video.push(TrackState::new( + "video[0]".to_string(), + Url::parse("file:///refused.m3u8").unwrap(), + select_muxed(), + sink, + )); + assert!(matches!( + futures::FutureExt::now_or_never(import.run()), + Some(Err(Error::ResourceRefused)) + )); + } + + #[tokio::test] + async fn local_import_allows_local_resources() { + let dir = temp_dir(); + let path = dir.join("resource"); + std::fs::write(&path, b"sentinel").unwrap(); + let url = Url::from_file_path(&path).unwrap(); + let fetcher = Fetcher::new(&Config::new(url.clone())).unwrap(); + assert_eq!(fetcher.fetch_url(url.clone()).await.unwrap(), b"sentinel"[..]); + assert_eq!(fetcher.fetch(&ranged_resource(url)).await.unwrap(), b"nti"[..]); + } + + #[tokio::test] + async fn unsupported_resources_are_refused() { + let entry = Url::parse("https://example.com/master.m3u8").unwrap(); + for scheme in ["ftp", "data", "custom"] { + let url = Url::parse(&format!("{scheme}://example.com/resource?token=secret")).unwrap(); + assert!(matches!( + Fetcher::new(&Config::new(url.clone())), + Err(Error::ResourceRefused) + )); + let error = fetcher(&entry).fetch_url(url).await.unwrap_err(); + assert!(matches!(error, Error::ResourceRefused)); + assert!(!format!("{error} {error:?}").contains("secret")); + } } fn ranged_resource(url: Url) -> Resource { @@ -1472,8 +1646,8 @@ mod tests { async fn ranged_http_resource_accepts_matching_partial_response() { let (url, server) = serve_response("206 Partial Content", &[("Content-Range", "bytes 2-4/6")], b"cde").await; - let bytes = fetcher().fetch(&ranged_resource(url)).await.unwrap(); - let request = String::from_utf8(server.await.unwrap()).unwrap(); + let bytes = fetcher(&url).fetch(&ranged_resource(url.clone())).await.unwrap(); + let request = String::from_utf8(server.await.unwrap().remove(0)).unwrap(); assert_eq!(bytes, b"cde".as_slice()); assert!( @@ -1487,7 +1661,7 @@ mod tests { async fn ranged_http_resource_slices_full_response() { let (url, server) = serve_response("200 OK", &[], b"abcdef").await; - let bytes = fetcher().fetch(&ranged_resource(url)).await.unwrap(); + let bytes = fetcher(&url).fetch(&ranged_resource(url.clone())).await.unwrap(); server.await.unwrap(); assert_eq!(bytes, b"cde".as_slice()); @@ -1497,7 +1671,7 @@ mod tests { async fn ranged_http_resource_rejects_wrong_response_length() { let (url, server) = serve_response("206 Partial Content", &[("Content-Range", "bytes 2-4/6")], b"cd").await; - let err = fetcher().fetch(&ranged_resource(url)).await.unwrap_err(); + let err = fetcher(&url).fetch(&ranged_resource(url.clone())).await.unwrap_err(); server.await.unwrap(); assert!(matches!( @@ -1514,7 +1688,7 @@ mod tests { async fn ranged_http_resource_rejects_mismatched_content_range() { let (url, server) = serve_response("206 Partial Content", &[("Content-Range", "bytes 3-5/6")], b"def").await; - let err = fetcher().fetch(&ranged_resource(url)).await.unwrap_err(); + let err = fetcher(&url).fetch(&ranged_resource(url.clone())).await.unwrap_err(); server.await.unwrap(); assert!(matches!(err, Error::ByteRangeResponseMismatch { start: 2, end: 4, .. })); @@ -1879,7 +2053,8 @@ mod tests { ); let before = "#EXTM3U\n#EXT-X-TARGETDURATION:2\n#EXT-X-MAP:URI=\"init.mp4\"\n#EXTINF:1,\n0.m4s\n"; let before = m3u8_rs::parse_media_playlist_res(before.as_bytes()).unwrap(); - track.consume_segments(&fetcher(), &before).await.unwrap(); + let local = fetcher(&track.playlist); + track.consume_segments(&local, &before).await.unwrap(); let name = |snapshot: &moq_mux::catalog::hang::Catalog| { if video { snapshot.video.renditions.keys().next() @@ -1902,7 +2077,7 @@ mod tests { "#EXTM3U\n#EXT-X-TARGETDURATION:2\n#EXT-X-MAP:URI=\"init.mp4\"\n#EXTINF:1,\n0.m4s\n{tags}#EXTINF:1,\n1.m4s\n" ); let after = m3u8_rs::parse_media_playlist_res(after.as_bytes()).unwrap(); - let result = track.consume_segments(&fetcher(), &after).await; + let result = track.consume_segments(&local, &after).await; let after_name = name(&catalog.snapshot()); let (old_groups, terminal) = group_stamps(&mut old); let new_groups = if before_name != after_name {