diff --git a/quest/m0/README.md b/quest/m0/README.md index d6c4b15d3a4..8c3f4064763 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -24,7 +24,6 @@ edge, which embeds moq-relay; it moved there on 2026-09-28. - [Release credential leaks](/quest/m0/security-audit/README.md) - tokens stay out of logs and outbound HTTP clients verify their peer, on `main` and `release` - [Interop and Video pass on main](/quest/m0/interop-green/README.md) - the scheduled Interop and Video workflows pass, so red means a new break - [Release shutdown backport](/quest/m0/ffi-shutdown-backport.md) - `release` publishes a non-breaking moq-ffi with an async `drain()` and qmux sessions on the QUIC stream limit -- [Backport @moq/flate bounds](/quest/m0/backport-flate-budget.md) - `@moq/flate` on `release` stops inflating past its cap and stops pinning pako scratch - [Backport audio groupDuration](/quest/m0/backport-audio-group-duration.md) - `@moq/publish` 0.5.4 on `release` can pack several Opus frames per group, making Cloudflare's relay drop audio less often - [Backport the SETUP-path and iroh-key fixes](/quest/m0/backport-security-fixes.md) - `release` refuses a conflicting SETUP Path and keeps persisted iroh keys private, in one patch release - [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in a batched merge gate as one `just ci` pass per batch, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last diff --git a/quest/m0/backport-flate-budget.md b/quest/m0/backport-flate-budget.md deleted file mode 100644 index 693f50ff293..00000000000 --- a/quest/m0/backport-flate-budget.md +++ /dev/null @@ -1,23 +0,0 @@ -# [S] Backport the @moq/flate inflate bounds to release - -## Goal - -`@moq/flate` on `release` stops inflating a hostile slice once it passes its -cap and stops pinning pako's scratch buffer, as -[moq-dev/moq#6900](https://github.com/moq-dev/moq/pull/6900) fixed on main, and -publishes a patch release (npm has 0.1.0 to 0.1.3). - -## Plan - -Decided 2026-10-27 (maintainer): one PR against `release` per backport, in m0. - -`release`'s `js/flate/src/index.ts` is the older per-frame version (a 64 MiB -`maxFrameSize`, not main's 32 MiB group budget), so port the two fixes onto its -shape rather than main's budget: - -- `onData` only returns past the cap, so pako inflates the rest of the slice. - Stop inflation at the cap and fail the frame. -- `concat()` passes a single chunk through, pinning pako's scratch buffer. - Copy it out. - -Public API: none. Wire: none. diff --git a/quest/m1/README.md b/quest/m1/README.md index fe863117b99..a6a89d23d21 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -152,7 +152,6 @@ removed). - [moq-transport carries no epoch](/quest/m1/ietf-epoch-cut.md) - Rust and JS moq-transport sessions stop negotiating MoQ Epoch, so every moq-transport hop is epochless; the draft stays - [moq-quic's API improves on quinn's](/quest/m1/quic-api/README.md) - moq-quic and both QUIC drivers ship one mirrored API: `send::`/`recv::` namespaces, one error per direction, consuming terminal operations, plain configs, no quinn baggage - [On-demand JavaScript FETCH](/quest/m1/js-fetch-on-demand.md) - browser publishers answer IETF FETCH for evicted or uncached groups through the JS ranges request surface -- [Rust FETCH parity](/quest/m1/ietf-fetch-parity.md) - Rust finishes an answered joining FETCH's head, refuses a lost head with DOES_NOT_EXIST, and sends no timestamps without TIMESCALE, as the JS publisher does - [Rust cancels an unclaimed fill head when a subscription ends](/quest/m1/ietf-fill-end-cancel.md) - SUBSCRIPTION_ENDED cancels, not finishes, a fill no group was sent for, as in JS - [SUBSCRIBE honors the properties opt-out](/quest/m1/subscribe-timestamps.md) - the Rust publisher sends untimed objects to a SUBSCRIBE that declines Track Properties, and the Rust subscriber asks for them when it needs timestamps - [A resumed untimed copy keeps DYNAMIC_GROUPS](/quest/m1/ietf-dynamic-groups-resume.md) - opting out of properties on a draft-22 resume no longer forgets DYNAMIC_GROUPS diff --git a/quest/m1/ietf-fetch-parity.md b/quest/m1/ietf-fetch-parity.md deleted file mode 100644 index cf987ee5fb4..00000000000 --- a/quest/m1/ietf-fetch-parity.md +++ /dev/null @@ -1,55 +0,0 @@ -# [S] Rust answers IETF FETCH as the JS publisher does - -## Goal - -Rust's IETF publisher (`rs/moq-net/src/ietf/publisher.rs`) matches the JS -publisher from [moq-dev/moq#6945](https://github.com/moq-dev/moq/pull/6945) on -three FETCH behaviors: - -1. A joining FETCH that already sent FETCH_OK finishes sending its head even - if its subscription ends first. Today Rust resets it with the subscription. -2. A joining FETCH whose head group the cache lost is refused with - DOES_NOT_EXIST before FETCH_OK. Today Rust rejects it with - `Error::Evicted`, which goes out as INTERNAL_ERROR. -3. A FETCH that opts out of Track Properties (INCLUDE_PROPERTIES `0`) gets - no TIMESCALE in FETCH_OK and no timestamps on its objects, so the track - stays untimed for that requester. Today Rust still stamps each object, which - the requester has no timescale to read. - -Non-goals: answering a joining FETCH that arrives after its subscription -ended ([Join after a quick finish](/quest/m2/ietf-join-after-finish.md)), -and draft-22 fills. - -## Plan - -Decided 2026-10-27 (maintainer, planning the follow-up of -[moq-dev/moq#6945](https://github.com/moq-dev/moq/pull/6945)): the JS behavior -is the better one on all three, so Rust moves to it rather than JS back. -One quest, since each is a small change to the same publisher, in m1 so both -publishers answer FETCH the same way at the cut. - -- **In-flight head.** Once FETCH_OK has gone out, the head group is already - held and the requester can still cancel, so the FETCH keeps its own - lifetime instead of the subscription's. It holds nothing a hostile peer - could not already make it hold through an open FETCH. Check whether this - is part of the draft-18 `tail` flake with a Rust publisher that - [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) describes, - and note the result there. -- **Lost head.** Refuse with DOES_NOT_EXIST, as a missing prefix already is - (`a_joining_fetch_refuses_a_missing_prefix_before_fetch_ok`). Update - `a_joining_fetch_refuses_a_prefix_group_it_lost`. -- **Timestamps.** Only TIMESCALE opts a track into timestamps, so the object - writer takes the timescale only when Track Properties are wanted. - -Test each in `publisher.rs`'s unit tests on mock time. Run -`just test interop --tail --publishers rust --versions ietf` and -`just test interop --all`. - -Public API: none. Wire: Rust sends a different refusal code for a lost -head, and omits object timestamps when the requester opted out of Track -Properties. - -## Related - -- [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) - the late joining FETCH this leaves out -- [Headless moq-transport groups](/quest/m2/ietf-headless-group.md) - what a subscriber does with a tail whose head never comes diff --git a/quest/m1/ietf-publisher-unknown-range.md b/quest/m1/ietf-publisher-unknown-range.md index 39d403122bd..ed7ea39828d 100644 --- a/quest/m1/ietf-publisher-unknown-range.md +++ b/quest/m1/ietf-publisher-unknown-range.md @@ -20,8 +20,10 @@ than only an eviction. `NotFound` and `Old` map to `0x10C`. Decide in the PR whether `Unsupported` and `NotFetchable` mean "unknown" too, or need their own handling, and record the reason. -Decided 2026-10-28 (maintainer): it starts after the three quests below land, -since each edits the same publisher. The third was IETF fill timeout, which +Decided 2026-10-28 (maintainer): it starts after the quests below land, +since each edits the same publisher. Rust FETCH parity, also listed then, +landed in [moq-dev/moq#6999](https://github.com/moq-dev/moq/pull/6999). A +third was IETF fill timeout, which [moq-dev/moq#7194](https://github.com/moq-dev/moq/pull/7194) dropped for [one group](https://github.com/moq-dev/moq/pull/7204), which edited the same paths and has landed. A FETCH or fill now lists at most one group, so this @@ -34,5 +36,4 @@ Public API: none. Wire: none; the marker exists in both drafts. ## Required -- [Rust FETCH parity](/quest/m1/ietf-fetch-parity.md) - edits the same FETCH answer path - [SUBSCRIBE honors the properties opt-out](/quest/m1/subscribe-timestamps.md) - edits the same publisher diff --git a/quest/m1/ietf-withdraw-credit.md b/quest/m1/ietf-withdraw-credit.md index a8f5800f7ea..9c784b867ab 100644 --- a/quest/m1/ietf-withdraw-credit.md +++ b/quest/m1/ietf-withdraw-credit.md @@ -16,16 +16,12 @@ flow-control credit pins a withdrawn advertisement. Decided 2026-10-28 (maintainer): its own quest rather than part of [one request lease](/quest/m2/ietf-request-lease.md), since a peer making us -hold state is a hostile-peer bound to fix now; it waits for -[Rust FETCH parity](/quest/m1/ietf-fetch-parity.md) only because that PR edits -the same file. Use #7016's pattern: buffer each message and flush it inside +hold state is a hostile-peer bound to fix now; it waited for Rust FETCH parity +([moq-dev/moq#6999](https://github.com/moq-dev/moq/pull/6999)) only because +that PR edits the same file. Use #7016's pattern: buffer each message and flush it inside the wait that already watches for withdrawal, and reset the stream rather than FIN it when withdrawn before the message fully left. Test with a peer that never grants send credit, on drafts 18 and 22. Public API: none. Wire: none. - -## Required - -- [Rust FETCH parity](/quest/m1/ietf-fetch-parity.md) - edits `publisher.rs`, so this follows it diff --git a/quest/m2/ietf-join-after-finish.md b/quest/m2/ietf-join-after-finish.md index fa1f5271071..94ca706ed1b 100644 --- a/quest/m2/ietf-join-after-finish.md +++ b/quest/m2/ietf-join-after-finish.md @@ -32,7 +32,11 @@ It flakes the pinned-pass interop `tail` cells with a Rust publisher, about one run in six (seen 2026-10-22 while landing [moq-dev/moq#6407](https://github.com/moq-dev/moq/pull/6407)): the tail test finishes its track before SUBSCRIBE_OK, so PUBLISH_DONE follows within -milliseconds. The relay's settle grace (about a second) ends long before the +milliseconds. A FETCH that already had FETCH_OK being reset with its +subscription is not the cause: it has finished since +[moq-dev/moq#6999](https://github.com/moq-dev/moq/pull/6999), and neither +lifetime failed in six runs of the draft-18 Rust-publisher `tail` cells on +2026-10-28. The relay's settle grace (about a second) ends long before the 10 s registration wait refuses the FETCH, and the empty subscription stream for the Largest group then becomes an empty group, which `rust -> rust` and `rust -> js-native-*` read as "empty group". diff --git a/rs/moq-net/src/auth.rs b/rs/moq-net/src/auth.rs index 353f54f13a4..408631a4d77 100644 --- a/rs/moq-net/src/auth.rs +++ b/rs/moq-net/src/auth.rs @@ -1841,7 +1841,7 @@ pub(crate) struct RequestVerdict { /// Why a request token did not authorize its request: the error whose code the peer is /// sent, and the reason phrase with it, which the acceptor chose when it refused. -#[derive(Debug)] +#[derive(Debug, Clone)] pub(crate) struct Refusal { pub error: Error, pub reason: String, @@ -1961,6 +1961,58 @@ struct Held { /// The token of the renewal being verified, until the serve loop applies its verdict. renewing: Option>, clock: crate::time::Deadlines, + /// Why the request ended, once it has. A channel, so every clone waiting on the grant + /// wakes when [`RequestGrant::end`] latches it. + ended: kio::Producer>, +} + +impl Held { + /// See [`RequestGrant::poll_ended`]. + fn poll_end(&mut self, waiter: &kio::Waiter) -> Poll { + loop { + // The acceptor granted a renewal the serve loop has yet to apply: it takes over rather + // than the request ending between the two, whether the old grant lapsed or dropped. + // Watched, so a request that abandons the renewal, such as one cancelled while a + // joined FETCH outlives it, falls back to the old grant's end. + let handover = self.renewing.as_ref().is_some_and(|issue| { + issue + .poll(waiter, |issue| match granted(issue) { + true => Poll::Pending, + false => Poll::Ready(()), + }) + .is_pending() + }); + if self.deadline.poll(waiter).is_ready() { + if handover { + return Poll::Pending; + } + return Poll::Ready(Refusal::new(Error::Expired, "request token expired")); + } + match self.verdict.poll_reply(waiter) { + Poll::Ready(None) if handover => return Poll::Pending, + Poll::Ready(None) => { + return Poll::Ready(Refusal::new( + Error::Session(SessionError::Unauthorized), + "grant dropped", + )); + } + Poll::Ready(Some(Reply::Grant(grant))) => { + if !self.kind.covers(&grant, self.path.as_str()) { + return Poll::Ready(Refusal::new(Error::Unauthorized, "request token no longer granted")); + } + self.deadline.set(grant.expires); + self.grant = grant; + // Re-poll: the new deadline may already have lapsed, or another reply may + // be waiting. + continue; + } + Poll::Ready(Some(Reply::Refuse { code, reason })) => { + return Poll::Ready(Refusal::new(Error::Session(code), &reason)); + } + Poll::Pending => return Poll::Pending, + } + } + } } impl RequestGrant { @@ -1990,6 +2042,7 @@ impl RequestGrant { hold, renewing: None, clock: clock.clone(), + ended: kio::Producer::new(None), })))) } @@ -2039,47 +2092,37 @@ impl RequestGrant { Ok(()) } + /// End the grant with `refusal`, waking every clone still watching it, unless it already + /// ended. A refused renewal ends the request without the grant ending itself, so the serve + /// loop calls this to end anything else serving under the grant, such as a joining FETCH + /// that outlives its subscription. + pub(crate) fn end(&self, refusal: Refusal) { + latch(&self.held().ended, refusal); + } + /// Resolve with the refusal that ends the request: the deadline lapsing /// ([`Error::Expired`]), the acceptor revoking or dropping the grant (its code and /// reason), or an acceptor-side update that no longer covers the request /// ([`Error::Unauthorized`]). A covering update, such as a lowered expiry, is folded in and /// polling continues. Never resolves while the grant still stands. pub(crate) fn poll_ended(&self, waiter: &kio::Waiter) -> Poll { - let mut held = self.held(); - loop { - // The acceptor granted a renewal the serve loop has yet to apply: it takes over rather - // than the request ending between the two, whether the old grant lapsed or dropped. - let handover = held.renewing.as_ref().is_some_and(granted); - if held.deadline.poll(waiter).is_ready() { - if handover { - return Poll::Pending; - } - return Poll::Ready(Refusal::new(Error::Expired, "request token expired")); - } - match held.verdict.poll_reply(waiter) { - Poll::Ready(None) if handover => return Poll::Pending, - Poll::Ready(None) => { - return Poll::Ready(Refusal::new( - Error::Session(SessionError::Unauthorized), - "grant dropped", - )); - } - Poll::Ready(Some(Reply::Grant(grant))) => { - if !held.kind.covers(&grant, held.path.as_str()) { - return Poll::Ready(Refusal::new(Error::Unauthorized, "request token no longer granted")); - } - held.deadline.set(grant.expires); - held.grant = grant; - // Re-poll: the new deadline may already have lapsed, or another reply may - // be waiting. - continue; - } - Poll::Ready(Some(Reply::Refuse { code, reason })) => { - return Poll::Ready(Refusal::new(Error::Session(code), &reason)); - } - Poll::Pending => return Poll::Pending, - } + let held = &mut *self.held(); + // Latched, as each acceptor reply is read once: every clone watching the grant sees + // the same end, whichever polled first. + let latched = match held.ended.poll_ref(waiter, |ended| match &**ended { + Some(ended) => Poll::Ready(ended.clone()), + None => Poll::Pending, + }) { + Poll::Ready(Ok(ended)) => Some(ended), + // Never closed: the grant owns its only producer. + Poll::Ready(Err(_)) | Poll::Pending => None, + }; + if let Some(ended) = latched { + return Poll::Ready(ended); } + let ended = std::task::ready!(held.poll_end(waiter)); + latch(&held.ended, ended.clone()); + Poll::Ready(ended) } /// Run `body`, everything a request does once admitted, until it finishes or `grant` @@ -2095,6 +2138,15 @@ impl RequestGrant { } } +/// Record `refusal` as why a request grant ended, unless an earlier end already is. +fn latch(ended: &kio::Producer>, refusal: Refusal) { + if let Ok(mut ended) = ended.write() + && ended.is_none() + { + *ended = Some(refusal); + } +} + /// Run `body` until it finishes, or until `ended` resolves first, which drops `body` wherever /// it waits. /// @@ -2124,10 +2176,10 @@ fn live(clock: &crate::time::Deadlines, kind: RequestKind, path: &str, grant: &G kind.covers(grant, path) && grant.expires.is_none_or(|expires| expires > clock.now()) } -/// Whether the acceptor granted the token behind `issue` and still holds the grant. -fn granted(issue: &kio::Shared) -> bool { - let issue = issue.read(); - !issue.done && issue.outbox.iter().any(|reply| matches!(reply, Reply::Grant(_))) +/// Whether the acceptor granted the token behind `issue` and still holds the grant, for a +/// request still waiting to apply it. +fn granted(issue: &Issue) -> bool { + issue.peer.is_none() && !issue.done && issue.outbox.iter().any(|reply| matches!(reply, Reply::Grant(_))) } /// A grant issued to one of the peer's tokens, which the session enforces. Dropping it diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index e1e2dcd18b6..5fa5b1099c0 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -649,6 +649,10 @@ enum Served { /// (draft-18 section 10.9.1). The update was already answered, and PUBLISH_DONE says /// UPDATE_FAILED. UpdateFailed(Error), + /// A renewal was refused, ending the request's grant. Unlike [`Self::UpdateFailed`] the + /// update is not answered yet: its REQUEST_ERROR is written once serving stops, outside + /// the grant's guard, which the ended grant would otherwise trip first. + RenewalRefused(crate::auth::Refusal), /// FORWARD=0 paused the subscription, which stays established. True when an update set /// it and is not answered yet, false when a SUBSCRIBE started paused. Paused(bool), @@ -692,7 +696,7 @@ impl Served { (ietf::PublishDoneStatus::Unauthorized, "token expired") } Self::Track(Err(_)) => (ietf::PublishDoneStatus::InternalError, "internal error"), - Self::UpdateFailed(_) => (ietf::PublishDoneStatus::UpdateFailed, "update failed"), + Self::UpdateFailed(_) | Self::RenewalRefused(_) => (ietf::PublishDoneStatus::UpdateFailed, "update failed"), // Pausing ends no subscription, so nothing reports it. Self::Paused(_) => (ietf::PublishDoneStatus::InternalError, "paused"), } @@ -816,6 +820,10 @@ impl Drop for Serve { /// The snapshot a joining FETCH inherits from its subscription. #[derive(Clone)] +#[allow( + clippy::large_enum_variant, + reason = "one per live subscription, and the small variants refuse at once; an allow, since the size gap varies by feature set" +)] enum Joined { /// This subscription's filter is not supported by the joining FETCH handler. Unsupported, @@ -827,9 +835,9 @@ enum Joined { cache: track::Consumer, /// The subscription's namespace, which the fetch is held to the grant on. namespace: crate::Path, - /// The subscription's own request token authorized it, so the limit does not bound - /// the fetch either. - token: bool, + /// The grant the subscription's own request token earned, if one authorized it. The + /// fetch is held to it too, past the subscription's end, and the limit bounds neither. + grant: Option, /// The subscription's ranking, which the fetch joins. ranking: Ranking, }, @@ -1392,7 +1400,7 @@ impl Publisher { // again on every resume. let filter = msg.filter; let namespace = msg.track_namespace.clone(); - let token = request_grant.is_some(); + let grant = request_grant.clone(); let save_join = |largest: Option, cache: &crate::track::Consumer, ranking: &Ranking>| { @@ -1405,7 +1413,7 @@ impl Publisher { }, cache: cache.clone(), namespace: namespace.clone(), - token, + grant: grant.clone(), ranking: ranking.clone(), }, (Filter::NextObject, None) => Joined::Empty, @@ -1562,12 +1570,21 @@ impl Publisher { }) .await }; - // A FETCH joining this subscription ends with it, before the peer reads a word. + // A FETCH joining this subscription is refused if it has not answered yet. drop(_join); // The subscription's demand is gone; only its credit may still wait on a verification. drop((request_grant, _slot)); let (served, streams) = match subscribed { + Ok(Ok(Subscribed::Served(Served::RenewalRefused(refusal), streams))) => { + let res = self + .write_subscribe_error(&mut stream.writer, &refusal.error, &refusal.reason) + .await; + match res { + Ok(()) => (Served::UpdateFailed(refusal.error), streams), + Err(err) => (Served::Track(Err(err)), streams), + } + } Ok(Ok(Subscribed::Served(served, streams))) => (served, streams), Ok(Ok(Subscribed::Refused(reason, err))) => { return self.reject_subscribe(stream, &err, &reason).await; @@ -1680,6 +1697,7 @@ impl Publisher { let res = match served { Served::Track(res) => res, Served::UpdateFailed(err) => Err(err), + Served::RenewalRefused(refusal) => Err(refusal.error), Served::Paused(_) => Ok(()), }; @@ -1837,14 +1855,10 @@ impl Publisher { }; // Per draft-18 section 10.9.1, a refused or uncovered // renewal ends only this subscription: REQUEST_ERROR, then PUBLISH_DONE - // UPDATE_FAILED. The session stays up and the old grant does not survive. - if let Err(err) = self - .write_subscribe_error(&mut stream.writer, &refusal.error, &refusal.reason) - .await - { - return Some(Served::Track(Err(err))); - } - return Some(Served::UpdateFailed(refusal.error)); + // UPDATE_FAILED. The session stays up and the old grant does not survive, + // so a FETCH joined to it stops at once rather than finishing under it. + rg.end(refusal.clone()); + return Some(Served::RenewalRefused(refusal)); } // An update behind a pending verdict waits its turn, so its answer follows the // verdict's. @@ -2745,10 +2759,12 @@ impl Publisher { // A standalone one is also checked here, before it reaches the origin; a joining one // was checked when its subscription was. // - // One joining a subscription its own request token authorized holds no authority of its - // own, so it ends with that subscription: on the token's expiry, revocation, or a - // refused renewal. + // A joining one is refused if its subscription ends before FETCH_OK. After that the + // head is already held and the requester can still cancel, so it finishes on its own + // authority: the gate, and the token grant if one authorized the subscription, which + // ends it on the token's expiry, revocation, or a refused renewal. let mut lapses_with = None; + let mut token_grant = None; let (track, start, end, joined, ranking, mut gate) = match fetch_type { FetchType::Standalone { namespace, @@ -2813,19 +2829,20 @@ impl Publisher { return Err(Error::Unauthorized); }; debug_assert_eq!(id, subscriber_request_id); - let (end, cache, namespace, ranking, token) = match self + let (end, cache, namespace, ranking, grant) = match self .joined(&mut stream, &mut finished, &mut messages, subscriber_request_id) .await? { Ok(joined) => joined, Err((err, reason)) => return self.reject_fetch(stream, &err, reason).await, }; - // The cache outlives the subscription, so the fetch holds its own gate and ends - // with the subscription rather than trusting the cache. One the subscription's - // request token authorized is held to the peer's grant alone, as the subscription is. + // The cache outlives the subscription, so the fetch holds its own gate rather than + // trusting the cache. One the subscription's request token authorized is held to + // the peer's grant alone, as the subscription is. let gate = crate::auth::Gate::new(auth, namespace, crate::auth::Direction::Publish); - let gate = if token { gate.named() } else { gate }; + let gate = if grant.is_some() { gate.named() } else { gate }; lapses_with = Some(subscriber_request_id); + token_grant = grant; let start = match fetch_type { FetchType::RelativeJoining { group_offset: 0, .. } => end.group, FetchType::RelativeJoining { .. } => { @@ -2863,7 +2880,7 @@ impl Publisher { }; let joins = self.joins.clone(); - let lapsed = |waiter: &kio::Waiter| { + let unsubscribed = |waiter: &kio::Waiter| { lapses_with.is_some_and(|id| { joins .poll(waiter, |joins| match joins.contains_key(&id) { @@ -2873,6 +2890,11 @@ impl Publisher { .is_ready() }) }; + let expired = |waiter: &kio::Waiter| { + token_grant + .as_ref() + .is_some_and(|grant: &crate::auth::RequestGrant| grant.poll_ended(waiter).is_ready()) + }; // Read the group before FETCH_OK promises it, while a refusal is still possible. let compression = self.peer_setup.get().await.flate; @@ -2891,7 +2913,7 @@ impl Publisher { { return Poll::Ready(None); } - if gate.poll_denied(waiter).is_ready() || lapsed(waiter) { + if gate.poll_denied(waiter).is_ready() || unsubscribed(waiter) || expired(waiter) { let err = Error::Unauthorized; let reason = err.to_string(); return Poll::Ready(Some(Err((err, reason)))); @@ -2907,9 +2929,9 @@ impl Publisher { }; let end = range.end; let group = group.without_status(); - // FETCH keeps each object's Timestamp, in the units FETCH_OK declares wherever the - // draft can declare them. Opting out of the properties strips nothing from the objects. - let timescale = group.info.timescale; + // Only the TIMESCALE in FETCH_OK opts the answer into timestamps, so a requester that + // opted out of the properties gets an untimed track. + let timescale = group.info.timescale.filter(|_| msg.properties_wanted); let frames = group .frames .iter() @@ -2934,7 +2956,7 @@ impl Publisher { // group has to be here in full or the two leave a gap. if group.sequence != end.group || group.end() != end.object { return self - .reject_fetch(stream, &Error::Evicted, "joining prefix unavailable") + .reject_fetch(stream, &Error::NotFound, "joining prefix unavailable") .await; } end @@ -2978,7 +3000,7 @@ impl Publisher { let res = { let mut ok = std::pin::pin!(ok); kio::wait(|waiter| { - if gate.poll_denied(waiter).is_ready() || lapsed(waiter) { + if gate.poll_denied(waiter).is_ready() || unsubscribed(waiter) || expired(waiter) { return Poll::Ready(Err(Error::Unauthorized)); } waiter.poll_future(ok.as_mut()) @@ -2991,6 +3013,7 @@ impl Publisher { } // Cancellation releases the buffered answer even while its stream is flow controlled. + // A joining one no longer ends with its subscription. let respond = async { let mut rank = ranking.rank(range.start); let uni = self @@ -3018,7 +3041,7 @@ impl Publisher { { return Poll::Ready(None); } - if gate.poll_denied(waiter).is_ready() || lapsed(waiter) { + if gate.poll_denied(waiter).is_ready() || expired(waiter) { return Poll::Ready(Some(Err(Error::Unauthorized))); } waiter.poll_future(respond.as_mut()).map(Some) @@ -3059,7 +3082,7 @@ impl Publisher { track::Consumer, crate::Path, Ranking>, - bool, + Option, ), (Error, &'static str), >, @@ -3102,8 +3125,8 @@ impl Publisher { cache, namespace, ranking, - token, - })) => return Ok(Ok((end, cache, namespace, ranking, token))), + grant, + })) => return Ok(Ok((end, cache, namespace, ranking, grant))), Ok(None) => ( match self.version { Version::Draft18 => Error::InvalidJoiningRequestId, @@ -10150,15 +10173,11 @@ mod serve_tests { assert!(h.log.resets().is_empty()); } - /// A joining FETCH reaching back past an evicted prefix group is refused, even when an - /// earlier group it holds has as many objects as the prefix promised. + /// A joining FETCH whose prefix group was evicted is refused as missing before FETCH_OK. #[moq_clock::test] async fn a_joining_fetch_refuses_a_prefix_group_it_lost() { let version = Version::Draft18; let h = serve(version); - let mut earlier = h.track.create_group(group::Info { sequence: 3 }).unwrap(); - earlier.write_frame(timestamp(), b"frame".as_slice()).unwrap(); - earlier.finish().unwrap(); let mut prefix = h.track.create_group(group::Info { sequence: 5 }).unwrap(); prefix.write_frame(timestamp(), b"frame".as_slice()).unwrap(); settle().await; @@ -10187,7 +10206,7 @@ mod serve_tests { group_order: GroupOrder::Ascending, fetch_type: FetchType::RelativeJoining { subscriber_request_id: RequestId(REQUEST_ID), - group_offset: 2, + group_offset: 0, }, range_filters: false, fill_timeout: false, @@ -10202,6 +10221,15 @@ mod serve_tests { crate::coding::decode_varint(&mut response, version).unwrap(), ietf::RequestError::ID ); + assert_eq!( + crate::coding::decode_buf(&mut response, version, |r, v| Ok( + ietf::RequestError::decode(r, v)?.error_code + )) + .unwrap(), + DOES_NOT_EXIST + ); + assert!(response.is_empty()); + assert!(h.log.resets().is_empty()); } /// A joining FETCH that arrives after its subscription ended has no live edge to name, so @@ -10513,8 +10541,8 @@ mod serve_tests { } /// FETCH_OK carries the properties SUBSCRIBE_OK would, as far as each draft has room for - /// them, and INCLUDE_PROPERTIES=0 empties the block. Either way the objects keep their - /// Timestamps wherever the draft can declare their units. + /// them, and INCLUDE_PROPERTIES=0 empties the block. The objects carry Timestamps only + /// when that block declares their units. #[moq_clock::test] async fn fetch_ok_carries_the_track_properties() { let info = track::Info { @@ -10557,8 +10585,8 @@ mod serve_tests { .unwrap(); assert_eq!( occurrences(&h.log, &properties), - 1, - "{version} wanted={wanted}: object unstamped" + usize::from(wanted), + "{version} wanted={wanted}: stamped only with a TIMESCALE" ); } } @@ -11475,19 +11503,41 @@ mod serve_tests { } } - /// A joining FETCH holds no authority of its own: revocation or dropping its parent - /// ends it even while stream credit blocks the response after FETCH_OK. + /// How a joining FETCH's subscription ends while the answer, past FETCH_OK, waits on + /// stream credit. + #[derive(Clone, Copy, Debug, PartialEq)] + enum JoinEnd { + /// The subscription ends, and the session grant still covers the fetch. + Ended, + /// A token-authorized subscription ends, and its token is still granted. + TokenEnded, + /// The token's grant stops covering the track while its subscription still serves. + Narrowed, + /// The token is revoked after its subscription ended. + RevokedAfter, + } + + /// Past FETCH_OK, a joining FETCH finishes its head on its own authority: the subscription + /// ending leaves it to finish, but the token that authorized the subscription still ends + /// it, before or after the subscription has. #[moq_clock::test] - async fn a_joining_fetch_ends_with_its_subscription() { - for (version, token) in JOINING_DRAFTS - .into_iter() - .flat_map(|version| [false, true].map(|token| (version, token))) - { + async fn a_joining_fetch_outlives_its_subscription_but_not_its_token() { + const PAYLOAD: &[u8] = b"joined-prefix"; + for (version, end) in JOINING_DRAFTS.into_iter().flat_map(|version| { + [ + JoinEnd::Ended, + JoinEnd::TokenEnded, + JoinEnd::Narrowed, + JoinEnd::RevokedAfter, + ] + .map(|end| (version, end)) + }) { + let token = end != JoinEnd::Ended; let origin = crate::origin::Config::new(crate::Hop::new(1).unwrap()).produce(); let broadcast = origin.publish("room", crate::origin::Route::default()).unwrap(); let track = broadcast.create_track("video", crate::track::Info::timed()).unwrap(); let mut group = track.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(timestamp(), b"joined-prefix".to_vec()).unwrap(); + group.write_frame(timestamp(), PAYLOAD.to_vec()).unwrap(); settle().await; let bi = kio::Producer::new(true); @@ -11521,7 +11571,7 @@ mod serve_tests { msg.authorization_token = None; } let mut subscription = Box::pin(publisher.clone().run_subscribe_stream(stream, msg)); - let enforced = if token { + let mut enforced = if token { let accepted = async { requests.next().await.expect("a request") }; let request = match futures::future::select(std::pin::pin!(accepted), subscription.as_mut()).await { futures::future::Either::Left((request, _)) => request, @@ -11576,22 +11626,205 @@ mod serve_tests { "{version}: FETCH_OK never went out" ); - if let Some(enforced) = enforced { - enforced.revoke(crate::SessionError::Unauthorized, "revoked"); - let _ = moq_clock::timeout(std::time::Duration::from_secs(1), subscription.as_mut()) - .await - .unwrap_or_else(|_| panic!("{version}: the revoked subscription kept serving")); + let mut subscription = Some(subscription); + if end == JoinEnd::Narrowed { + enforced + .as_ref() + .expect("a token") + .update(crate::auth::Grant::default()); + // The subscription reads the update first, and the fetch still sees it. The + // subscription's request then waits on the fetch, which holds its token's slot. + let serving = subscription.as_mut().expect("still serving"); + assert!(futures::poll!(serving.as_mut()).is_pending(), "{version}"); } else { - // Session authorization stays valid, and the cache stays alive. - drop(subscription); + subscription = None; + for _ in 0..10 { + assert!( + futures::poll!(fetch.as_mut()).is_pending(), + "{version} {end:?}: the fetch ended with its subscription" + ); + moq_clock::sim::yield_now().await; + } + if end == JoinEnd::RevokedAfter { + let enforced = enforced.take().expect("a token"); + enforced.revoke(crate::SessionError::Unauthorized, "revoked"); + } } + let Ok(mut open) = credit.write() else { + panic!("credit gate closed"); + }; + *open = true; + drop(open); let res = moq_clock::timeout(std::time::Duration::from_secs(1), fetch) .await - .unwrap_or_else(|_| panic!("{version}: the fetch outlived its subscription")); - assert!(matches!(res, Err(Error::Unauthorized)), "{version}: {res:?}"); - drop(credit); + .unwrap_or_else(|_| panic!("{version} {end:?}: the fetch never ended")); + let finished = matches!(end, JoinEnd::Ended | JoinEnd::TokenEnded); + match finished { + true => assert!(res.is_ok(), "{version} {end:?}: {res:?}"), + false => assert!(matches!(res, Err(Error::Unauthorized)), "{version} {end:?}: {res:?}"), + } + if let Some(subscription) = subscription { + let _ = moq_clock::timeout(std::time::Duration::from_secs(1), subscription) + .await + .unwrap_or_else(|_| panic!("{version}: the revoked subscription kept serving")); + } + assert_eq!( + occurrences(&session.log, PAYLOAD), + usize::from(finished), + "{version} {end:?}: the head on the wire" + ); + drop(enforced); + } + } + + /// A token subscription on draft 18 whose renewal the acceptor is still verifying, and a + /// FETCH joined to it that is past FETCH_OK and parked on stream credit, on its own task + /// so only a wake of its own resumes it. Its first grant lapses in 60s. + struct AnsweredJoin { + h: Serve, + serving: std::pin::Pin>>>, + renewal: crate::auth::Request, + fetch: moq_clock::sim::JoinHandle>, + /// The gate holding the fetch's answer back, shut. + _credit: kio::Producer, + _first: crate::auth::Issued, + _cred: crate::auth::Credential, + } + + const ANSWERED_HEAD: &[u8] = b"joined-head"; + + async fn answered_join() -> AnsweredJoin { + const VERSION: Version = Version::Draft18; + + let (auth, mut requests, _cred) = auth_covering_other(); + let credit = kio::Producer::new(false); + let session = ScriptedSession::per_stream(vec![subscribe_update_with_token(VERSION, true).await]) + .with_open_uni_gate(credit.consume()); + let h = serve_on(VERSION, auth, session); + let rt = h.publisher.runtime.clone(); + + let mut group = h.track.create_group(group::Info { sequence: 0 }).unwrap(); + group.write_frame(timestamp(), ANSWERED_HEAD.to_vec()).unwrap(); + group.finish().unwrap(); + settle().await; + + let stream = Stream::open(&mut crate::driver::Transport::new(h.session.clone()), VERSION) + .await + .unwrap(); + let mut serving: std::pin::Pin>>> = + Box::pin(h.publisher.clone().run_subscribe_stream(stream, token_subscribe())); + let first = async { requests.next().await.expect("a request") }; + let first = match futures::future::select(std::pin::pin!(first), serving.as_mut()).await { + futures::future::Either::Left((request, _)) => request, + futures::future::Either::Right((res, _)) => panic!("subscription ended: {res:?}"), + }; + let first = first.accept(grant_all_expiring(&rt, Some(60))); + let renewal = async { requests.next().await.expect("a renewal") }; + let renewal = match futures::future::select(std::pin::pin!(renewal), serving.as_mut()).await { + futures::future::Either::Left((request, _)) => request, + futures::future::Either::Right((res, _)) => panic!("subscription ended: {res:?}"), + }; + let joined = h.publisher.joins.wait(|joins| match joins.get(&RequestId(REQUEST_ID)) { + Some(Some(_)) => Poll::Ready(()), + _ => Poll::Pending, + }); + match futures::future::select(std::pin::pin!(joined), serving.as_mut()).await { + futures::future::Either::Left(_) => {} + futures::future::Either::Right((res, _)) => panic!("subscription ended: {res:?}"), } + + let stream = Stream::open(&mut crate::driver::Transport::new(h.session.clone()), VERSION) + .await + .unwrap(); + let mark = h.log.writes.lock().unwrap().len(); + let fetch = moq_clock::sim::spawn(h.publisher.clone().run_fetch_stream( + stream, + ietf::Fetch { + request_id: FETCH_ID, + subscriber_priority: 128, + group_order: GroupOrder::Ascending, + fetch_type: FetchType::RelativeJoining { + subscriber_request_id: RequestId(REQUEST_ID), + group_offset: 0, + }, + range_filters: false, + fill_timeout: false, + properties_wanted: true, + epoch: None, + }, + )); + for _ in 0..100 { + if h.log.writes.lock().unwrap().len() > mark { + break; + } + moq_clock::sim::yield_now().await; + } + assert!(h.log.writes.lock().unwrap().len() > mark, "FETCH_OK never went out"); + settle().await; + assert!(!fetch.is_finished(), "never waited on credit"); + + AnsweredJoin { + h, + serving, + renewal, + fetch, + _credit: credit, + _first: first, + _cred, + } + } + + /// A refused renewal ends the token's authority, not only the subscription: a joining + /// FETCH already past FETCH_OK stops serving under the refused token rather than finishing + /// on the old grant, while the subscription still ends with UPDATE_FAILED. + #[moq_clock::test] + async fn a_refused_renewal_ends_an_answered_joining_fetch() { + let join = answered_join().await; + + // Credit stays shut: the refusal alone must wake the fetch parked on it. The + // subscription's request then waits on the fetch, which holds its token's slot. + join.renewal.reject(crate::SessionError::Unauthorized, "no"); + let served = moq_clock::timeout(Duration::from_secs(1), join.serving) + .await + .expect("the refused subscription kept serving"); + assert!( + matches!(served, Err(Error::Session(crate::SessionError::Unauthorized))), + "{served:?}" + ); + let (status, _) = Wire::read(&join.h.log, Version::Draft18) + .publish_done + .expect("PUBLISH_DONE was sent"); + assert_eq!(status, ietf::PublishDoneStatus::UpdateFailed.code(), "UPDATE_FAILED"); + let res = moq_clock::timeout(Duration::from_secs(1), join.fetch) + .await + .expect("the fetch never ended") + .expect("the fetch panicked"); + assert!( + matches!(res, Err(Error::Unauthorized)), + "kept serving under the refused token: {res:?}" + ); + assert_eq!(occurrences(&join.h.log, ANSWERED_HEAD), 0, "the head on the wire"); + } + + /// A renewal the subscription abandons, granted but never applied because the + /// subscription ended first, does not keep the old grant alive for a joined FETCH that + /// outlives it: the FETCH still ends when the old grant lapses. + #[moq_clock::test] + async fn an_abandoned_renewal_leaves_an_answered_joining_fetch_the_old_expiry() { + let join = answered_join().await; + let rt = join.h.publisher.runtime.clone(); + + // Granted, and the grant kept, but the subscription ends before it reads the verdict. + let _renewed = join.renewal.accept(grant_all_expiring(&rt, None)); + drop(join.serving); + + let res = moq_clock::timeout(Duration::from_secs(61), join.fetch) + .await + .expect("the fetch outlived its token") + .expect("the fetch panicked"); + assert!(matches!(res, Err(Error::Unauthorized)), "{res:?}"); + assert_eq!(occurrences(&join.h.log, ANSWERED_HEAD), 0, "the head on the wire"); } /// Where a token subscription is parked when its grant ends. diff --git a/rs/moq-net/src/ietf/publisher/tracks.rs b/rs/moq-net/src/ietf/publisher/tracks.rs index 16812739861..416d5fe5331 100644 --- a/rs/moq-net/src/ietf/publisher/tracks.rs +++ b/rs/moq-net/src/ietf/publisher/tracks.rs @@ -1248,7 +1248,7 @@ impl Publisher { }, cache: track.clone(), namespace: namespace.clone(), - token: false, + grant: None, ranking: serve.ranking.clone(), }, (Filter::NextObject, None) => Joined::Empty, diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index f70b2ec62b2..8455f53e96f 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -5453,8 +5453,9 @@ impl Subscriber { range_filters: false, fill_timeout: false, // The copy learned the track before any fetch, from SUBSCRIBE_OK or - // TRACK_STATUS_OK, so a FETCH_OK repeating it is waste. - properties_wanted: false, + // TRACK_STATUS_OK, so a FETCH_OK repeating it is waste, except that only its + // TIMESCALE opts the answer into the timestamps a timed copy needs. + properties_wanted: timescale.is_some(), epoch: self.epoch.clone(), }, ); diff --git a/rs/moq-net/src/lite/test_transport.rs b/rs/moq-net/src/lite/test_transport.rs index a17bf539c68..24339382557 100644 --- a/rs/moq-net/src/lite/test_transport.rs +++ b/rs/moq-net/src/lite/test_transport.rs @@ -936,6 +936,8 @@ pub struct ScriptedSession { /// Set by [`Self::gated_open`]: parks `poll_open_bi` until the gate opens, standing in /// for a transport header blocked on send capacity. open_gate: Option>, + /// Set by [`Self::with_open_uni_gate`]. + uni_open_gate: Option>, credit: kio::Producer, /// Scripts the peer pushes at us on unidirectional streams, popped by `accept_uni` /// in order. See [`Self::with_incoming_unis`]. @@ -980,6 +982,7 @@ impl ScriptedSession { script: Arc::new(Mutex::new(script)), queue: None, open_gate: None, + uni_open_gate: None, credit: kio::Producer::new(true), incoming_unis: Arc::new(Mutex::new(std::collections::VecDeque::new())), incoming_bidis: Arc::new(Mutex::new(std::collections::VecDeque::new())), @@ -1004,6 +1007,12 @@ impl ScriptedSession { self } + /// Hold unidirectional stream opens until `gate` grants stream credit. + pub fn with_open_uni_gate(mut self, gate: kio::Consumer) -> Self { + self.uni_open_gate = Some(gate); + self + } + /// Never acknowledge an opened stream's FIN, like a peer that stopped reading. pub fn with_unacked_fin(mut self) -> Self { self.unacked_fin = true; @@ -1188,9 +1197,15 @@ impl poll::Session for ScriptedSession { fn poll_open_uni( &self, - _waiter: &kio::Waiter, + waiter: &kio::Waiter, options: poll::SendOptions<'_, SinkGroup>, ) -> Poll> { + if let Some(gate) = &self.uni_open_gate { + match gate.poll(waiter, |open| (**open).then_some(()).map_or(Poll::Pending, Poll::Ready)) { + Poll::Ready(Ok(())) => {} + Poll::Ready(Err(_)) | Poll::Pending => return Poll::Pending, + } + } let mut send = SinkSend::new(self.log.clone()); send.unacked_fin = self .unacked_unis