diff --git a/quest/m0/test-flakes-3/interop-finish-unroutable.md b/quest/m0/test-flakes-3/interop-finish-unroutable.md deleted file mode 100644 index b2a0fce22e..0000000000 --- a/quest/m0/test-flakes-3/interop-finish-unroutable.md +++ /dev/null @@ -1,47 +0,0 @@ -# [M] A request right after its announcement is not refused unroutable - -## Goal - -`just test interop --all` passes its `finish` lane under load. A relay never -answers UNROUTABLE to a request for a claim it announced milliseconds -earlier, and a deterministic test (a kio simulation or a relay integration -test) fails without the fix. - -## Plan - -Seen 2026-11-17 on -[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987): 3 of about 30 -runs of the lane, all while the machine was loaded by other builds, and none -in 19 runs once the load fell or under 8 busy loops. Whether `main` before -that PR does the same was not checked. - -The lane's publisher ends its broadcast when the linger runs out and -announces `.health` on the same lite-07 session. Every reader -waits for the first to go and subscribes to `live` on the second the moment -it is announced. In each failing run the relay logged, in order: its session -with the publisher announcing the health route -(`lite::subscriber::routed: announce`), then about 5 ms later -`subscribed error ... err=unroutable` for the first three or four readers, -then about 8 ms later `source closed` for the ended broadcast on the -publisher's session, then the upstream subscribe starting for the readers -that asked after that, which all pass. The readers refused never retry, so -they pass their deadline. - -Ruled out with tracing: `route::node::State::forward` returned no refusal in -a failing run, so the node found the claim's slot serving and the refusal -came from the origin beneath it. The sites left are the ones answering -`Error::Unroutable` in `rs/moq-net/src/model/origin.rs` (`Requesting::poll_ok` -when its lookup fails while discovery is synced, the lookup's own -`best_route` under the node filter, and the pinned instance check) and the -validity checks in `rs/moq-net/src/model/broadcast.rs`. Tracing them did not -catch a failure, since it stopped reproducing. - -Do not fix it in the test with a retry: a viewer that subscribes as a -broadcast appears would see the same refusal. - -Public API: none. Wire: none expected. - -## Related - -- [The lite-07 route hang](/quest/m0/nightly/lite07-route-hang.md) - another timing fault on the lite-07 route path, with a bench that reproduces -- [What the route switch left](/quest/m2/multicast/switch.md) - the PR that saw it diff --git a/quest/m0/test-flakes-3/py-finish-publisher-ack.md b/quest/m0/test-flakes-3/py-finish-publisher-ack.md new file mode 100644 index 0000000000..cc041e252c --- /dev/null +++ b/quest/m0/test-flakes-3/py-finish-publisher-ack.md @@ -0,0 +1,20 @@ +# [S] The python finish publisher acknowledges every run + +## Goal + +`just test interop --all --finish` passes its `python -> *` row: the python +publisher prints `finish acknowledged` once the harness writes `complete`. + +## Plan + +Seen once on 2026-11-17 while validating +[moq-dev/moq#9052](https://github.com/moq-dev/moq/pull/9052): every reader of the +python publisher passed (`health=ok`), the relay refused nothing, yet the +publisher (`test/interop/clients/python/interop.py`, the `finish_ack` branch) +printed nothing after `finish ready linger_us=` and its session stayed open +until the harness reaped it about two minutes later. A rerun of the row passed. +So it hung after announcing `.health`: in `acknowledge()` reading +the ack FIFO, or in `client.session.state()`. Trace which, under load, and fix +that cause rather than the harness timeout. + +Public API: none. Wire: none. diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index ac587c4fde..51dd6d76a5 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -184,7 +184,8 @@ impl Shared { /// the node `node` that claims it: this node's own when it serves the claim itself, /// else the claim it follows to `node`. `None` when nothing reaches that node or the /// way would lead back to the peer. A request naming no node resolves through any - /// route. Waits while the session the request would leave on has yet to serve the claim. + /// route. Waits while the session the request would leave on has yet to serve the claim, + /// whether or not the request names a node. fn poll_node_filter( &self, origin: origin::Consumer, @@ -193,14 +194,24 @@ impl Shared { node: Option, waiter: &kio::Waiter, ) -> Poll> { + let claim = origin.resolve(broadcast).zip(instance.cloned()); let Some(node) = node else { + // The node shows a peer's claim to the sessions it reaches before the session + // serving it lands its route in the origin, each on its own task. A peer that + // asks the moment it hears must find that route, not an origin without one. + if let Some(claim) = claim { + let _ = ready!( + origin + .route_node() + .poll_forward(self.peer_setup.tag(), None, &claim, waiter) + ); + } return Poll::Ready(Some(origin)); }; // A node is named for the claim of one instance. - let (Some(path), Some(instance)) = (origin.resolve(broadcast), instance) else { + let Some(claim) = claim else { return Poll::Ready(None); }; - let claim = (path, instance.clone()); let via = ready!( origin .route_node() diff --git a/rs/moq-net/src/lite/subscriber/routed/tests.rs b/rs/moq-net/src/lite/subscriber/routed/tests.rs index dff0577a51..30af3e429e 100644 --- a/rs/moq-net/src/lite/subscriber/routed/tests.rs +++ b/rs/moq-net/src/lite/subscriber/routed/tests.rs @@ -512,3 +512,71 @@ async fn an_unresolved_claim_outlives_another_prefixs_removal() { let ended = routed.receive(Wire::End { id: 2, lost: false }); assert!(matches!(ended, Err(Error::ProtocolViolation)), "{ended:?}"); } + +/// The node shows a peer's claim to the sessions it reaches before the session serving it +/// lands its route in the origin, each on its own task. A reader asking for the claim the +/// moment it hears of it waits for that route, and is not refused for want of it. +#[kio::test] +async fn a_request_for_a_claim_shown_waits_for_its_route() { + use crate::model::ProduceTest; + const TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); + let version = crate::Version::lite(crate::lite::Version::Lite07); + let (subscriber, _consumer) = forwarding(); + let mut upstream = asking(&subscriber, &Permit::default()); + + // A reader on lite-07, served from the relay's origin. + let reader = crate::origin::Config::new(std::num::NonZeroU64::new(3).unwrap()).produce(); + let (client, server) = crate::transport::memory::pair(crate::transport::memory::Config { + protocol: Some(version.alpn().to_string()), + ..Default::default() + }); + let dial = crate::Client::new() + .with_versions(version.into()) + .with_subscriber(reader.clone()); + let serve = crate::Server::new() + .with_versions(version.into()) + .with_publisher(subscriber.origin.consume()); + let (_client, _server) = futures::join!( + async { + let (session, driver) = dial.connect(kio::clock::now(), client).await.unwrap(); + drop(kio::clock::sim::spawn(crate::time::run(driver))); + session + }, + async { + let (session, driver) = serve.accept(kio::clock::now(), server).await.unwrap(); + drop(kio::clock::sim::spawn(crate::time::run(driver))); + session + }, + ); + + // The claim reaches the node, which shows it to the reader, while the session it + // arrived on has yet to land its route. + upstream.receive(start("live")).unwrap(); + upstream.flush(); + let live = kio::clock::timeout(TIMEOUT, reader.consume().follow("live", None).unwrap().available()) + .await + .expect("the reader hears of the claim") + .unwrap() + .expect("the claim is shown"); + // It asks at once, and its request reaches the relay before the route does. + let _subscribing = kio::clock::sim::spawn(async move { live.track("video").unwrap().subscribe(None).await }); + kio::clock::sleep(std::time::Duration::from_millis(100)).await; + + // The route lands, and the request goes on to the session serving the claim. Refused + // instead, the reader would wait for a route change that never comes. + let stream = upstream.stream(); + while let Some(output) = stream.try_serve() { + upstream.output(&subscriber, output); + } + let minted = kio::clock::timeout( + TIMEOUT, + kio::wait(|waiter| { + upstream.poll(&subscriber, waiter); + subscriber.sources.poll_pop(waiter) + }), + ) + .await + .expect("the request reaches the claim") + .unwrap(); + assert_eq!(minted.path.as_str(), "live"); +}