diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 4bb385ce55d..e5f1d6741c1 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -101,6 +101,6 @@ Public API: none. Wire: none. - [reload.test.ts's close-drain tests pass under load](/quest/m0/test-flakes-2/reload-close-drain.md) - the drain cases in `js/net`'s reload tests pass in loaded multi-file runs - [dispatch-query's terminate test survives load](/quest/m0/test-flakes-2/dispatch-query-terminate.md) - `test_terminating_the_run_stops_the_running_phase` passes reliably under host load, re-enabled once fixed - [release.test.ts never reaches the network](/quest/m0/test-flakes-2/release-test-offline.md) - the JSR release test fakes its registry lookup and passes offline -- [moq-ffi's datagram tests don't hinge on one datagram](/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md) - the default-connection datagram tests pass under CI load +- [Diagnose and re-enable the FFI datagram connection test](/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md) - diagnose the default-connection timeout and re-enable the quarantined test - [Test PulseAudio servers stay off the desktop's D-Bus](/quest/m0/test-flakes-2/pulse-dbus.md) - concurrent media runs on one desktop never collide on `org.PulseAudio1` - [The open-GOP browser fixture fails loudly](/quest/m0/test-flakes-2/open-gop-fixture.md) - its broadcast stub matches current stats dependencies and an effect exception fails the test diff --git a/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md b/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md index 36f4ca9c5a8..8629ae1993e 100644 --- a/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md +++ b/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md @@ -1,21 +1,59 @@ -# [S] moq-ffi's datagram tests don't hinge on one datagram +# [S] Diagnose and re-enable the FFI datagram connection test ## Goal `datagrams_cross_a_default_connection` and its `requested_` sibling -(`rs/moq-ffi/src/test.rs`) pass reliably under CI load, without depending on -a single unreliable datagram arriving. +(`rs/moq-ffi/src/test.rs`) pass reliably under CI load after the timeout +cause is confirmed and fixed. ## Plan Decided 2026-11-05 (maintainer, follow-up from [moq-dev/moq#7914](https://github.com/moq-dev/moq/pull/7914)): ✅ m0. +Decided 2026-11-05 (maintainer, /quest-plan): ✅ Keep only +`datagrams_cross_a_default_connection` quarantined and land the quarantine in +[moq-dev/moq#7954](https://github.com/moq-dev/moq/pull/7954). Re-diagnose under +CI load after [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) +lands. Re-enabling now is rejected. Do not add repeated sending without +evidence of ordinary loss. + The test sends one datagram over a real connection and waits 10 s for it; -it timed out in Check on #7914 while passing 30 of 30 runs on an idle host. -Confirm the cause first. If it's loss, keep writing datagrams until the first -arrives and assert on what arrives, rather than waiting longer. If it's the -datagram answer race planned in -[moq-dev/moq#7929](https://github.com/moq-dev/moq/pull/7929) -(`quest/m1/datagram-tracks/answer-race.md`), say so here and wait on that -quest instead. +it timed out in Check on +[moq-dev/moq#7914](https://github.com/moq-dev/moq/pull/7914) while passing +30 of 30 runs on an idle host. Confirm the cause with send/route diagnostics +before choosing the fix and re-enabling the test. + +Investigation 2026-11-05 on `main` at `419b799c75`: + +- The unchanged three connection cases pass locally. With temporary diagnostic + prints at `DatagramSend::send` and `Subscriber::route_datagram`, + `quest gate just rs test -p moq-ffi --lib -E 'test(datagrams_cross)' + --stress-count 100 --no-capture` passes all 300 cases. Every send returns + `Ok(())`, and all 300 received datagrams find a route. The diagnostic + prints were removed. This does not establish the cause of the CI timeout. +- All three tests await `consumer.info()` before appending. In + `DatagramServe::poll_serve`, the answer decodes before `request.accept` + exposes that info; route registration follows synchronously on that driver + before its next datagram receive pass. The answer race in + [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) is real, + but was not reproduced by these tests. Do not claim it caused this timeout. +- `datagrams_cross_a_default_connection` is disabled pending diagnosis, per + the repository's flaky-test rule. Its requested siblings have not failed + in this investigation and remain enabled. No retry or longer wait was added. + +## Waiting + +- moq-dev/moq: wait for + [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) to merge, then + capture a failing run under CI load with send/route diagnostics. Check the + PR's merged state with `gh pr view 7951 --json state,mergedAt`; once merged, + run the ignored case locally under CI-like concurrent build/test load, + with diagnostics at `DatagramSend::send` and `Subscriber::route_datagram`, + retaining a failing trace. Explicitly include the quarantine with + `quest gate just rs test -p moq-ffi --lib + -E 'test(datagrams_cross_a_default_connection)' --run-ignored all --no-capture`; + normal Check and queue runs intentionally skip it. Use it to distinguish ordinary transport loss, a route miss, or + another lifecycle bug and choose the corresponding fix before re-enabling. + Passing stress runs alone do not clear this wait or establish transport loss. + No workflow dispatch or host change is authorized for this investigation. diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index 29f84a9d701..68ca499d3e0 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -6041,6 +6041,7 @@ async fn datagram_connection( /// A native datagram-only reader reaches a publisher on the default lite-06 path. #[tokio::test] +#[ignore = "flaky under CI load; tracked in quest/m0/test-flakes-2/ffi-datagram-roundtrip.md"] async fn datagrams_cross_a_default_connection() { let (broadcast, remote, session, _server) = datagram_connection("sensor", vec![]).await; let producer = broadcast diff --git a/sh/gh/test-inventory.allow b/sh/gh/test-inventory.allow index 6f44c275b3f..8562a330974 100644 --- a/sh/gh/test-inventory.allow +++ b/sh/gh/test-inventory.allow @@ -3,6 +3,7 @@ # orphan missing here, and fails on an entry that has a caller. # Flaky under load; the quest makes it reliable, then removes its `#[ignore]`. +ignore moq-ffi::test::datagrams_cross_a_default_connection /quest/m0/test-flakes-2/ffi-datagram-roundtrip.md ignore moq-gst::source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad /quest/m0/test-flakes-2/gst-reconnect-interleave.md # Need a GPU, a desktop, or a device that no CI runner has.