diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 6497b3f5b9a..37c1294631c 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -108,9 +108,14 @@ tests' shared locked-memory budget, one child each. The pool-debt test overshot because shed frames whose payload canonical content owned stayed billed until crossbeam-epoch reclamation, which other tests' reads in the same process held back; eviction now lets go of that content itself, fixed by -[moq-dev/moq#8052](https://github.com/moq-dev/moq/pull/8052). On 2026-11-06, moq-ffi's -concurrent-ready test, quarantined here instead of in the unrelated -[moq-dev/moq#7981](https://github.com/moq-dev/moq/pull/7981). +[moq-dev/moq#8052](https://github.com/moq-dev/moq/pull/8052). + +moq-ffi's concurrent-ready test drained its reader until the group ended, which +an unfinished group never does. Its `ffi_caught_up` returned without yielding +whenever the FFI runtime answered first, so under load the woken `ready` calls +missed their turn until every frame was read. Fixed by reading at most what was +written and yielding after each read, in +[moq-dev/moq#8053](https://github.com/moq-dev/moq/pull/8053). Added 2026-11-06: `js/net`'s flate tail-shed test, which timed out in a merge-queue batch, one child. @@ -129,6 +134,5 @@ Public API: none. Wire: none. - [dispatch-query's terminate test survives load](/quest/m0/test-flakes-2/dispatch-query-terminate.md) - `test_terminating_the_run_stops_the_running_phase` passes reliably under host load, re-enabled once fixed - [Test PulseAudio servers stay off the desktop's D-Bus](/quest/m0/test-flakes-2/pulse-dbus.md) - concurrent media runs on one desktop never collide on `org.PulseAudio1` - [The Python audio playout-delay test reads its frames](/quest/m0/test-flakes-2/audio-playout-delay.md) - `test_audio_consumer_reports_its_playout_delay` stops timing out on its first read in the merge queue -- [The FFI concurrent-ready test wakes both calls under load](/quest/m0/test-flakes-2/ffi-concurrent-ready.md) - `concurrent_ready_calls_all_wake` stops stalling its reader drain under load and is re-enabled - [The datagram-end close test survives load](/quest/m0/test-flakes-2/datagram-end-close.md) - `a session close right after a datagram track's end keeps the end` stops aborting the track under load and is re-enabled - [OBS TSan runs report no leaked FFI worker thread](/quest/m0/test-flakes-2/obs-tsan-thread-leak.md) - the OBS TSan tests stop flagging a finished `moq-ffi-worker` thread at exit under load diff --git a/quest/m0/test-flakes-2/ffi-concurrent-ready.md b/quest/m0/test-flakes-2/ffi-concurrent-ready.md deleted file mode 100644 index 814cb07794b..00000000000 --- a/quest/m0/test-flakes-2/ffi-concurrent-ready.md +++ /dev/null @@ -1,33 +0,0 @@ -# [S] The FFI concurrent-ready test wakes both calls under load - -## Goal - -`concurrent_ready_calls_all_wake` in `rs/moq-ffi/src/test.rs` passes -reliably in loaded runs, fixed at its cause under the -[epic](/quest/m0/test-flakes-2/README.md)'s rules, and its `#[ignore]` and -`sh/gh/test-inventory.allow` entry are removed. Done when the PR explains -the stall and shows the loaded run that reproduced it now passing. - -## Plan - -Added 2026-11-06 (maintainer's standing rule that flakes join this epic, -`/quest-complete`). The `/quest-iterate` agent on -[moq-dev/moq#7981](https://github.com/moq-dev/moq/pull/7981) saw it time out -in a local `quest gate just ci` while draining the reader during the two -concurrent `ready` calls; it passed on unchanged `main`, and #7981 doesn't -touch moq-ffi. Quarantined in -[moq-dev/moq#8024](https://github.com/moq-dev/moq/pull/8024) with every -assertion kept. - -Lead, unverified: the drain loop reads until both parked `ready` calls have -finished, but it checks them only between reads. If the reader takes the -last of the 25 frames before both calls finish (their wakes run on the FFI -runtime, which lags under load), the next `read_frame` waits on a group that -gets no more frames and never finishes, so it times out as "the read -stalled". Check whether that is the failure before choosing between waiting -on the calls once the window has room and fixing a lost wake in the group. - -Reproduce first: run the test repeatedly with `--run-ignored all` beside -`just check --all` load under `quest gate`, and keep the failing output. - -Public API: none. Wire: none. diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index 09467397c9f..1f2a37b96dd 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -9453,7 +9453,6 @@ async fn ready_after_abort_is_closed() { /// Two calls waiting on the same group both wake once the reader drains, though the group /// keeps only one writer waker. #[tokio::test] -#[ignore = "flaky under load: the reader can drain every frame before both ready calls finish; see quest/m0/test-flakes-2/ffi-concurrent-ready.md"] async fn concurrent_ready_calls_all_wake() { let broadcast = MoqBroadcastProducer::new().unwrap(); let track = broadcast.publish_track("tcp".into(), None).unwrap(); @@ -9482,9 +9481,8 @@ async fn concurrent_ready_calls_all_wake() { ]; assert!(waits.iter().all(|wait| !wait.is_finished()), "the window was not full"); - // Every read wakes both calls. Run them before the next read, and read no more than - // was written: draining the window faster than the runtime polls a woken call must - // not leave this loop waiting on a group nobody writes again. + // A read that makes room wakes both calls, which run when this task yields. Read no + // more than was written: the group never ends, so a read past its last frame waits forever. for _ in 0..24 { if waits.iter().all(|wait| wait.is_finished()) { break; @@ -9495,7 +9493,6 @@ async fn concurrent_ready_calls_all_wake() { .unwrap() .expect("the group ended early"); tokio::task::yield_now().await; - ffi_caught_up().await; } for wait in waits { kio::clock::timeout(TIMEOUT, wait) diff --git a/sh/gh/test-inventory.allow b/sh/gh/test-inventory.allow index 6c56be00373..c717f0fba8c 100644 --- a/sh/gh/test-inventory.allow +++ b/sh/gh/test-inventory.allow @@ -2,9 +2,6 @@ # one: ` `. sh/gh/test-inventory.py prints the name of any # orphan missing here, and fails on an entry that has a caller. -# Flaky under load; the quest makes it reliable, then removes its `#[ignore]`. -ignore moq-ffi::test::concurrent_ready_calls_all_wake /quest/m0/test-flakes-2/ffi-concurrent-ready.md - # Need a GPU, a desktop, or a device that no CI runner has. ignore moq-carla-bridge::gpu_video_reaches_the_worker /quest/m5/carla-bridge.md recipe carla::bridge-gpu /quest/m5/carla-bridge.md