diff --git a/nix/Cargo.nix b/nix/Cargo.nix index b4b3b5a4df..35bb6d16c9 100644 --- a/nix/Cargo.nix +++ b/nix/Cargo.nix @@ -21751,6 +21751,11 @@ rec { } ]; devDependencies = [ + { + name = "hang"; + packageId = "hang"; + features = [ "async" ]; + } { name = "kio"; packageId = "kio"; diff --git a/quest/m0/test-flakes-3/srt-server-load.md b/quest/m0/test-flakes-3/srt-server-load.md index aec489260f..6322c43ac1 100644 --- a/quest/m0/test-flakes-3/srt-server-load.md +++ b/quest/m0/test-flakes-3/srt-server-load.md @@ -1,16 +1,18 @@ -# [S] moq-srt's server tests hold under load +# [XS] Tests queued for an mbx permit time out under load ## Goal -`moq-srt`'s `server::tests` pass however loaded the machine is, and its test target compiles on its own. +No test hits nextest's timeout while it waits for an mbx scheduler permit, so `moq-srt`'s `server::tests` (and every other test) pass however loaded the host is. ## Plan -Four of them (`a_new_clock_flushes_the_partial_chunk_first`, `a_new_clock_re_anchors_the_pacing`, `a_publisher_marker_keeps_an_srt_receiver_playing`, `a_reorder_is_not_a_rewind`) hit nextest's 60s timeout under a full `just check` on 2026-11-18 at load average 40, on a branch that touched no moq-srt code ([moq-dev/moq#9123](https://github.com/moq-dev/moq/pull/9123)). -Run alone, all fifteen pass in 3.4s. -Reproduce under load first, then fix what waits on wall time. +Four `moq-srt` `server::tests` hit nextest's 60s timeout in a loaded `just check` ([moq-dev/moq#9123](https://github.com/moq-dev/moq/pull/9123)); three are purely synchronous, so the wait was never in the test code. +The cause is mbx, the Cargo wrapper: the host's global mbx config sets `scheduler.tests = true`, so each test waits for a host-wide permit inside `mbx-test-runner`, and nextest counts that wait toward the test's `slow-timeout`. +Any test in the workspace can hit it; moq-srt's were just in flight. -Run alone, the test target does not compile: `src/server.rs` calls `hang::Catalog::subscribe`, which needs hang's `async` feature, and only workspace feature unification enables it. -Enable it in moq-srt's dev-dependencies, as moq-json does for its own wrappers. +Decided 2026-11-19 (maintainer, [moq-dev/moq#9163](https://github.com/moq-dev/moq/pull/9163#issuecomment-7020242924)): fix it in mbx or the dotfiles mbx config, not with a checked-in `.mbx.toml` opt-out or a longer `terminate-after` here, so host-wide test scheduling stays on. +Once that lands, reproduce below to confirm, then delete this quest. -Decided 2026-11-19 (coordinator, maintainer away, merge granted): keep this follow-up. +## Waiting + +- mbx or the dotfiles mbx config: stop a test's queue wait for a scheduler permit from counting toward nextest's timeout, for example by taking the permit before nextest starts the test's clock. Check with the one-core reproduction from [moq-dev/moq#9163](https://github.com/moq-dev/moq/pull/9163): run `cargo nextest run -p moq-srt --lib --test-threads 4 --stress-count 20` with `KIO_DENY_HOST_TIMERS=1` inside `systemd-run --user --scope -p CPUQuota=100%`, with no `MBX_SCHEDULER_TESTS` override. Before the fix about 20 tests time out at 60s, each printing `mbx: waiting for machine-wide scheduler capacity`; after it, all pass. diff --git a/rs/moq-srt/Cargo.toml b/rs/moq-srt/Cargo.toml index 842a550ae8..733bd89d69 100644 --- a/rs/moq-srt/Cargo.toml +++ b/rs/moq-srt/Cargo.toml @@ -28,6 +28,8 @@ tokio = { workspace = true, features = ["full"] } tracing = { workspace = true } [dev-dependencies] +# `Catalog::subscribe` in the tests. +hang = { path = "../hang", features = ["async"] } kio = { workspace = true, features = ["runtime"] } srt-protocol = "0.4" tracing-subscriber = { workspace = true }