From 8067e269d0fc0a8e55d9b524552c55f9a254c944 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 19 Nov 2026 00:43:43 -0800 Subject: [PATCH 1/4] chore(quest): claim quest/m0/test-flakes-3/srt-server-load.md Quest-Claim: d209b48a6637cc40 From 2a1fac9af01c64cc72781d01b525123d93beff62 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 19 Nov 2026 01:36:55 -0800 Subject: [PATCH 2/4] fix(test): tests never queue for an mbx permit, and moq-srt's tests build alone nextest starts a test's timeout when it starts mbx's runner, so a test that waits for a host-wide scheduler permit is killed before it runs. Disable mbx test scheduling for this workspace; quest gate and nextest's thread count already bound test load. moq-srt's ts tests call `Catalog::subscribe`, which needs hang's `async` feature, so enable it in the dev-dependencies. Co-Authored-By: Claude Opus 5.5 --- .mbx.toml | 7 +++++++ nix/Cargo.nix | 5 +++++ quest/m0/test-flakes-3/srt-server-load.md | 16 ---------------- rs/moq-srt/Cargo.toml | 2 ++ 4 files changed, 14 insertions(+), 16 deletions(-) create mode 100644 .mbx.toml delete mode 100644 quest/m0/test-flakes-3/srt-server-load.md diff --git a/.mbx.toml b/.mbx.toml new file mode 100644 index 00000000000..553bc1dd01e --- /dev/null +++ b/.mbx.toml @@ -0,0 +1,7 @@ +# Workspace settings for mbx, the Cargo wrapper that caches our builds. + +[scheduler] +# Never queue a test for a scheduler permit. nextest starts a test's timeout when +# it starts mbx's runner, so a test held in the queue on a loaded host is killed +# before it runs. `quest gate` and nextest's own thread count bound test load. +tests = false diff --git a/nix/Cargo.nix b/nix/Cargo.nix index c65defdac34..df93cf66068 100644 --- a/nix/Cargo.nix +++ b/nix/Cargo.nix @@ -21616,6 +21616,11 @@ rec { } ]; devDependencies = [ + { + name = "hang"; + packageId = "hang"; + features = [ "async" ]; + } { name = "kio"; packageId = "kio"; diff --git a/quest/m0/test-flakes-3/srt-server-load.md b/quest/m0/test-flakes-3/srt-server-load.md deleted file mode 100644 index aec489260fb..00000000000 --- a/quest/m0/test-flakes-3/srt-server-load.md +++ /dev/null @@ -1,16 +0,0 @@ -# [S] moq-srt's server tests hold under load - -## Goal - -`moq-srt`'s `server::tests` pass however loaded the machine is, and its test target compiles on its own. - -## Plan - -Four of them (`a_new_clock_flushes_the_partial_chunk_first`, `a_new_clock_re_anchors_the_pacing`, `a_publisher_marker_keeps_an_srt_receiver_playing`, `a_reorder_is_not_a_rewind`) hit nextest's 60s timeout under a full `just check` on 2026-11-18 at load average 40, on a branch that touched no moq-srt code ([moq-dev/moq#9123](https://github.com/moq-dev/moq/pull/9123)). -Run alone, all fifteen pass in 3.4s. -Reproduce under load first, then fix what waits on wall time. - -Run alone, the test target does not compile: `src/server.rs` calls `hang::Catalog::subscribe`, which needs hang's `async` feature, and only workspace feature unification enables it. -Enable it in moq-srt's dev-dependencies, as moq-json does for its own wrappers. - -Decided 2026-11-19 (coordinator, maintainer away, merge granted): keep this follow-up. diff --git a/rs/moq-srt/Cargo.toml b/rs/moq-srt/Cargo.toml index 842a550ae82..733bd89d69a 100644 --- a/rs/moq-srt/Cargo.toml +++ b/rs/moq-srt/Cargo.toml @@ -28,6 +28,8 @@ tokio = { workspace = true, features = ["full"] } tracing = { workspace = true } [dev-dependencies] +# `Catalog::subscribe` in the tests. +hang = { path = "../hang", features = ["async"] } kio = { workspace = true, features = ["runtime"] } srt-protocol = "0.4" tracing-subscriber = { workspace = true } From 247165becb99ec8447175c8f624933c5f2e11deb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 19 Nov 2026 11:44:06 -0800 Subject: [PATCH 3/4] fix(moq-srt): drop the .mbx.toml opt-out; the quest waits on the mbx fix Co-Authored-By: Claude Opus 5.5 --- .mbx.toml | 7 ------- quest/m0/test-flakes-3/srt-server-load.md | 18 ++++++++++++++++++ 2 files changed, 18 insertions(+), 7 deletions(-) delete mode 100644 .mbx.toml create mode 100644 quest/m0/test-flakes-3/srt-server-load.md diff --git a/.mbx.toml b/.mbx.toml deleted file mode 100644 index 553bc1dd01e..00000000000 --- a/.mbx.toml +++ /dev/null @@ -1,7 +0,0 @@ -# Workspace settings for mbx, the Cargo wrapper that caches our builds. - -[scheduler] -# Never queue a test for a scheduler permit. nextest starts a test's timeout when -# it starts mbx's runner, so a test held in the queue on a loaded host is killed -# before it runs. `quest gate` and nextest's own thread count bound test load. -tests = false diff --git a/quest/m0/test-flakes-3/srt-server-load.md b/quest/m0/test-flakes-3/srt-server-load.md new file mode 100644 index 00000000000..b0991dc3074 --- /dev/null +++ b/quest/m0/test-flakes-3/srt-server-load.md @@ -0,0 +1,18 @@ +# [XS] Tests queued for an mbx permit time out under load + +## Goal + +No test hits nextest's timeout while it waits for an mbx scheduler permit, so `moq-srt`'s `server::tests` (and every other test) pass however loaded the host is. + +## Plan + +Four `moq-srt` `server::tests` hit nextest's 60s timeout in a loaded `just check` ([moq-dev/moq#9123](https://github.com/moq-dev/moq/pull/9123)); three are purely synchronous, so the wait was never in the test code. +The cause is mbx, the Cargo wrapper: the host's global mbx config sets `scheduler.tests = true`, so each test waits for a host-wide permit inside `mbx-test-runner`, and nextest counts that wait toward the test's `slow-timeout`. +Any test in the workspace can hit it; moq-srt's were just in flight. + +Decided 2026-11-19 (maintainer, [moq-dev/moq#9163](https://github.com/moq-dev/moq/pull/9163#issuecomment-7020242924)): fix it in mbx or the dotfiles mbx config, not with a checked-in `.mbx.toml` opt-out or a longer `terminate-after` here, so host-wide test scheduling stays on. +Once that lands, reproduce below to confirm, then delete this quest. + +## Waiting + +- mbx or the dotfiles mbx config: stop a test's queue wait for a scheduler permit from counting toward nextest's timeout (for example, take the permit before nextest starts the test's clock, or keep `scheduler.tests` off for nextest). Check with the one-core reproduction from [moq-dev/moq#9163](https://github.com/moq-dev/moq/pull/9163): run `cargo nextest run -p moq-srt --lib --test-threads 4 --stress-count 20` with `KIO_DENY_HOST_TIMERS=1` inside `systemd-run --user --scope -p CPUQuota=100%`, with no `MBX_SCHEDULER_TESTS` override. Before the fix about 20 tests time out at 60s, each printing `mbx: waiting for machine-wide scheduler capacity`; after it, all pass. From f12bbdc82b5cb8e3d66f4943ccab3aef579d1eb9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 19 Nov 2026 11:44:49 -0800 Subject: [PATCH 4/4] quest: keep host-wide test scheduling in the waiting item Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-3/srt-server-load.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quest/m0/test-flakes-3/srt-server-load.md b/quest/m0/test-flakes-3/srt-server-load.md index b0991dc3074..6322c43ac18 100644 --- a/quest/m0/test-flakes-3/srt-server-load.md +++ b/quest/m0/test-flakes-3/srt-server-load.md @@ -15,4 +15,4 @@ Once that lands, reproduce below to confirm, then delete this quest. ## Waiting -- mbx or the dotfiles mbx config: stop a test's queue wait for a scheduler permit from counting toward nextest's timeout (for example, take the permit before nextest starts the test's clock, or keep `scheduler.tests` off for nextest). Check with the one-core reproduction from [moq-dev/moq#9163](https://github.com/moq-dev/moq/pull/9163): run `cargo nextest run -p moq-srt --lib --test-threads 4 --stress-count 20` with `KIO_DENY_HOST_TIMERS=1` inside `systemd-run --user --scope -p CPUQuota=100%`, with no `MBX_SCHEDULER_TESTS` override. Before the fix about 20 tests time out at 60s, each printing `mbx: waiting for machine-wide scheduler capacity`; after it, all pass. +- mbx or the dotfiles mbx config: stop a test's queue wait for a scheduler permit from counting toward nextest's timeout, for example by taking the permit before nextest starts the test's clock. Check with the one-core reproduction from [moq-dev/moq#9163](https://github.com/moq-dev/moq/pull/9163): run `cargo nextest run -p moq-srt --lib --test-threads 4 --stress-count 20` with `KIO_DENY_HOST_TIMERS=1` inside `systemd-run --user --scope -p CPUQuota=100%`, with no `MBX_SCHEDULER_TESTS` override. Before the fix about 20 tests time out at 60s, each printing `mbx: waiting for machine-wide scheduler capacity`; after it, all pass.