From b5a2e8fb7716144abd756ae81cd9a780b200be0b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 05:15:33 -0800 Subject: [PATCH 1/7] chore(quest): claim quest/m1/unbounded-groups/log-lock-profile.md Quest-Claim: 105be10455aa5d77 From b723d92ee9f7610f0ddc589224a2bcbe41fee3c3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 05:40:33 -0800 Subject: [PATCH 2/7] docs(quest): record mixed quiet-host log lock evidence Co-Authored-By: GPT-6.1 Sol --- quest/m1/unbounded-groups/log-lock-profile.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/quest/m1/unbounded-groups/log-lock-profile.md b/quest/m1/unbounded-groups/log-lock-profile.md index 7aee26299ca..235541e9700 100644 --- a/quest/m1/unbounded-groups/log-lock-profile.md +++ b/quest/m1/unbounded-groups/log-lock-profile.md @@ -38,3 +38,28 @@ or holds. `group::Consumer::poll_read_frame_once` → `Prefetch::pop_cached`, which #6782 alone did not. If they persist on the quiet rerun, suggest a follow-up quest profiling `d854bf98b1` against `main`. +- A 2026-11-03 rerun began on `desktop` at 0.21 one-minute load, + with no other T3 agents running, and captured both historical bases at + 1/4/16 workers ([moq-dev/moq#7742](https://github.com/moq-dev/moq/pull/7742)). + At 4 workers, `GroupState` wait fell from 8 us (0.50% of mutex wait) to + none. At 16 workers, its share fell from 11.72% to 1.78%, but its absolute + wait rose from 47 us to 599 us in `Consumer::poll_read_frame_once` -> + `Cursor::poll_read_frame`. Total mutex wait rose from 0.401 ms to + 33.722 ms, so the share alone hides the contrary absolute result. + These single five-second captures do not establish a regression or its + absence. The PR retains the reduced evidence and stack diff. +- Current `main` at the claim's source revision has group-state waits in the + cached read path at 4 workers in both runs (26 and 18 us), and 937 us at + 16 workers in the first run. The second current 16-worker capture was + refused for nonalternating switch records, so it supplies no measurement. + A separate epoch-versus-main investigation remains a suggested follow-up, + not work owned by this quest. + +## Waiting + +- Maintainer: resolve the contrary 16-worker absolute wait before calling this + a no-regression result. Recommend repeating the two historical revisions in + alternating order with warm direct-Nix-Cargo builds, keeping the workload, + worker counts and capture validation unchanged. Alternatively, explicitly + accept the share-only goal despite the increased absolute wait. No + production change is justified by these single captures. From a648cdaf3895fc0a7a9fd00b8c01769cbd85de25 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 05:43:38 -0800 Subject: [PATCH 3/7] docs(quest): qualify profiling evidence and pin current source Co-Authored-By: GPT-6.1 Sol --- quest/m1/unbounded-groups/log-lock-profile.md | 24 ++++++++++++------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/quest/m1/unbounded-groups/log-lock-profile.md b/quest/m1/unbounded-groups/log-lock-profile.md index 235541e9700..d9ce7b5c5ee 100644 --- a/quest/m1/unbounded-groups/log-lock-profile.md +++ b/quest/m1/unbounded-groups/log-lock-profile.md @@ -41,25 +41,31 @@ or holds. - A 2026-11-03 rerun began on `desktop` at 0.21 one-minute load, with no other T3 agents running, and captured both historical bases at 1/4/16 workers ([moq-dev/moq#7742](https://github.com/moq-dev/moq/pull/7742)). + At 1 worker, both historical captures had zero blocked mutex wait, so + their shares are undefined; this does not measure the per-frame pin cost. At 4 workers, `GroupState` wait fell from 8 us (0.50% of mutex wait) to none. At 16 workers, its share fell from 11.72% to 1.78%, but its absolute wait rose from 47 us to 599 us in `Consumer::poll_read_frame_once` -> `Cursor::poll_read_frame`. Total mutex wait rose from 0.401 ms to 33.722 ms, so the share alone hides the contrary absolute result. - These single five-second captures do not establish a regression or its - absence. The PR retains the reduced evidence and stack diff. -- Current `main` at the claim's source revision has group-state waits in the - cached read path at 4 workers in both runs (26 and 18 us), and 937 us at - 16 workers in the first run. The second current 16-worker capture was + Quietness was checked before the attempt, not recorded per capture, so + these are candidate observations, not quiet-host proof. These single + five-second captures do not establish a regression or its absence. The PR retains the reduced evidence and stack diff. +- The current source was [b5a2e8fb77](https://github.com/moq-dev/moq/commit/b5a2e8fb7716144abd756ae81cd9a780b200be0b), + whose tree equals main at [801fd9a2f2](https://github.com/moq-dev/moq/commit/801fd9a2f2f5c88371fa833d805620d97d372465). + Both 4-worker captures have group-state waits under + `Consumer::poll_read_frame_once<..., Prefetch::pop_cached>` (26 and 18 us), + with 937 us of total group-state wait at 16 workers in the first run. The second current 16-worker capture was refused for nonalternating switch records, so it supplies no measurement. - A separate epoch-versus-main investigation remains a suggested follow-up, - not work owned by this quest. + The foreground agent should plan the triggered epoch-versus-main follow-up; + this child was explicitly instructed not to create it. ## Waiting - Maintainer: resolve the contrary 16-worker absolute wait before calling this a no-regression result. Recommend repeating the two historical revisions in - alternating order with warm direct-Nix-Cargo builds, keeping the workload, - worker counts and capture validation unchanged. Alternatively, explicitly + alternating order with warm direct-Nix-Cargo builds, recording host load + per capture and verifying no other agents run throughout. Keep the + workload, worker counts and capture validation unchanged. Alternatively, explicitly accept the share-only goal despite the increased absolute wait. No production change is justified by these single captures. From d59c48148852192eeec26cd39b2898f775e8bb02 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 05:46:24 -0800 Subject: [PATCH 4/7] docs(quest): make the conditional follow-up durable Co-Authored-By: GPT-6.1 Sol --- quest/m1/unbounded-groups/log-lock-profile.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/quest/m1/unbounded-groups/log-lock-profile.md b/quest/m1/unbounded-groups/log-lock-profile.md index d9ce7b5c5ee..4fd4bba75ac 100644 --- a/quest/m1/unbounded-groups/log-lock-profile.md +++ b/quest/m1/unbounded-groups/log-lock-profile.md @@ -57,8 +57,8 @@ or holds. `Consumer::poll_read_frame_once<..., Prefetch::pop_cached>` (26 and 18 us), with 937 us of total group-state wait at 16 workers in the first run. The second current 16-worker capture was refused for nonalternating switch records, so it supplies no measurement. - The foreground agent should plan the triggered epoch-versus-main follow-up; - this child was explicitly instructed not to create it. + Once a rerun establishes the quiet-host requirement, suggest the separate + epoch-versus-main quest if these cached-read waits persist. ## Waiting From 1b9f4998cb129cbf17b5b5c980ce795d8ea8490e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 11:25:13 -0800 Subject: [PATCH 5/7] docs(quest): record the nixos2 log lock repeat The repeat found no quiet window. Its one completed pre-epoch run shows the 16-worker GroupState wait on the same stack as the epoch capture, so that wait predates #6782; no valid epoch capture exists on that host yet. Co-Authored-By: Claude Opus 5.5 --- quest/m1/unbounded-groups/log-lock-profile.md | 41 +++++++++++++++---- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/quest/m1/unbounded-groups/log-lock-profile.md b/quest/m1/unbounded-groups/log-lock-profile.md index 4fd4bba75ac..28794af4241 100644 --- a/quest/m1/unbounded-groups/log-lock-profile.md +++ b/quest/m1/unbounded-groups/log-lock-profile.md @@ -59,13 +59,40 @@ or holds. refused for nonalternating switch records, so it supplies no measurement. Once a rerun establishes the quiet-host requirement, suggest the separate epoch-versus-main quest if these cached-read waits persist. +- Decided 2026-11-03 (maintainer): repeat both historical revisions in + alternating order on a quiet window, with warm direct-Nix-Cargo builds, + per-capture host load, and no other agents running throughout, keeping the + workload, worker counts and capture validation. `desktop` was busy, so the + repeat used `nixos2` (i7-8700, 6 cores / 12 threads, no hybrid pinning, + `perf_event_paranoid=-1`), with a per-capture monitor of load, CPU used + outside the run, and other agents' tool commands. +- That 2026-11-03 repeat found no quiet window. Of three runs, only one + pre-epoch run (`d854bf98b1~1`) completed its historical captures. Other + agents ran light tool commands during it, using 0.06 to 0.21 cores on + average (peak 1.0). In it, the 16-worker capture had 709 us of `GroupState` + wait (1.83% of 38.8 ms), all under `Consumer::poll_read_frame_once` -> + `poll_read_frame` -> `poll`. That is the stack behind the + epoch capture's 599 us on `desktop`, so that wait predates #6782, and the + `desktop` contrast (47 us before, 599 us after) is within one revision's + capture-to-capture spread. At 4 workers it had 9 us (1.10% of 0.82 ms). Its + current 4-worker capture had no `GroupState` wait; its current 16-worker + capture was refused for nonalternating switch records. Two other runs are + void: another session's moq bench builds used 5 to 6 cores throughout the + epoch run, whose 16-worker capture perf also refused for a dropped event, + and the first pre-epoch run overlapped another agent's builds. At 1 worker, + the relay on this CPU delivers only 20 to 23k of the 31k offered frames per + second (about 35% lifetime loss), so 1-worker captures on `nixos2` fail the + zero-loss check; they had no mutex wait. No valid epoch capture exists on + `nixos2`, so the share comparison has no same-host pair yet. ## Waiting -- Maintainer: resolve the contrary 16-worker absolute wait before calling this - a no-regression result. Recommend repeating the two historical revisions in - alternating order with warm direct-Nix-Cargo builds, recording host load - per capture and verifying no other agents run throughout. Keep the - workload, worker counts and capture validation unchanged. Alternatively, explicitly - accept the share-only goal despite the increased absolute wait. No - production change is justified by these single captures. +- Maintainer: a quiet benchmark window. `nixos2` and `desktop` both host + agent sessions and other sessions' benchmarks, so no window stayed free of + them for the roughly 45 minutes that two alternating pairs need (about + 11 minutes per run, including its 5-minute base rebuild). Recommend + reserving one x86 host with `perf_event_paranoid=-1`, launching no agents + or benchmarks there for that time, then repeating the alternating pairs. + Alternatively, accept "no regression attributable to #6782" from the + pre-epoch replicate showing the same stack at a similar absolute wait. + No production change is justified by these captures. From 37d59f306f69ecad1f45368b5f99f09e33700333 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 12:03:07 -0800 Subject: [PATCH 6/7] docs(quest): accept the log lock profile and delete its quest The maintainer accepted no lock-contention regression attributable to #6782: the pre-epoch run shows the same GroupState wait stack at a similar wait (709 vs 599 us). Record that in the unbounded-groups epic. Co-Authored-By: Claude Opus 5.5 --- quest/m1/unbounded-groups/README.md | 5 +- quest/m1/unbounded-groups/log-lock-profile.md | 98 ------------------- 2 files changed, 4 insertions(+), 99 deletions(-) delete mode 100644 quest/m1/unbounded-groups/log-lock-profile.md diff --git a/quest/m1/unbounded-groups/README.md b/quest/m1/unbounded-groups/README.md index 003c394c37e..91f178d135c 100644 --- a/quest/m1/unbounded-groups/README.md +++ b/quest/m1/unbounded-groups/README.md @@ -114,6 +114,10 @@ Decided 2026-10-16 (maintainer interview): advancing the base (`rs/kio/src/log.rs`). A ring buffer was rejected: an uncapped group has no size to give it. Numbers are in #6782 and the [kio contention epic](/quest/m2/perf/kio-channel-contention/README.md). + Accepted 2026-11-03 (maintainer), from the lock profiles in + [moq-dev/moq#7742](https://github.com/moq-dev/moq/pull/7742): no + lock-contention regression attributable to #6782. The pre-epoch run shows + the same `GroupState` wait stack at a similar wait (709 vs 599 µs). - Readers pin nothing: reads refresh recency as today. Within a group, victims go front to back. - Whole cold groups are still evicted first. A group loses part of itself @@ -153,6 +157,5 @@ covered only at the model layer (maintainer, 2026-10-30). - [Flights](/quest/m1/unbounded-groups/flights.md) - a reader at a shed frame replays through a separate paced copy that hands over to the copy ahead - [Byte holes](/quest/m1/unbounded-groups/byte-holes.md) - segments inside a frame shed like front frames and flights start at `(frame, byte)` on the current lite wip; the frame cap goes - [JS byte holes](/quest/m1/unbounded-groups/js-byte-holes.md) - `@moq/net` mirrors the byte-offset wire and continues a frame cut off mid-way -- [Log lock profile](/quest/m1/unbounded-groups/log-lock-profile.md) - a lock profile on a perf host confirms the epoch-based `kio::log` cut `GroupState` contention - [Compressed frames have no size cap and inflate as a bounded stream](/quest/m1/unbounded-groups/compressed-stream.md) - A compressed frame of any size publishes and is received in Rust and `@moq/net`, like a plain one - [FETCH egress and ingest stream large frames without buffering](/quest/m1/unbounded-groups/fetch-stream.md) - `@moq/net` serves a moq-transport FETCH object by object as it reads them, never collecting a requested group slice whole, so a FETCH of a group with large frames holds no more than the group tail diff --git a/quest/m1/unbounded-groups/log-lock-profile.md b/quest/m1/unbounded-groups/log-lock-profile.md deleted file mode 100644 index 28794af4241..00000000000 --- a/quest/m1/unbounded-groups/log-lock-profile.md +++ /dev/null @@ -1,98 +0,0 @@ -# [XS] Log lock profile - -## Goal - -`just bench-profile lock video-heavy` runs before and after the epoch-based -`kio::log` ([moq-dev/moq#6782](https://github.com/moq-dev/moq/pull/6782)) on a -quiet profiling host (see `bench/README.md`), and shows no lock-contention -regression: `GroupState`'s share of mutex wait with multi-frame groups drops -or holds. - -## Plan - -- Compare the commit before #6782 with the one that merged it, at each worker - count, and post the wait-by-stack diff on the PR that finishes this. The - recipe's base revision - ([moq-dev/moq#6680](https://github.com/moq-dev/moq/pull/6680)) covers the - "before" side in one run: - `just bench-profile lock video-heavy "1 4 16" tokio-workers d854bf98b1~1`. -- Decided 2026-10-27 (maintainer): the profiling host is the 24-core - `desktop` workstation (`kernel.perf_event_paranoid=-1`), run while quiet: - no other agents running and load near zero. That retired the profiling-host - condition this waited on. -- #6782's body holds the criterion numbers, taken on an oversubscribed host: - `group_contended` costs 13 to 61% less per reader across cores, and about - 4 ns more per frame on one thread. A profile that contradicts them goes back - to the maintainer. -- A 2026-10-30 attempt ran on `desktop` while other agents kept the 1-minute - load between 5 and 30 - ([moq-dev/moq#7300](https://github.com/moq-dev/moq/pull/7300)). At 4 workers, - `GroupState` held 1% of mutex wait before #6782 and none after it, but - unprofiled 1-worker throughput swung from 11k to 30k fps between runs, and - the 16-worker capture failed on perf dropping events or recording - nonalternating switches. Profiling `d854bf98b1` as the base as well isolates - #6782 from later changes. -- Decided 2026-10-30 (maintainer): keep this quest and rerun both bases on a - quiet `desktop`; the noisy attempt does not finish it. On `main` (4 workers) - that attempt also showed `GroupState` waits under - `group::Consumer::poll_read_frame_once` → `Prefetch::pop_cached`, which - #6782 alone did not. If they persist on the quiet rerun, suggest a follow-up - quest profiling `d854bf98b1` against `main`. -- A 2026-11-03 rerun began on `desktop` at 0.21 one-minute load, - with no other T3 agents running, and captured both historical bases at - 1/4/16 workers ([moq-dev/moq#7742](https://github.com/moq-dev/moq/pull/7742)). - At 1 worker, both historical captures had zero blocked mutex wait, so - their shares are undefined; this does not measure the per-frame pin cost. - At 4 workers, `GroupState` wait fell from 8 us (0.50% of mutex wait) to - none. At 16 workers, its share fell from 11.72% to 1.78%, but its absolute - wait rose from 47 us to 599 us in `Consumer::poll_read_frame_once` -> - `Cursor::poll_read_frame`. Total mutex wait rose from 0.401 ms to - 33.722 ms, so the share alone hides the contrary absolute result. - Quietness was checked before the attempt, not recorded per capture, so - these are candidate observations, not quiet-host proof. These single - five-second captures do not establish a regression or its absence. The PR retains the reduced evidence and stack diff. -- The current source was [b5a2e8fb77](https://github.com/moq-dev/moq/commit/b5a2e8fb7716144abd756ae81cd9a780b200be0b), - whose tree equals main at [801fd9a2f2](https://github.com/moq-dev/moq/commit/801fd9a2f2f5c88371fa833d805620d97d372465). - Both 4-worker captures have group-state waits under - `Consumer::poll_read_frame_once<..., Prefetch::pop_cached>` (26 and 18 us), - with 937 us of total group-state wait at 16 workers in the first run. The second current 16-worker capture was - refused for nonalternating switch records, so it supplies no measurement. - Once a rerun establishes the quiet-host requirement, suggest the separate - epoch-versus-main quest if these cached-read waits persist. -- Decided 2026-11-03 (maintainer): repeat both historical revisions in - alternating order on a quiet window, with warm direct-Nix-Cargo builds, - per-capture host load, and no other agents running throughout, keeping the - workload, worker counts and capture validation. `desktop` was busy, so the - repeat used `nixos2` (i7-8700, 6 cores / 12 threads, no hybrid pinning, - `perf_event_paranoid=-1`), with a per-capture monitor of load, CPU used - outside the run, and other agents' tool commands. -- That 2026-11-03 repeat found no quiet window. Of three runs, only one - pre-epoch run (`d854bf98b1~1`) completed its historical captures. Other - agents ran light tool commands during it, using 0.06 to 0.21 cores on - average (peak 1.0). In it, the 16-worker capture had 709 us of `GroupState` - wait (1.83% of 38.8 ms), all under `Consumer::poll_read_frame_once` -> - `poll_read_frame` -> `poll`. That is the stack behind the - epoch capture's 599 us on `desktop`, so that wait predates #6782, and the - `desktop` contrast (47 us before, 599 us after) is within one revision's - capture-to-capture spread. At 4 workers it had 9 us (1.10% of 0.82 ms). Its - current 4-worker capture had no `GroupState` wait; its current 16-worker - capture was refused for nonalternating switch records. Two other runs are - void: another session's moq bench builds used 5 to 6 cores throughout the - epoch run, whose 16-worker capture perf also refused for a dropped event, - and the first pre-epoch run overlapped another agent's builds. At 1 worker, - the relay on this CPU delivers only 20 to 23k of the 31k offered frames per - second (about 35% lifetime loss), so 1-worker captures on `nixos2` fail the - zero-loss check; they had no mutex wait. No valid epoch capture exists on - `nixos2`, so the share comparison has no same-host pair yet. - -## Waiting - -- Maintainer: a quiet benchmark window. `nixos2` and `desktop` both host - agent sessions and other sessions' benchmarks, so no window stayed free of - them for the roughly 45 minutes that two alternating pairs need (about - 11 minutes per run, including its 5-minute base rebuild). Recommend - reserving one x86 host with `perf_event_paranoid=-1`, launching no agents - or benchmarks there for that time, then repeating the alternating pairs. - Alternatively, accept "no regression attributable to #6782" from the - pre-epoch replicate showing the same stack at a similar absolute wait. - No production change is justified by these captures. From 686ba58acb23608b0309a842cde4726304c65a53 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 12:44:35 -0800 Subject: [PATCH 7/7] docs(quest): name the host beside each log lock wait figure The 709 and 599 us waits come from different hosts, so the epic now says they are not comparable while keeping the accepted conclusion. Co-Authored-By: Claude Opus 5.5 --- quest/m1/unbounded-groups/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/quest/m1/unbounded-groups/README.md b/quest/m1/unbounded-groups/README.md index 91f178d135c..65283e88e80 100644 --- a/quest/m1/unbounded-groups/README.md +++ b/quest/m1/unbounded-groups/README.md @@ -116,8 +116,10 @@ Decided 2026-10-16 (maintainer interview): [kio contention epic](/quest/m2/perf/kio-channel-contention/README.md). Accepted 2026-11-03 (maintainer), from the lock profiles in [moq-dev/moq#7742](https://github.com/moq-dev/moq/pull/7742): no - lock-contention regression attributable to #6782. The pre-epoch run shows - the same `GroupState` wait stack at a similar wait (709 vs 599 µs). + lock-contention regression attributable to #6782. The pre-epoch run on + `nixos2` shows the same `GroupState` wait stack (709 µs) as the epoch run on + `desktop` (599 µs). The figures come from different hosts and are not + comparable; no quiet same-host repeat was completed. - Readers pin nothing: reads refresh recency as today. Within a group, victims go front to back. - Whole cold groups are still evicted first. A group loses part of itself