From 7ae1c5cede572d3c1e63226f01536e8214df5e02 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:34:00 -0800 Subject: [PATCH 1/5] chore(quest): claim quest/m0/test-flakes-2/dispatch-query-terminate.md Quest-Claim: 315b61fc2339f94f From b28097d62c283ee0cd2a00f39b34b15e49bb6438 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 23:09:27 -0800 Subject: [PATCH 2/5] docs(quest): checkpoint dispatch termination investigation Co-Authored-By: GPT-6.1 Sol --- .../test-flakes-2/dispatch-query-terminate.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/quest/m0/test-flakes-2/dispatch-query-terminate.md b/quest/m0/test-flakes-2/dispatch-query-terminate.md index 617fea4286d..868ac6d7abc 100644 --- a/quest/m0/test-flakes-2/dispatch-query-terminate.md +++ b/quest/m0/test-flakes-2/dispatch-query-terminate.md @@ -15,3 +15,33 @@ that PR does not touch `sh/`. It is skipped meanwhile, per AGENTS.md's rule to disable a flaky test at once. Reproduce it under load (repeated runs beside a CPU-heavy job), find the deadline or ordering it races on, and assert on the event instead of a wait, as the epic's rules say. + +Investigation 2026-11-05: + +- [pullfrog's candidate fix](https://github.com/moq-dev/moq/commit/e6690144ca14f9ec8c48a56b4ed26c5a65a47708), + removed from [moq-dev/moq#7789](https://github.com/moq-dev/moq/pull/7789), + changes the EXIT trap's bare `wait` to `wait || true`, claiming a cancelled + lane's status skips reporting under errexit. This remains unverified. +- With the skip removed temporarily, the existing test passed alone and in + 100 loaded runs on the pinned Nix Bash 5.3.20. The loaded harness restricted + itself and four CPU workers to two CPUs; each run covers CHECK_JOBS=1 and 3. +- The same harness using host Bash 5.3.9 for fixture shells had two + `communicate(timeout=60)` errors in 100 runs, both CHECK_JOBS=3. This is a + hang rather than the originally reported missing cancellation timing line. +- A candidate that cancelled Bash's job table rather than the `running` map + still had one older-shell hang in 100 runs. It is insufficient; no dispatcher + change is retained, and the test stays skipped pending a verified fix. +- A scratch launch barrier can hold the parent before registering a Rust lane. + The job-table candidate passed that forced window, but this does not explain + the residual older-shell hang or prove the original report's cause. +- A gated full `just ci origin/main` with the temporary candidate completed: + every lane except flake and CARLA passed, including the unskipped tooling + test. Flake failed on `/proc/self/uid_map` permission; CARLA's + `sim/python.test.sh` saw inherited Nix PYTHONPATH. Both are also reported in + [moq-dev/moq#7710](https://github.com/moq-dev/moq/pull/7710); they were not + repaired here. These results do not validate a final implementation. + +Next: capture the EXIT trap and child process state at an actual missing-report +failure, and distinguish that failure from the older-shell hang before choosing +between trap-status handling and launch/cancellation changes. Preserve the +existing assertions and remove the skip only with a verified cause. From ffc792a4e7a8172bce8b973ed9475b66c37574b0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 23:13:02 -0800 Subject: [PATCH 3/5] docs(quest): record deterministic launch-race comparison Co-Authored-By: GPT-6.1 Sol --- quest/m0/test-flakes-2/dispatch-query-terminate.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/quest/m0/test-flakes-2/dispatch-query-terminate.md b/quest/m0/test-flakes-2/dispatch-query-terminate.md index 868ac6d7abc..4e911c8f3d4 100644 --- a/quest/m0/test-flakes-2/dispatch-query-terminate.md +++ b/quest/m0/test-flakes-2/dispatch-query-terminate.md @@ -32,8 +32,10 @@ Investigation 2026-11-05: still had one older-shell hang in 100 runs. It is insufficient; no dispatcher change is retained, and the test stays skipped pending a verified fix. - A scratch launch barrier can hold the parent before registering a Rust lane. - The job-table candidate passed that forced window, but this does not explain - the residual older-shell hang or prove the original report's cause. + The original PID-map cancellation hung for the existing 60-second test + deadline in that forced window; job-table cancellation passed in 0.072 s. + This isolates a launch race, but does not explain the residual older-shell + hang or prove the original missing-report failure's cause. - A gated full `just ci origin/main` with the temporary candidate completed: every lane except flake and CARLA passed, including the unskipped tooling test. Flake failed on `/proc/self/uid_map` permission; CARLA's From 142ed286ae817ec7a7201ee7fce91dfa4021b4ee Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 08:52:53 -0800 Subject: [PATCH 4/5] docs(quest): retain launch race and clarify trace evidence Co-Authored-By: GPT-6.1 Sol --- .../m0/test-flakes-2/dispatch-query-terminate.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/quest/m0/test-flakes-2/dispatch-query-terminate.md b/quest/m0/test-flakes-2/dispatch-query-terminate.md index 4f10a837cb9..244932d9788 100644 --- a/quest/m0/test-flakes-2/dispatch-query-terminate.md +++ b/quest/m0/test-flakes-2/dispatch-query-terminate.md @@ -1,10 +1,12 @@ -# [XS] dispatch-query's terminate test survives load +# [M] dispatch-query's terminate test survives load ## Goal `sh/dispatch-query.test.py`'s `test_terminating_the_run_stops_the_running_phase` passes reliably under host load, fixed at its cause under the epic's rules, -and its `@unittest.skip` is removed. +and its `@unittest.skip` is removed. Cancellation also covers a lane launched +before registration in the PID map, so the confirmed launch race cannot leave +an uncancelled lane holding the EXIT trap open. ## Plan @@ -23,7 +25,10 @@ Investigation 2026-11-05: reports a failure in 1 of 420 runs on Bash 5.3.20: the EXIT trap's `wait` returns a cancelled lane's 143 and errexit skips `report`. Its `wait || true` patch reportedly passed 420/420. This remains a lead to - verify with a failing trace in the original environment. + verify with a failing trace in the original environment: argument-less + `wait` normally returns zero, while a trapped signal can interrupt it. + Record the actual status and whether a signal interrupted the wait rather + than assuming it propagated a child's status. - With the skip removed temporarily, the existing test passed alone and in 100 loaded runs on the pinned Nix Bash 5.3.20. The loaded harness restricted itself and four CPU workers to two CPUs; each run covers CHECK_JOBS=1 and 3. @@ -51,7 +56,9 @@ Investigating the older-Bash hang first was rejected because it differs from the reported missing timing line. Keep the test skipped until its cause is verified. This PR retains evidence only; the job-table cancellation candidate was discarded before this head. The isolated launch race is real, but does not -establish the cause of the pinned-shell failure. +establish the cause of the pinned-shell failure. After capturing that failure, +fix and verify the confirmed launch/cancellation race as part of this quest, +even if the missing-report failure has a different cause. ## Waiting From b58f11fd58e3e4bdf1ad61c8be512aa118099b52 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 08:59:47 -0800 Subject: [PATCH 5/5] docs(quest): separate remaining scope from maintainer decision Co-Authored-By: GPT-6.1 Sol --- quest/m0/test-flakes-2/dispatch-query-terminate.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/quest/m0/test-flakes-2/dispatch-query-terminate.md b/quest/m0/test-flakes-2/dispatch-query-terminate.md index 244932d9788..32b6bc290a5 100644 --- a/quest/m0/test-flakes-2/dispatch-query-terminate.md +++ b/quest/m0/test-flakes-2/dispatch-query-terminate.md @@ -56,8 +56,9 @@ Investigating the older-Bash hang first was rejected because it differs from the reported missing timing line. Keep the test skipped until its cause is verified. This PR retains evidence only; the job-table cancellation candidate was discarded before this head. The isolated launch race is real, but does not -establish the cause of the pinned-shell failure. After capturing that failure, -fix and verify the confirmed launch/cancellation race as part of this quest, +establish the cause of the pinned-shell failure. + +Remaining scope: after capturing that failure, fix and verify the confirmed launch/cancellation race as part of this quest, even if the missing-report failure has a different cause. ## Waiting