From 2d63b169d807939cbf097ff26e4a66c821fb12ea Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Mon, 9 Nov 2026 19:07:25 -0800 Subject: [PATCH] test(interop): quarantine the finish scenario for every publisher Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/README.md | 2 +- quest/m0/test-flakes-2/interop-finish.md | 29 +++++++++++++++--------- test/interop/interop.sh | 22 ++++++++++-------- 3 files changed, 32 insertions(+), 21 deletions(-) diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index a293a8c68ec..91489dfca6c 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -258,5 +258,5 @@ Public API: none. Wire: none. ## Required - [js/net's lite-07 datagram queue test holds up under load](/quest/m0/test-flakes-2/datagram-queues.md) - the test waits for the first track's queue instead of asserting at once, and its `test.skip` is removed -- [Interop's cpp and rust finish cases hold up under load](/quest/m0/test-flakes-2/interop-finish.md) - the publishers' finish cells pass under load, and their quarantine is removed +- [Interop's finish cases hold up under load](/quest/m0/test-flakes-2/interop-finish.md) - every publisher's finish cells pass under load, and the scenario's quarantine is removed - [The Go wrapper's cancelled-RecvGroup test holds up under load](/quest/m0/test-flakes-2/go-recv-cancel.md) - a `RecvGroup` after a cancelled one is woken by the next group, and its `t.Skip` is removed diff --git a/quest/m0/test-flakes-2/interop-finish.md b/quest/m0/test-flakes-2/interop-finish.md index fda97fc79a7..7f6e40c3f12 100644 --- a/quest/m0/test-flakes-2/interop-finish.md +++ b/quest/m0/test-flakes-2/interop-finish.md @@ -1,17 +1,17 @@ -# [S] Interop's cpp and rust finish cases hold up under load +# [S] Interop's finish cases hold up under load ## Goal -`just test interop --finish --all` passes its `finish cpp -> *` and -`finish rust -> *` cells on a loaded host, and their quarantine in -`test/interop/interop.sh`'s `run_finishes` is removed. +`just test interop --finish --all` passes every `finish -> *` cell +on a loaded host, and the scenario's quarantine in `test/interop/interop.sh`'s +`run_finishes` is removed. ## Plan Planned 2026-11-09 (maintainer, /quest-complete), under the repository's -flaky-test rule. Two agents' local `just test interop --all` runs, both under a -host load average above 30, failed once each and passed on rerun. No logs were -kept. +flaky-test rule, and widened the same day to every publisher. Local +`just test interop --all` runs under a host load average above 30 failed once +each and passed on rerun. No logs were kept. - All nine `finish cpp -> *` cells failed together, while iterating [moq-dev/moq#8358](https://github.com/moq-dev/moq/pull/8358). @@ -19,11 +19,18 @@ kept. [moq-dev/moq#8220](https://github.com/moq-dev/moq/pull/8220): the Rust publisher never printed `finish acknowledged`, though every reader ended clean. +- The `finish go -> *` cells failed, while iterating + [moq-dev/moq#8366](https://github.com/moq-dev/moq/pull/8366). The same run + also failed `js -> c` and `rust-swapped -> js-native-node` once; those lanes + are not quarantined, only recorded here as seen under the same load. + +Suspected, not confirmed: with three publishers failing `finish` the same way, +the cause is shared, most likely the harness's own waits under load (the +publisher's wait for the finish acknowledgement, the FIFO the runner writes +after every reader ends, and the readers' deadlines), not one implementation. Reproduce it first, under load as the epic describes, with the interop logs -kept. Suspected, not confirmed: the publisher's wait for the finish -acknowledgement (the FIFO the runner writes after every reader ends). That -cpp's cells all failed together also points at its publisher rather than any -reader. Fix the cause, never with a longer limit, then remove the quarantine. +kept, and check whether the two unquarantined lanes fail the same way. Fix the +cause, never with a longer limit, then remove the quarantine. Public API: none. Wire: none. diff --git a/test/interop/interop.sh b/test/interop/interop.sh index ad911f4db78..5dfb3b4af5a 100755 --- a/test/interop/interop.sh +++ b/test/interop/interop.sh @@ -1770,6 +1770,7 @@ run_finish_gst() { echo "finish tail frames=10 end=n/a" } +# shellcheck disable=SC2329 # unreachable while finish is quarantined finish_ready() { local pid="$1" log="$2" marker="$3" deadline deadline=$(awk -v now="$SECONDS" -v t="$TIMEOUT" 'BEGIN { print int(now + t + 1) }') @@ -1789,6 +1790,7 @@ run_finish_reader() { } # The line a finish reader prints once every check it can make has passed. +# shellcheck disable=SC2329 # unreachable while finish is quarantined finish_marker() { if [[ "$1" == gst ]]; then echo 'finish tail frames=10 end=n/a' @@ -1810,17 +1812,19 @@ run_finishes() { done return fi + # Quarantined: quest/m0/test-flakes-2/interop-finish.md. The cpp, go, and rust publishers' + # finish cases failed under a loaded host and passed on rerun. + for pub in "${PUB_LIST[@]}"; do + for sub in "${SUB_LIST[@]}"; do + echo " SKIP finish $pub -> $sub (quarantined)" + record skip case=finish publisher="$pub" subscriber="$sub" detail="quarantined: quest/m0/test-flakes-2/interop-finish.md" + done + done + return + + # shellcheck disable=SC2317 # unreachable while quarantined for pub in "${PUB_LIST[@]}"; do name="finish-$pub" - # Quarantined: quest/m0/test-flakes-2/interop-finish.md. The cpp and rust publishers' - # finish cases failed under a loaded host and passed on rerun. - if [[ "$pub" == cpp || "$pub" == rust ]]; then - for sub in "${SUB_LIST[@]}"; do - echo " SKIP finish $pub -> $sub (quarantined)" - record skip case=finish publisher="$pub" subscriber="$sub" detail="quarantined: quest/m0/test-flakes-2/interop-finish.md" - done - continue - fi # The fixture and its .health sibling share a parent, so one prefix grant covers both: # moq-transport's AUTH extension carries prefixes alone. broadcast="$name-$$-$RANDOM/fixture.hang"