Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion quest/m0/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,5 +258,5 @@ Public API: none. Wire: none.
## Required

- [js/net's lite-07 datagram queue test holds up under load](/quest/m0/test-flakes-2/datagram-queues.md) - the test waits for the first track's queue instead of asserting at once, and its `test.skip` is removed
- [Interop's cpp and rust finish cases hold up under load](/quest/m0/test-flakes-2/interop-finish.md) - the publishers' finish cells pass under load, and their quarantine is removed
- [Interop's finish cases hold up under load](/quest/m0/test-flakes-2/interop-finish.md) - every publisher's finish cells pass under load, and the scenario's quarantine is removed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Required entry now covers every publisher, but the provenance paragraph three lines up (quest/m0/test-flakes-2/README.md:252-254) still reads "interop's finish cases with a cpp or rust publisher … They are quarantined until their child fixes them". Since the code now quarantines all five publishers, that decision record understates what the epic is tracking. Widening it to "interop's finish cases with any publisher" (and noting the go failures) keeps the epic consistent with run_finishes.

- [The Go wrapper's cancelled-RecvGroup test holds up under load](/quest/m0/test-flakes-2/go-recv-cancel.md) - a `RecvGroup` after a cancelled one is woken by the next group, and its `t.Skip` is removed
29 changes: 18 additions & 11 deletions quest/m0/test-flakes-2/interop-finish.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,36 @@
# [S] Interop's cpp and rust finish cases hold up under load
# [S] Interop's finish cases hold up under load

## Goal

`just test interop --finish --all` passes its `finish cpp -> *` and
`finish rust -> *` cells on a loaded host, and their quarantine in
`test/interop/interop.sh`'s `run_finishes` is removed.
`just test interop --finish --all` passes every `finish <publisher> -> *` cell
on a loaded host, and the scenario's quarantine in `test/interop/interop.sh`'s
`run_finishes` is removed.

## Plan

Planned 2026-11-09 (maintainer, /quest-complete), under the repository's
flaky-test rule. Two agents' local `just test interop --all` runs, both under a
host load average above 30, failed once each and passed on rerun. No logs were
kept.
flaky-test rule, and widened the same day to every publisher. Local
`just test interop --all` runs under a host load average above 30 failed once
each and passed on rerun. No logs were kept.

- All nine `finish cpp -> *` cells failed together, while iterating
[moq-dev/moq#8358](https://github.com/moq-dev/moq/pull/8358).
- The `finish rust -> *` cells failed, while iterating
[moq-dev/moq#8220](https://github.com/moq-dev/moq/pull/8220): the Rust
publisher never printed `finish acknowledged`, though every reader ended
clean.
- The `finish go -> *` cells failed, while iterating
[moq-dev/moq#8366](https://github.com/moq-dev/moq/pull/8366). The same run
also failed `js -> c` and `rust-swapped -> js-native-node` once; those lanes
are not quarantined, only recorded here as seen under the same load.

Suspected, not confirmed: with three publishers failing `finish` the same way,
the cause is shared, most likely the harness's own waits under load (the
publisher's wait for the finish acknowledgement, the FIFO the runner writes
after every reader ends, and the readers' deadlines), not one implementation.

Reproduce it first, under load as the epic describes, with the interop logs
kept. Suspected, not confirmed: the publisher's wait for the finish
acknowledgement (the FIFO the runner writes after every reader ends). That
cpp's cells all failed together also points at its publisher rather than any
reader. Fix the cause, never with a longer limit, then remove the quarantine.
kept, and check whether the two unquarantined lanes fail the same way. Fix the
cause, never with a longer limit, then remove the quarantine.

Public API: none. Wire: none.
22 changes: 13 additions & 9 deletions test/interop/interop.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1770,6 +1770,7 @@ run_finish_gst() {
echo "finish tail frames=10 end=n/a"
}

# shellcheck disable=SC2329 # unreachable while finish is quarantined
finish_ready() {
local pid="$1" log="$2" marker="$3" deadline
deadline=$(awk -v now="$SECONDS" -v t="$TIMEOUT" 'BEGIN { print int(now + t + 1) }')
Expand All @@ -1789,6 +1790,7 @@ run_finish_reader() {
}

# The line a finish reader prints once every check it can make has passed.
# shellcheck disable=SC2329 # unreachable while finish is quarantined
finish_marker() {
if [[ "$1" == gst ]]; then
echo 'finish tail frames=10 end=n/a'
Expand All @@ -1810,17 +1812,19 @@ run_finishes() {
done
return
fi
# Quarantined: quest/m0/test-flakes-2/interop-finish.md. The cpp, go, and rust publishers'
# finish cases failed under a loaded host and passed on rerun.
for pub in "${PUB_LIST[@]}"; do
for sub in "${SUB_LIST[@]}"; do
echo " SKIP finish $pub -> $sub (quarantined)"
record skip case=finish publisher="$pub" subscriber="$sub" detail="quarantined: quest/m0/test-flakes-2/interop-finish.md"
done
done
return

# shellcheck disable=SC2317 # unreachable while quarantined
for pub in "${PUB_LIST[@]}"; do
name="finish-$pub"
# Quarantined: quest/m0/test-flakes-2/interop-finish.md. The cpp and rust publishers'
# finish cases failed under a loaded host and passed on rerun.
if [[ "$pub" == cpp || "$pub" == rust ]]; then
for sub in "${SUB_LIST[@]}"; do
echo " SKIP finish $pub -> $sub (quarantined)"
record skip case=finish publisher="$pub" subscriber="$sub" detail="quarantined: quest/m0/test-flakes-2/interop-finish.md"
done
continue
fi
# The fixture and its .health sibling share a parent, so one prefix grant covers both:
# moq-transport's AUTH extension carries prefixes alone.
broadcast="$name-$$-$RANDOM/fixture.hang"
Expand Down
Loading