Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions quest/m0/test-flakes-3/track-replacement-bun.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# [M] Track replacement under Bun finishes every run

## Goal

`just test interop --all` passes `track replacement js-native-bun` on
WebSocket and WebTransport on every run.

## Plan

Seen while validating two PRs on 2026-11-18, and not caused by either:

- [moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053): 1 of 4 runs
on a clean `origin/main`, 2 of 4 on the branch, on either transport.
- [moq-dev/moq#9084](https://github.com/moq-dev/moq/pull/9084): once on
WebSocket; the rerun passed.

The client (`test/interop/clients/js-native/replace.ts`, run by
`run_replacement_case` in `test/interop/interop.sh`) hangs until the harness
timeout instead of printing `replacement automatic=64 immediate=2 resumed=2
missing-group-fetch=2`. Every failure recorded was under Bun; whether Node
hangs too is not known.

The case runs on moq-lite-07-wip, whose relay side
[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) replaced. Find
first whether the hang predates it, then whether it sits in the relay,
`js/net`, or the native addon under Bun, and fix that cause rather than the
timeout.

Public API: none. Wire: none expected.
4 changes: 3 additions & 1 deletion quest/m2/multicast/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@ chain name its announcing relay, then each grant's exporter and adopter.
Its proof is
`rs/moq-net/verify/route-algebra/`, and the quests written against the old
wire were retargeted or deleted. The lite-07 cut still waits on this epic's
last child, [the uncovered cases](/quest/m2/multicast/accept.md), and on
children, [the uncovered cases](/quest/m2/multicast/accept.md),
[route counters](/quest/m2/multicast/route-counters.md), and
[the mesh replay](/quest/m2/multicast/mesh-replay.md), and on
[route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring
the lite-06 mitigations. After the cut come warmth's wire
([cache-aware routing](/quest/m3/cache-routing/README.md)),
Expand Down
75 changes: 75 additions & 0 deletions quest/m2/multicast/mesh-replay.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# [L] Real relays replay a mesh and its faults

## Goal

One recipe starts a `moq-relay` process for each relay of a mesh read from a
file, joins them over links that queue until they drop, and plays a script
of faults against it while publishers and subscribers read through:

- a link reopening, and one relay reopening every link it has, twice;
- a relay killed and restarted;
- a roll that drains each relay with GOAWAY and replaces its binary, one
relay at a time, from the last published release to this checkout and back.

It fails when a reader with a route to its broadcast stays unfed, a
broadcast stays shown after its source ended, or a relay does not go quiet
once the script does. For each event it reports the route and announce
messages and bytes, the time until the mesh is quiet, the broadcasts
rewritten into the origin, and each reader's interruption.

Nightly runs it on a fixed mesh whose routes cross three links. A deployment
replays its own graph by passing its own file.

## Plan

Decided 2026-11-18 (maintainer, in the coordinating session, from a survey of
the moq.pro deployment): ✅ planned (rejected: leaving it out). What it adds
to the tests that exist:

- The route simulator and its sweeps run generated shapes over mock
sessions. The drills inject one fault over real sockets into a relay or a
pair. `just test wire-compat` and the cut's two-relay smoke pair one
released relay with one from this checkout.
- moq.pro rolls one relay at a time, so its mesh spends hours with the
newest and previous release meshed, and lite-07 is chosen link by link:
only two new relays speak it to each other. Nothing runs a mesh in that
state, where one path crosses both versions.
- The same decision keeps the per-broadcast rewrite of
[route benches](/quest/m3/route-benches.md) in m3 on the condition that it
is measured on a deployed mesh's shape before that mesh's cluster links
move to lite-07. This quest's report is that measurement.

Decided 2026-11-18 (maintainer, on
[moq-dev/moq#9110](https://github.com/moq-dev/moq/pull/9110)): ✅ it blocks
the lite-07 cut, under this epic (rejected: m3, starting now without the cut
waiting on it). After the cut the lite-07 wire is frozen, so a wire flaw
found on a real mesh mid-roll would wait for lite-08. A flaw it finds is
fixed on lite-07-wip, in the quest that owns the code, before the cut.

Agent's choices, for the implementer to revisit:

- Reuse the drills' shaper for the links and `test/interop/compat.sh`'s
download of the released relay.
- The file names relays and links, each link with its cost and whether
either end marks it. The script is a list of timed events, so two runs
share one sequence and their reports compare.
- Sweep the report over both axes: relays in the mesh and broadcasts per
relay.
- The counts are a report, not a gate, as benchmarks are (2026-10-20,
maintainer).
- A pair of edges joined over lite-06 can still transit, a known gap of the
lite-06 path that ends when the mesh finishes its roll. Name it in the
report instead of failing on it.

Public API: none. Wire: none.

## Required

- [A relay counts what its route layer does](/quest/m2/multicast/route-counters.md) - the counts the report reads from each relay

## Related

- [Route benches](/quest/m3/route-benches.md) - the costs this measures on a real mesh
- [The cases no test covers run on real sessions](/quest/m2/multicast/accept.md) - the same failures on mock sessions with mocked time
- [Random fault schedules](/quest/m4/cluster-fault-schedules.md) - searches fault timings on the simulated clock
- [A nightly soak](/quest/m4/relay-soak.md) - one real relay under churn
53 changes: 53 additions & 0 deletions quest/m2/multicast/route-counters.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# [M] A relay counts what its route layer does

## Goal

From a relay's `/metrics` and its stats, an operator rolling a mesh onto
moq-lite-07 can tell, per peer label and without reading logs:

- how often its routes change, and how many broadcasts each change rewrote
into the origin;
- how many withdrawals it sent and received, DEAD and DRAIN apart;
- why it refused a route or a claim, by kind, where `routes_refused` gives
one number today;
- whether a peer's announce queue fell behind or overran;
- and how many sessions it closed for a route error, by kind.

`doc/concept/stats.md` says what a rise in each one means during a rollout.

## Plan

Decided 2026-11-18 (maintainer, in the coordinating session, from a survey of
the moq.pro deployment): ✅ planned (rejected: leaving it out). The route
layer's only signal in a relay's stats is `routes_refused`, and `/metrics`
has announce and control counters and nothing for routes. moq.pro judges a
canary relay by its counters against its neighbours', so a route layer that
churns or disagrees between releases is invisible to the roll that would
catch it.

Decided 2026-11-18 (maintainer, on
[moq-dev/moq#9110](https://github.com/moq-dev/moq/pull/9110)): ✅ it blocks
the lite-07 cut, under this epic, with
[the mesh replay](/quest/m2/multicast/mesh-replay.md) that reads these
(rejected: m3). The release a deployment rolls onto lite-07 then carries
them.

Agent's choices, for the implementer to revisit:

- Count in `rs/moq-net/src/stats.rs` beside `routes_refused`, per label and
never per broadcast or per node, so the cost and the output stay flat as
the mesh grows.
- The count of broadcasts rewritten is the cost
[route benches](/quest/m3/route-benches.md) measures, read from a running
relay.
- Say what the announce counters mean on a lite-07 session, where a route
change is no longer one announce per broadcast, and fix the doc or the
counter if they now mislead.

Public API: stats fields and metric names, additive; nest them under their
shared role. Wire: none.

## Related

- [Route benches](/quest/m3/route-benches.md) - measures the rewrite these count
- [Real relays replay a mesh and its faults](/quest/m2/multicast/mesh-replay.md) - reads these in its report
44 changes: 44 additions & 0 deletions quest/m3/route-race-stress.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# [M] Nightly races the route layer across threads

## Goal

Nightly runs a seeded stress of a publisher, a relay, and readers over mock
moq-lite-07 sessions on a multi-threaded runtime, mixing the steps whose
order two tasks decide: a broadcast ending as another starts on the same
session, a request arriving as its route lands, a grant widening, and a
session closing during an edit. A reader that hangs, or a request refused
for a claim the reader was shown, fails the run with its seed, its round,
and the steps in flight. PRs run a short fixed count.

## Plan

Decided 2026-11-18 (maintainer, in the coordinating session): ✅ planned
(rejected: leaving it out).

[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) put a claim in
front of a reader before its route reached the origin, so a request could be
refused as unroutable. Every route test scripts its order on one thread, and
only the interop `finish` lane saw it, intermittently.
[moq-dev/moq#9052](https://github.com/moq-dev/moq/pull/9052) found it with a
scratch stress that was not kept: one publisher, one relay, and 8 readers,
ending one broadcast and announcing another on the same session. It failed
within 72 to 827 rounds before the fix and ran 18,000 clean after.

Agent's choices, for the implementer to revisit:

- m3: it tests code, not the wire, so a bug it finds after the cut is a
patch release.
- Show it has teeth: with #9052's fix reverted it fails inside the nightly
budget. Measure the rounds that takes and report it.
- Real threads do not replay from a seed. Try a controlled scheduler first,
such as `shuttle`, if the mock sessions and `kio` run under one; otherwise
log the steps in flight so a failure can be read without a replay.
- [Random fault schedules](/quest/m4/cluster-fault-schedules.md) searches
fault timings on the simulated clock and may come to cover this. Say in
the PR which orders it cannot reach.

Public API: none. Wire: none.

## Related

- [Random fault schedules](/quest/m4/cluster-fault-schedules.md) - the simulated search this sits beside