Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .claude/skills/verify-failover/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,26 +16,27 @@ Each rule is an INVARIANT, except `Settles`, which is TLC's deadlock check: ever
| `CleanEndIsComplete` | The client's track ends cleanly only with every group handed out. | The viewer loses media silently at the end. |
| `CleanEndIsDeclared` | The client's track ends cleanly only after the origin ended it. | A failover reads as the broadcast ending. |
| `NoSkipWhileHeld` | A group the client started fails only once `E` has no route left. | A group is cut short while another path holds the rest. |
| `SettledIsWhole` | A finished read holds every frame of every group. | A group ends short of its last frame. |
| `SettledIsWhole` | A finished read holds every frame of every group, unless `E` has no route left. | A group ends short of its last frame. |
| `Settles` (deadlock) | Every path ends with the read over, or, once the client left, with no relay holding a copy. | Playback hangs, or a relay keeps its upstream subscribed after the viewer left. |

## Run

```sh
quest gate just rs tla moq-net/verify/failover/FailoverChain.tla
quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Leave.cfg
quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Deaths.cfg
quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Mesh.cfg -simulate num=200000 -depth 100
```

The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 3.8M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 4.1M states. Each takes about three minutes and passes at the sync commit. The third adds `B`'s route through `A`, so the route also moves at the middle hop, after `B`'s track ended too (`Action::Reroute`). It is too large to exhaust (past 139M distinct states at depth 38 after an hour, its queue still growing), so its `Simulate:` line samples 800k random behaviors in about 7 minutes, and Nightly samples new ones each run. An exhaustive run passed every state through depth 37. Before the reroute, a sample that size missed the depth-31 hang that BFS found after about 48M states, so a passing sample is weak evidence: check a change at the middle hop by running it without the `-simulate` arguments for as long as it takes.
The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 6.0M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 6.6M states. With one death `E` always has a route, so the third lets both `A` and `B` die: only there does `E`'s front end the track (`FrontFail`), once no dead relay's route still shows (`Withdraw`): about 9.2M states. Each takes three to four minutes and passes at the sync commit. The fourth adds `B`'s route through `A`, so the route also moves at the middle hop, after `B`'s track ended too (`Action::Reroute`). It is too large to exhaust (past 139M distinct states at depth 38 after an hour, its queue still growing), so its `Simulate:` line samples 800k random behaviors in about 7 minutes, and Nightly samples new ones each run. An exhaustive run passed every state through depth 37. Before the reroute, a sample that size missed the depth-31 hang that BFS found after about 48M states, so a passing sample is weak evidence: check a change at the middle hop by running it without the `-simulate` arguments for as long as it takes.

## Reading a failure

Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the real stack in `rs/moq-net/tests/integration/failover_chain.rs`: relays are `origin::Producer`s linked over the memory transport (`link`, `priced_link`, a cost to make a route win), a relay or hop dies by aborting its sessions, a group read past a failed copy is what makes `Recover` fetch, and time is simulated. Pin a path the code follows as a `residual_*` test with its own config so the others still pass.

## Coverage

Sync commit: `61f89568394638553a519d7d8cfc0019385dd3a0`, plus [moq-dev/moq#9048](https://github.com/moq-dev/moq/pull/9048), which mirrored the code since 88572c583e and kept a finished track a read holds past its linger, and [moq-dev/moq#9155](https://github.com/moq-dev/moq/pull/9155), which resumes the newest group past the frames held and lends them to the new copy (`Resume`, `Lend`). Run `git log 61f8956839.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/front.rs rs/moq-net/src/model/origin.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
Sync commit: `46059266ec9e149c37147577bc40ca3a7cea556a`, plus the splice-stall fix, which holds a track while a dead relay's route still shows, since a session closes its sources before it fails their copies (`Shows`, `Withdraw`). Run `git log 46059266ec.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/front.rs rs/moq-net/src/model/origin.rs rs/moq-net/src/model/track.rs rs/moq-net/src/lite/subscriber.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.

| Code | Model |
| --- | --- |
Expand All @@ -47,9 +48,10 @@ Sync commit: `61f89568394638553a519d7d8cfc0019385dd3a0`, plus [moq-dev/moq#9048]
| `rs/moq-net/src/model/resume.rs` `Recover::poll_fetch`, `Fetching`; `rs/moq-net/src/model/track.rs` `fetch_group` | `Ask` |
| `rs/moq-net/src/model/front.rs` `track_ended`, `Action::Finish`; `rs/moq-net/src/model/origin.rs` `TrackIo::end` | `FrontFinish`, `Whole` |
| `rs/moq-net/src/model/front.rs` failover and its unroutable end, `Action::Reroute` | `Switch`, `FrontFail` |
| `rs/moq-net/src/lite/subscriber.rs` `TrackServe::close_if_ending`, `SubscriberDriver::abort` | `Shows`, `Withdraw`, `FrontFail` |
| `rs/moq-net/src/model/resume.rs` `Route::answering`, `Phase::Finished` | `Answering`, `LetGo` |
| `rs/moq-net/src/model/front.rs` `Action::Forget`; `rs/moq-net/src/model/track.rs` `TrackWeak::is_held`, `TrackWeak::poll_held_changed` | `Forget`, `Reads` |

## Not covered

Routing is abstracted to any live upstream winning, so which route wins and announce order are the router's own tests' (`rs/moq-net/src/route/tests`); how a reader picks among its copies, bounds replaced copies, and orders groups is `verify-resume`'s; drops, skips, and re-offers on a hop are `verify-subscribe`'s. The client's track fails only through `FrontFail`, so whether one dead relay can end playback is that step's guard, not a rule. Every hop has an hour's budget, so nothing goes stale or expires. No source refuses the track, so a finished track's reroute never concludes (`Action::Conclude`). Time is abstracted, so the linger may run out at any point once nothing reads the track. The origin writes each group whole and never fails, `E` never dies, a dead relay never returns, and a relay's cache is dropped when it parks. A fetched group holds its head from frame 0 (`lend_head`), and compressed groups are left out. The model lends only at a switch, so the query a front opens on a route before switching to it (`Action::Query`, `follow_demand`) is the unit test `a_query_resuming_a_new_copy_lends_it_the_head`, and a loan never runs out of cache budget. A pass covers two groups of two frames, one death, one switch per relay, and one client.
Routing is abstracted to any live upstream winning, so which route wins and announce order are the router's own tests' (`rs/moq-net/src/route/tests`); how a reader picks among its copies, bounds replaced copies, and orders groups is `verify-resume`'s; drops, skips, and re-offers on a hop are `verify-subscribe`'s. The client's track fails only through `FrontFail`, so whether a dead relay whose route still shows can end playback is that step's guard, not a rule. Every hop has an hour's budget, so nothing goes stale or expires. No source refuses the track, so a finished track's reroute never concludes (`Action::Conclude`). Time is abstracted, so the linger may run out at any point once nothing reads the track. The origin writes each group whole and never fails, `E` never dies, a dead relay never returns, and a relay's cache is dropped when it parks. A fetched group holds its head from frame 0 (`lend_head`), and compressed groups are left out. The model lends only at a switch, so the query a front opens on a route before switching to it (`Action::Query`, `follow_demand`) is the unit test `a_query_resuming_a_new_copy_lends_it_the_head`, and a loan never runs out of cache budget. A pass covers two groups of two frames, one death (two in `Deaths.cfg`), one switch per relay, and one client.
117 changes: 55 additions & 62 deletions quest/m2/multicast/splice-stall.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,70 +19,63 @@ or in the nightly ring meshes, once `[[known]]` no longer names this quest in
Found by [the mesh replay](/test/mesh-replay/README.md). On main after
[roll-echo-stall](https://github.com/moq-dev/moq/pull/9174) and
[reopen-starve](https://github.com/moq-dev/moq/pull/9179), 19 of 20
`reopen.toml` runs (ring4, 32 broadcasts, debug build) passed. The failing run
starved 2 reader paths. After
[resend](https://github.com/moq-dev/moq/pull/9155) landed, 2 of 3 passed. The
third shows the same strand: `a`'s reader got Unroutable for `c/5` as `a`'s
links reopened, then read nothing for 73s, across a re-follow of the path.
That run's 128 paths starved at the kill came while the host ran at 100% CPU,
so they are load, not this bug.

What that run's logs show. `b`'s reader followed `d/0` and `d/3` through `a`.
The `a`-`b` link paused, and its session idle-timed out 10s later. At that
moment, relay `b` reset the reader's live subscriptions for exactly those two
paths with Unroutable (`subscribe error ... err=unroutable` in the reader's
log). The reader's follow logged no end and stayed online. Relay `b` never
withdrew or updated `d/0`, so the reader's front, which strands a track
refused as unroutable until its table moves (`model/front.rs`
`track_ended` and `refuse`, `Track::stranded`, `unstrand`), never asked again.
The other 30 paths at `b` switched to the copy through `c`. That copy handed
out about one group every 5s until the `a`-`b` link returned, then fed every
reader again.

Reopen-starve's first plan named this strand: a refused track waits for
its table to move. #9179 removed the refusal between relays (a request follows
a claim only through a neighbour that holds it). The strand remains one hop
down, between a relay and its client.

Open, and why it could not be settled yet:

- Why relay `b` ended those two tracks Unroutable. The run had info logs only.
Candidates: `Front::selected` finding no candidate (#9179's `holds` filter
leaves none while a neighbour's copy of the claims lags its route) with no
orphaned claim to wait on, so the front ends `Unroutable`. Or the publisher's
node filter refusing. Reruns with debug logs on `lite::publisher`,
`lite::subscriber`, and `model` failed only on hosts saturated by other
work (100% CPU, relays at 5-30% of a core). There the measuring reader
itself starved on loopback, so those runs show nothing about this bug.
- A mock-time reproduction. A ring4 integration test on `peer_with_latency`
links passed: 8 to 32 broadcasts from `d`, readers on `a`, `b`, and `c`
over their own sessions, one link cut or paused. It also passed with the
cut swept from 0 to 1.5s after the readers start. The window needs timing
the mocked clock does not hit.

Recommended fix, pending the cause: a hop must not answer a live subscription
with Unroutable while it keeps showing that broadcast available. Either the
front waits for a way as it does for an orphaned claim (`Front::waiting`), or
the hop updates its announcement so the downstream's table moves. Fixing only
the client to re-ask on a quiet table would need a timer and could loop on a
standing refusal. Rerun `/verify-failover`, since either changes the front
lifecycle it covers.

Next steps: capture one failure with debug logs on a host that is not
saturated, and record per-relay CPU with it to tell load apart from logic.
Find which path in relay `b` produced the Unroutable. Write the test at the
lowest layer that shows it, likely a front unit test in `model/front.rs`.
Then fix it and rerun `reopen.toml` at least eight times.
`reopen.toml` runs (ring4, 32 broadcasts, debug build) passed. In the failing
run, `b`'s reader followed `d/0` and `d/3` through `a`. When the `a`-`b`
session idle-timed out, relay `b` reset the reader's live subscriptions for
exactly those two paths with Unroutable, and never withdrew or updated `d/0`.
The reader's front strands a track refused as unroutable until its table moves
(`model/front.rs` `refuse`, `Track::stranded`, `unstrand`), so it never asked
again. The other 30 paths switched to the copy through `c`. That run had info
logs only.

Decided 2026-11-20 by the maintainer:

- ✅ The hop holds (rejected: the reader re-asks on a timer, which adds a
backstop timer and can loop on a refusal that stands). A hop never answers a
live subscription with Unroutable while it still shows that broadcast
available. Its front either waits for a route, as it already does for an
orphaned claim (`Front::waiting`), or changes its announcement so the reader
asks again.
- ✅ The lite-07 cut waits on this fix.

What [moq-dev/moq#9219](https://github.com/moq-dev/moq/pull/9219) did:

- Closed one way a hop broke that rule. A front reads whether a source is
closing from its broadcast, and takes a failure from a source still open as
the track's verdict. A lite session could fail the copies it carries, or a
query waiting on TRACK_INFO, before it closed the broadcasts they came from:
a front that saw a copy die that way asked the dying source again and ended
the track with its answer, which a relay passes on as Unroutable. The lite
subscriber now closes a source before failing its tracks with the session's
end (`TrackServe::close_if_ending`, `SubscriberDriver::abort`), so the front
holds the track for the next route. It reads the session's state, not the
error, since a stream's own reset can carry the same error. The IETF
subscriber is unchanged. `/verify-failover` models the window: a dead relay's
route still shows until it leaves the table (`Shows`, `Withdraw`).
- Added a debug line for every front step (`moq_net::model`), so a failing run
shows how a front ended a track.

That race was not confirmed as this failure. On an unloaded host, 59 base runs
passed, as did 21 release runs of the fix that landed (and 24 debug runs of an
earlier, error-based fix it replaced), and no reader got Unroutable during the
single link reopen in any of them. Holding the drain open 2s after the transport closed
did not make the base fail either. Traced relays showed the source closing
before its copies end, the order the fix now guarantees. Debug runs on the CI host at load 70 or more starve
readers by load alone: sessions to the detours idle-time out too.

Next steps: capture one failure with
`RUST_LOG=info,moq_net::model=debug`, on a host under real but not saturating
load (relays below a core each), and record per-relay CPU with it. Rerun
`reopen.toml` on a debug build until a reader gets Unroutable during the
single link reopen; that alone marks the bug, starving or not. In the failing
relay's log, find the front step that ended that path. Then write the test at
the lowest layer that shows it, fix it as decided, rerun `/verify-failover`,
and rerun the replay at least 20 times before dropping the `[[known]]` entries
in `test/mesh-replay/script.toml`.

Pauses are not part of this quest. A debug build pauses about as long on
lite-06 as on lite-07 under the same load: link 13-14s, relay reopen 24-35s,
kill 25-30s. A release build pauses the same on both versions: link 11-12s,
relay reopen 17-19s, kill 20-23s. The release 0.17.17 binary pauses 10s, 16s,
and 19s at a lower load.

The lite-07 cut waits on it, as decided for reopen-starve on 2026-11-19
(under the maintainer's delegation for lite-07 routing calls): it is a
starved reader after a link flap or a crash, which a deployment sees daily.
lite-06 as on lite-07 under the same load, and a release build the same on
both versions.

Not a wire change.

Expand Down
Loading