Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions .claude/skills/verify-route/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH

# Verify the route rule

`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire.
`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. A route goes to a stream only while a claim meets it: one that meets none withdraws the claims sent and ends. `Timeout` fires the timers due together on one node in one update, as `Router::timeout` does: doubts lift, and each dead hop also expires on its own timer, so repeated failures of one node cannot retain all its old hops. A timer may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire.

## Rules

Expand Down Expand Up @@ -83,23 +83,29 @@ change; faster checkers are separate work.

Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`.

## Trace validation

`quest gate just test route-trace` checks that the router's own runs are behaviours of the model, as Trace validation in `rs/moq-net/verify/README.md` describes: `route::tests::Sim` records 150 runs each of `sweep_handoff` and `sweep_scoped_handoff` under `Rule::PATH` (`rs/moq-net/src/route/tests/trace.rs`), and `trace/RouteTrace.tla` follows each with every message and every touched relay's route and shown claims compared, checking the model's rules too except `Stable` (`quest/m2/cluster-routes/stable-repath.md`). Nightly picks a fresh window of scenarios (`MOQ_ROUTE_TRACE_FROM`); the printed settings replay it. Each sweep stays inside the model: one publisher whose every claim change changes a claim, relays asking for the root, no restart, collection only as a step the model does not see. A leaf that publishes nothing, the client, offers no route, so the traces drop it, its grant, and its interest edits; `the_rules_agree_on_every_script` keeps the full script, with a second origin, restarts, and collection. Costs come from the mesh (`Cost <- TCost`), and the timer is impatient (`Patient = FALSE`), since the sweeps hold messages past it.

A failure leaves its trace in the `.tmp/route-trace.*` directory the run names: a deadlock is a step the model cannot take, and `Agrees` breaks where its messages or state differ, in a state whose `step` names what the router did and whose `route`, `shown`, and `q` show what the model did. Fix the side that is wrong, as Reading a failure says; a broken rule on an accepted trace is a finding in the rule.

## Reading a failure

Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the router with `Sim::with(Rule::PATH, nodes)` (`deliver`, `disconnect`, `connect`, `announce`, `wait`), one call per step, except a timer. `Lift(n, x)` fires one timer on one relay, where `wait(HOP)` moves every relay's clock by one hop of delay: call it until that doubt lifts, and where another doubt lifted with it, the replay took a different path, which is no defect in the model.
Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the router with `Sim::with(Rule::PATH, nodes)` (`deliver`, `disconnect`, `connect`, `announce`, `wait`), one call per step, except a timer. `Timeout(n, X, H)` fires timers on one relay, where `wait(HOP)` moves every relay's clock by one hop of delay: call it until those timers fire, and where others fired with them, the replay took a different path, which is no defect in the model.

## Coverage

Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped path router in [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622). Run `git log 31b215b11.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
Sync commit: `12537d1c2272c39db154fbd210bf60b8f438da4d`, the merge of the scoped path router ([moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)), which trace validation checked the model against ([moq-dev/moq#8733](https://github.com/moq-dev/moq/pull/8733)). Run `git log 12537d1c2.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since; `test route-trace` fails when a change there takes a step the model does not. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Timeout`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.

| Code | Model |
| --- | --- |
| `rs/moq-net/src/route/path.rs` `Router::run`, `Router::select`, `Router::retain` | `Select` plans live routes and their reflected sources before `SelectKey` decides holds. |
| `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` |
| `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted` |
| `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` |
| `rs/moq-net/src/route/path.rs` `Router::want`, `Router::advertise`, `Router::report` | `Want`, `Advertise`, `Report` |
| `rs/moq-net/src/route/path.rs` `Router::want`, `Router::meets`, `Router::advertise`, `Router::end_sent`, `Router::report` | `Want`, `Meets`, `Withdraw`, `Advertise`, `Report`, `Collect` (an ended stream forgets what it reflected) |
| `rs/moq-net/src/route/path.rs` `Router::route`, `Router::batch`, `Router::dead`, `Router::fail`, `Router::disconnect`, `Router::connect`, `Router::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` |
| `rs/moq-net/src/route/path.rs` `Router::collect_doubt`, `Router::timeout` | `Collect`, `Lift`, `Expire`: each dead hop's own timer retires it independently of later failures of the same node; reporting an already retained hop again does not restart its expiry. |
| `rs/moq-net/src/route/path.rs` `Router::collect_doubt`, `Router::timeout`, `Router::due` | `Collect`, `Timeout`: each dead hop's own timer retires it independently of later failures of the same node; reporting an already retained hop again does not restart its expiry. |
| `rs/moq-net/src/route/path.rs` `Router::know`, `Router::shown_now`, `Router::show` | `Know`, `ShownNow` |
| `rs/moq-net/src/route/reader.rs` `Reader::apply` | `Events` |
| `rs/moq-net/src/route/tests/mod.rs` `Sim::deliver` | `Deliver` |
Expand All @@ -111,4 +117,4 @@ Concurrent disjoint regions of one permit need the
[interest set](/quest/m2/cluster-routes/interest-set.md), including per-prefix
SYNCED; neither this model nor the current scoped sweep covers them.

One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/paths.rs` prototypes; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, stream generations or cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn.
One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/paths.rs` prototypes; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, stream generations or cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. Every claim change changes a claim: the router's revision bump with no change is outside the model. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn.
5 changes: 3 additions & 2 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -359,12 +359,13 @@ jobs:
# viewer: diff-independent, since a break can arrive through the relay,
# js/net, or the wire between them.
# `rs quic-sim` sweeps moq-quic's simulated workloads from a fresh seed; PRs run a fixed 64.
# It also runs the bandwidth probe's acceptance suite, of which PRs run the short cases.
# `rs rung-disable` proves FFI simulcast disable and recovery on a shaped BBR3 uplink.
# `test resume-model` sweeps resume against its model from a fresh seed; PRs run a fixed 64.
# `test route-key-model` proves the route key proof with Tamarin, and that each variant breaks what it names.
# `test route-sweep` checks random lite-07 meshes against the route rules from a fresh seed; PRs run a fixed 8.
# It also runs the router's handoff sweeps over 20,000 fixed scenarios; PRs run 150.
# It also runs the bandwidth probe's acceptance suite, of which PRs run the short cases.
# `test route-trace` checks with TLC that 150 runs of those sweeps, from a fresh window, are behaviours of the route model.
# `test drill impaired` runs the relay drills over a freshly seeded shaper; PRs run loopback.
# `rs cli-export` runs moq-cli's export process tests, which take tens of seconds of real time.
# `rs hls-conformance` probes every segment moq-hls serves with ffprobe, against its playlists and manifest.
Expand All @@ -379,7 +380,7 @@ jobs:
# `rs v4l2` streams moq-video's V4L2 capture from the kernel's vivid
# virtual camera, a module load no pull request job should make.
# `js worklet-matrix` loads `?worklet` output through the bundlers pinned in test/worklet.
recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test drain", "test json-properties", "test resume-model", "test route-key-model", "test route-sweep", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"]
recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test drain", "test json-properties", "test resume-model", "test route-key-model", "test route-sweep", "test route-trace", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"]
include:
# Miri interprets kio's and moq-net's in-memory unsafe; too slow for the merge gate.
- recipe: rs miri
Expand Down
Loading