Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 53 additions & 15 deletions quest/m0/ci/nightly-relay-session-hour.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [S] Nightly's relay session benchmarks fit their hour
# [M] Nightly's relay session benchmarks fit their hour

## Goal

Expand All @@ -9,18 +9,56 @@ regression gate.

## Plan

The job was cancelled at the timeout on the 2026-11-08, 11-09, and 11-10
scheduled runs. Unlike the other cancelled jobs it does not hang:
The job does not hang and the sweep did not grow: one cell's teardown went
super-linear. In
[the 11-10 run](https://github.com/moq-dev/moq/actions/runs/42842827718/job/129922278379)
was still finishing `relay_session_join_broadcasts` cells seconds before the
cancel. The scheduled runs from 11-01 to 11-05 failed inside the workload after
14 to 20 minutes, the
[11-01 job](https://github.com/moq-dev/moq/actions/runs/41313466339/job/124868517800)
with `Evicted`. The
[11-06](https://github.com/moq-dev/moq/actions/runs/42236107954) and
[11-07](https://github.com/moq-dev/moq/actions/runs/42377867523) runs
completed the sweep in about 25 and 23 minutes (their jobs failed later, at
`Fail on paired regressions`). Find what grew between those sweeps and the
60-minute cancellations from 11-08, then trim it or split the group across
matrix entries, as the Rust benchmark tests did. The sweep still covers both
fan-out axes.
each pass (this commit, then the base) spent about 22 minutes between the
`relay_session_delivery_prefixes/moq-lite-07-wip/broadcasts=1025/follow=1024`
cell and the next one; on
[11-07](https://github.com/moq-dev/moq/actions/runs/42377867523) that gap was
Comment on lines +12 to +18

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This baseline predates lite-07's move onto the route layer: the 11-07 Nightly head is b5805859a (run created 13:38Z on 11-07), while #8023 "lite-07 routes through the route layer" merged at 20:53Z that day, and #8352 (11-10 00:11Z) moved the bench runtime per cell so a cell's teardown is now charged before the next one starts. Read together with "the sweep did not grow", the gap looks like a regression in existing code, when it is the route layer landing plus per-cell teardown. Naming those two settles the "find what grew" question the previous text asked and keeps a later reader from hunting a regression commit that never existed.

Technical details
# The 11-07 comparison is a pre-route-layer baseline

## Affected sites
- `quest/m0/ci/nightly-relay-session-hour.md:12` — "the sweep did not grow: one cell's teardown went super-linear" reads as a regression in pre-existing code.
- `quest/m0/ci/nightly-relay-session-hour.md:18` — the 11-07 datum is offered as the before, but that build had no lite-07 routing at all.

## Evidence
- Nightly `runs/42377867523` created 2026-11-07T13:38:12Z at `b5805859a`.
- `ce4b849a1` "feat(net)!: lite-07 routes through the route layer (#8023)" merged 2026-11-07T20:53:29Z.
- `d9800c1a6` "fix(net): the session bench gives each cell its own runtime and a proxying relay (#8352)" merged 2026-11-10T00:11:08Z, moving `let rt = runtime()` inside each cell and so charging `retire`'s `drop(built)` per cell.
- The 11-10 gap and the 60-minute cancel match the per-cell `retire` wait (`rs/moq-net/benches/session/mod.rs:436`).

## Required outcome
- The plan says when the cost arrived (the route layer for lite-07 announce streams) and what makes Nightly charge it per cell, instead of implying a route-layer regression between 11-07 and 11-08.

under a second. A local run of
`cargo bench -p moq-relay --features test-support --bench session -- --quick relay_session_delivery_prefixes/`
reproduces it: one thread spins at 100% for over 10 minutes after the lite-07
`follow=1024` cell, while the next cell alone finishes in 2 seconds.

The spinning thread is the cell's `retire`, dropping the viewer's lite-07
session driver. Each of its 1024 `AnnouncePrefix` streams drops a
`route::node::Stream` (`rs/moq-net/src/route/node.rs` `Drop`), which calls
`disconnect`, then `update`, `recompute`, `prune` (`route/mod.rs`), and
`View::restrict_all` over every region still certified
(`route/view.rs`). `restrict_all` merges one restriction per region, and each
`merge_claims` costs claims x floors in `as_of` plus floors squared in its
stale check and floor `retain`. With about 1024 regions, claims, and floors,
that is about 1e9 steps per closed prefix and cubic or worse over the session.
A relay pays this whenever a lite-07 peer holding many announce prefixes
leaves, so it is a relay CPU cost a client controls, not only a benchmark
problem.

✅ Decided 2026-11-12 (maintainer, on
[moq-dev/moq#8536](https://github.com/moq-dev/moq/pull/8536)): fix the route
layer's teardown to about linear (single-pass `restrict_all`, indexed floors,
batch disconnect), verified with `/verify-route` (rejected: temporarily drop
the lite-07 `follow=1024` Nightly cell, which hides a relay cost a client
controls).

[Routes](/quest/m2/cluster-routing/routes/README.md) item 27 (decided
2026-11-11) replaces Babel's distance, the seqno, ROUTE_REQUEST, and the
ROUTE_END echo with per-scope paths, and
[the router](/quest/m2/cluster-routing/routes/scope.md) that builds it
deletes `prune` and the per-region floors this teardown spends its time in.
This fix still lands first, unblocked: Nightly loses its paired gate every
night until it does, the router waits behind its model, and the batch
disconnect carries over to the rebuild. Spend least on what the rebuild
deletes: prefer the batch disconnect and single-pass `restrict_all` over
indexing the per-region floors, and add the latter only if the former leave
the cell super-linear.

Make the route layer's teardown of a neighbor's prefixes cost about linear in
what it held: for example, let `restrict_all` build one view from all regions
in a single pass instead of merging per region, index floors so `as_of` and
the floor `retain` stop scanning every floor, and disconnect a session's
prefixes as one batch. Keep `/verify-route` passing. Then confirm the Relay
session group finishes inside the hour (the 11-06 and 11-07 sweeps took
about 25 minutes) without dropping the `follow=1024` cell, which is the case
that exposed this. Add a route-layer benchmark sweeping held prefixes on

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rs/moq-net/benches/routes.rs already sweeps route work over claims x streams (watched, unwatched, received, lagging, confirmed), so a teardown case belongs there rather than in a new bench: drop N announce streams over a table of C claims, over the same two axes. Naming it keeps the existing sweep the primary reproducer instead of leaving the Nightly cell alone with the cost.

both axes if the session cell does not isolate the teardown cost.
2 changes: 2 additions & 0 deletions quest/m2/cluster-routing/routes/scope.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ Work:
publisher's routes as the model has it. Keep the reader's view a function
of what the router holds, computed incrementally the way `show` is today,
with the test build comparing it against the whole computation.
- Keep a session's teardown about linear in the prefixes it held, as
[the relay session hour](/quest/m0/ci/nightly-relay-session-hour.md) makes it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completed quests are deleted from the tree (for example #8352 removed quest/m0/nightly/session-bench-handshake.md and its entry in quest/m0/nightly/README.md), and this fix lands first while the router waits behind its model, so this link will dangle by the time the work item is read. Cite the fix's PR or commit once it lands, or state the requirement without the link.

- Remove what #8487 added to work around the old rule: `recompute`'s `prune`
switch, which skips pruning on a change of hops or RTT alone. Keep the
qualified-key test in `infeasible` unless the new rule makes it moot, and
Expand Down