Skip to content

fix(moq-net): a dead instance held unavailable never outranks a live one - #9156

Merged
kixelated merged 2 commits into
mainfrom
quest/m2/cluster-routes/dead-instance-flip
Nov 19, 2026
Merged

kixelated merged 2 commits into
mainfrom
quest/m2/cluster-routes/dead-instance-flip

Conversation

@kixelated

@kixelated kixelated commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m2/cluster-routes/dead-instance-flip.md and deletes it.

Problem

A reader of a path whose publisher restarted (old instance B, new instance C) could see the path go C → B → C. moq-dev/moq#9108 folds that flip into an Update when the reader doesn't poll in between. A reader that polls mid-flip still sees B, and the sweep's NoRegression rule would flag it.

The quest assumed B won through a stale path. Tracing the nightly case (MOQ_ROUTE_SEED=255657690 MOQ_ROUTE_CASE=41) showed something else. At relay 0, when a route change moves C's closest way from session A to session B, the route node demotes the old way on A's queue (primary: false) and serves the new one on B's queue. The two queues apply separately. In between, C's only entry is non-primary. route_order ranked !lane.primary ahead of draining || !available, so B won: B is unavailable (its relay lost the publisher's session and holds the claim for 800 ms), but it still counts as primary. C won back once B's queue applied.

Fix

  • route_order ranks the closest way below availability, as its own doc comment already said. The route layer marks every way to one broadcast available, or none, so primary still orders an instance's own ways. A live instance's other way now outranks a dead instance held unavailable.
  • Draft (lite-07-wip, Resolution): the receiver prefers a claim that is neither draining nor Unavailable, and the changelog says so. Before, the draft ranked by drain, priority, and cost only, so a cheaper Unavailable claim of a dead instance beat a live one. An Unavailable claim serves nothing (its sender fails requests with UNROUTABLE), so no case needs it to win. JS (compareRoutes) and Rust already ranked unavailable with draining; the draft was the outlier.
  • doc/concept/identity.md lists an unavailable route with a draining one. Stale docs on NodeFilter::Any and route_order now describe what primary means since feat(moq-net)!: lite-07 sessions run the new route rule #8987.

Impact

  • Public API: none.
  • Wire: no encoding change. lite-07-wip's Resolution rule now ranks an Unavailable claim below every available one, which matches both implementations. lite-03 to lite-06 and moq-transport are unchanged.

Decisions

  • Stop the flip at the rule, not only at the reader. This is the quest's recommendation. The route rule itself let the dead instance win, so the fix belongs there.
  • Route models. /verify-failover follows one track through one instance's routes, and /verify-warm models warm pulls. Neither has two instances at one path, or availability across instances, so neither can show a case where B must win. The argument stands without them: per the draft, an Unavailable claim's sender refuses every request for it, so ranking it above an available claim only routes requests to a refusal. The differential sweep below measures the effect.
  • moq-dev/moq#9108 is not made redundant. With this fix alone, the nightly case passes without fix(moq-net): a restart away and back before the next poll is an update #9108. fix(moq-net): a restart away and back before the next poll is an update #9108 still covers a restart away and back between two live instances (its own unit test is that case, two live instances whose ranks flip), which this PR doesn't touch. Nothing in fix(moq-net): a restart away and back before the next poll is an update #9108 should be removed.
  • Interop not run. Neither encoding nor negotiation changed, and the ranking change only matters with two instances at one path, which the interop lanes don't build.

Validation

  • New unit test a_dead_instance_never_outranks_a_live_one_between_its_ways: a cheaper dead instance held unavailable, and a live one whose closest way is demoted. It fails on main (the cursor restarts to the dead instance) and passes with the fix.
  • Nightly case MOQ_ROUTE_SEED=255657690 MOQ_ROUTE_CASE=41 (release, as route-sweep runs it): fails on main with RESTART to the same instance and passes with this fix, without fix(moq-net): a restart away and back before the next poll is an update #9108.
  • Full 500-case sweeps, seeds 255657690 and 7: pass.
  • Differential, second way: I instrumented every announce cursor (temporarily, not committed) to log each change of instance. Over 500 cases of seed 255657690 I counted the cursors that went back to an instance they had left, or onto an unavailable one:
  • just check passes (10,724 Rust tests). The first run hit four 60 s timeouts in util::idle and transport::tests::stop_discards_the_buffer under load. Those passed alone in 0.03 s and in a full rerun, and this change doesn't touch them.

Not covered

  • Both instances unavailable at once: if the live C also loses every route within B's 800 ms hold, the two rank by priority and cost, and the cheaper dead B can show until its hold ends. Nothing serves either one, and instances carry no order that would say which is newer, so I left it. It needs a second failure inside the hold window.
  • A lite-06 peer's announce cursor is split-horizon too. It can name B, held unavailable, to the peer the live C came from. lite-06 is outside this quest.

Caused by

  • moq-dev/moq#8987 - folded the route layer's selected flag into primary but kept primary where the old "narrowest route" meaning ranked, ahead of availability. A dead instance held unavailable then outranked a live instance's demoted way (found by check: nightly route-sweep and benchmark_route_mesh).

Follow-ups

None.

Models

Size: M
Implementer: Claude, claude-opus-5-5, unknown effort (arm B)

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 45ad829

  • Base: origin/main at 61e4a38

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1f307-9fbf-76b3-9863-6d584cbe4b4b, round 1

  • Score: 9/10, The shared ordering change prevents unavailable primary routes from displacing live alternatives while preserving specificity, epoch precedence, and existing JS behavior; no actionable regressions were found, though builds and tests were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 2 commits November 18, 2026 22:39
The origin ranked the route layer's closest way (Lane::primary) ahead of
availability, so while a route change moved a live instance's closest way to
another session, a restarted publisher's dead instance, held unavailable,
won the path back for a moment: a reader saw C -> B -> C. The closest way now
ranks below availability, where the doc already put it, and the lite-07 draft
ranks an Unavailable claim with a draining one, as JS already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title A dead instance's stale path never wins a lite-07 route fix(moq-net): a dead instance held unavailable never outranks a live one Nov 19, 2026
@kixelated
kixelated marked this pull request as ready for review November 19, 2026 09:01
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b45a5c42-8ac8-4d66-babb-19b3a2af201b
Commits

Reviewing files that changed from the base of the PR and between fded924 and 45ad829.

Files selected for processing: 4

  • doc/concept/identity.md
  • drafts/draft-lcurley-moq-lite.md
  • quest/m2/cluster-routes/dead-instance-flip.md
  • rs/moq-net/src/model/origin.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated
kixelated enabled auto-merge November 19, 2026 09:02

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. It fixes the cause in route_order: primary ranked ahead of availability, so a dead instance's held claim beat a live instance whose closest way is demoted for a moment. The fix doesn't hide the flip at the reader, and the new order matches the function's own doc, JS compareRoutes, and now the draft.

Reviewed changes

Reviewed the full diff at 45ad829: the route_order key change, the matching draft and doc updates, and the new regression test.

  • route_order reorder: !entry.lane.primary now ranks after draining || !available. This holds because route/node.rs relane marks every way available: true or flips them all to false together. Router::closest already ranks by claim_order with drain first, so primary and non-draining rarely disagree. Where they do, a live, non-draining way should win.
  • Draft lite-07-wip Resolution: the receiver now prefers a claim that is neither draining nor Unavailable, and the changelog says so. This matches the Rust and JS ranking. The encoding is unchanged.
  • Docs: identity.md step 3, NodeFilter::Any, and the route_order doc comment now describe primary as each broadcast's closest way.
  • Regression test: a cheaper dead instance is held unavailable while a live instance's way is demoted. The cursor stays on the live instance and requests go to its way. The test fails on main by construction.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 6dceaed Nov 19, 2026
10 checks passed
@kixelated
kixelated deleted the quest/m2/cluster-routes/dead-instance-flip branch November 19, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant