Skip to content

fix(moq-net): a resumed reader takes groups landing late on a drained copy - #9103

Merged
kixelated merged 5 commits into
mainfrom
quest/m1/precut-fixes/resume-range-reader
Nov 18, 2026
Merged

kixelated merged 5 commits into
mainfrom
quest/m1/precut-fixes/resume-range-reader

Conversation

@kixelated

@kixelated kixelated commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Claims quest/m1/precut-fixes/resume-range-reader.md.

Problem

The resume model's sweep fails on main at seed 3 (MOQ_RESUME_SEED=3 MOQ_RESUME_CASES=50000): a reader resumed across routes misses a group of the logical track. The quest expected range logic in Reader::window. The failure is not range-specific. A copy whose channel ran out cleanly (done = Some(Ok(()))) stayed done for the rest of the read. A received copy whose session settled its end at its live edge is complete at once, yet groups still in flight below that end may land on it afterwards (create_group below the end is allowed). If a replaced copy holds the read open meanwhile, those groups are never read. The unranged regression in resume_model.rs shows the same loss without a range.

TLC found a second ordering on the first fix: a late group under the serving copy's resume point is lost after the backfill that asked for it drained the complete track and was let go.

Approach

  • track::Consumer::arrived / poll_arrived (crate-private) count the groups that arrived on a copy, from TrackState::next_arrival, and wake a waiter once another lands.
  • resume::Copy records drained, the arrivals when its channel last ran out cleanly. poll_lowest reopens a drained copy once a group arrives after that, while the copy's span is not yet spent, and rechecks right after draining so a group landing during the read is not missed.
  • A late group on the serving copy under its floor marks the reader resumed again, so backfill_gaps asks its route for it once more.
  • JS mirrors it: a hooks.arrived count kept by track.ts, Copy.drained, and #reopen() before #backfillGaps in tryGroup.
  • ResumeReader.tla lands groups on a track after its session settled (Settle), reopens a drained copy in Scan, re-runs backfill on a late group (Pass), and CleanEndIsComplete now also requires every group each live track carried (Carried).

Impact

  • No public API or wire changes.
  • Behavior: a resumed reader now delivers groups that land late on a copy that already ran out, instead of losing them.

Alternatives

  • Never marking a received copy done while a replaced copy holds the read open: that keeps the reader polling a copy that may never deliver again, and its end still has to be learned. Reopening on an arrival costs one counter compare per drained copy per poll.
  • Extending the TLA+ model with a ranged reader, as the quest planned: the bug was not range logic, so the model gained late landings after a settled end instead.

Validation

  • Seed 3 at 50000 cases reproduced the failure on main at 4f3ba33. Two new regressions() cases (ranged and unranged) and two unit tests (a_late_group_on_a_drained_serving_copy_is_read, a_late_group_under_the_resume_point_is_backfilled_again) fail without the fix and pass with it.
  • TLC without the reopen reports a CleanEndIsComplete violation within a second. With the fix it passes: main config 19,871,459 distinct states in 4m53s, ThreeRoutes.cfg 7,384,328 in 1m12s.
  • Rust sweeps at 30000 cases pass on seeds 3, 1, 2, 7, 11, and 0x6d6f71. All 3328 moq-net lib tests pass.
  • JS: the two mirrored unit tests fail without the fix and pass with it. The resume, resume model regressions, and track tests (413) pass.
  • quest gate just check passes.
  • The JS model's random sweep still fails on fresh seeds with group 0 outside ranges, with or without this PR. moq-dev/moq#9027 introduced it (it passes on its parent), so Nightly's test/resume-model.sh fails on the JS side until resume-js-window lands.

Caused by

  • moq-dev/moq#4741 - a cleanly drained copy stayed done for the rest of the read, losing groups that landed late (found by check: resume_model sweep, seed 3)

Decisions

  • The quest's premise (range logic in Reader::window) was wrong. This PR fixes the actual loss and leaves Reader::window unchanged.
  • The two surviving Reader::window mutants the quest planned to kill move to a follow-up, resume-window-mutants, with the analysis of each.
  • The JS window regression from @moq/net addresses groups by start #9027 is a separate quest in this epic, resume-js-window, not fixed here: it needs Rust's "the window decides" filter with its owes_below guard ported into JS #lowest.

Follow-ups

Models

Size: M
Implementer: Claude, claude-opus-5-5, high effort

(Written by Claude Opus 5.5 with high)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 86fb205

  • Base: origin/main at b6c248c

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1f06e-aaea-7880-b1a1-19667f95ca41, round 2

  • Score: 10/10, Both findings are fixed, and inspection of the incremental changes found no new issues; no builds or tests were run.

  • Rounds: 2; first round 2 (medium 1, low 1); findings 2 (medium 1, low 1): fixed 2, refuted 0, open 0

  • Findings: none

  • The last round's findings, at b723596:

    • Fixed: medium, js/net/src/resume.ts:348: Carry the arrival snapshot inside the drained state; JS and Rust now carry the arrival snapshot inside the clean done variant, and reopening reads it directly at js/net/src/resume.ts:997 and rs/moq-net/src/model/resume.rs:1697.
    • Fixed: low, .claude/skills/verify-resume/SKILL.md:43: Anchor model coverage to the PR base; .claude/skills/verify-resume/SKILL.md:43 now anchors both references to the updated PR base 8c939bc9d05e and identifies this PR.

kixelated and others added 3 commits November 18, 2026 12:05
… copy

A copy whose channel ran out cleanly stayed done for the rest of the read, so a
group still in flight below the end its session settled was lost when it landed
while a replaced copy held the read open. A drained copy now reopens once a
group arrives on it after it ran out. A late group under the serving copy's
resume point asks its route again, since the backfill that wanted it may have
run out and been let go.

The resume TLA+ model lands groups after a session settles its end (Settle) and
checks a clean end carried every group each live track landed. JS mirrors the
fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title A ranged reader resumed across routes gets exactly the groups in its range fix(moq-net): a resumed reader takes groups landing late on a drained copy Nov 18, 2026
kixelated and others added 2 commits November 18, 2026 12:57
A copy's clean drain now holds the arrivals it saw in its done state, in Rust
and JS, instead of a separate field beside it. The resume model's sync commit
names the PR's base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 18, 2026 21:06
@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 8 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 03fb6c30-e73e-410c-a24b-5acd66c0c95f
Commits

Reviewing files that changed from the base of the PR and between 8c939bc and 86fb205.

Files selected for processing: 13

  • .claude/skills/verify-resume/SKILL.md
  • js/net/src/internal.ts
  • js/net/src/resume.model.test.ts
  • js/net/src/resume.test.ts
  • js/net/src/resume.ts
  • js/net/src/track.ts
  • quest/m0/nightly/resume-window-mutants.md
  • quest/m1/precut-fixes/resume-js-window.md
  • quest/m1/precut-fixes/resume-range-reader.md
  • rs/moq-net/src/model/resume.rs
  • rs/moq-net/src/model/resume_model.rs
  • rs/moq-net/src/model/track.rs
  • rs/moq-net/verify/resume/ResumeReader.tla
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated

Copy link
Copy Markdown
Collaborator Author

Decisions from the maintainer:

  • ✅ Merge once the required checks and Pullfrog pass.
  • ✅ Keep quest/m1/precut-fixes/resume-js-window.md, gating the cut (rejected: outside the gate, drop).
  • ✅ Keep quest/m0/nightly/resume-window-mutants.md (rejected: drop).

(Written by Claude Opus 5.5 with high)

@kixelated
kixelated enabled auto-merge November 18, 2026 21:13

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. The fix targets the real cause, a cleanly drained copy that stayed done for the rest of the read, rather than the quest's guessed range logic. It reopens on a plain arrival counter, so a drained copy costs one compare per poll.

Reviewed changes

I reviewed the full diff: the Rust and JS reader changes, the new arrival hooks, the TLA+ model extension, the regressions, and the quest bookkeeping.

  • Arrival count on a clean drain — Copy::done now carries Result<usize> in Rust and Error | number in JS. It records TrackState::next_arrival (or the JS sink's arrived) as of the drain. track::Consumer::arrived and poll_arrived are crate-private and park the reader's idle waiter, so a late landing wakes the read.
  • Reopen in poll_lowest — a drained copy that is not spent and has new arrivals is read again. The re-drain loop compares against a snapshot taken before the read, so a group landing during the read is not missed. Each extra pass needs a fresh arrival, so the loop is bounded.
  • Re-backfill on the serving copy — when the serving copy (not replaced, has a floor, current generation) reopens, the reader is marked resumed again. Backfill copies are replaced: true, so they never trigger this. Returning None hands control back to the caller's loop, which reruns backfill_gaps with the same (None, []) value that sync already uses when ranges change.
  • JS mirror — #reopen() runs before #backfillGaps in tryGroup. I also probed the separate async clean-end path (sub.closed.then → #conclude) with a fresh serving route and a late group under its floor:
    • If the group lands before the replaced copy fails, it is delivered.
    • If it lands after, the read ends cleanly, which Carried allows.
  • ResumeReader.tla — adds Settle (late landings after a settled end), reopening in Scan, a recursive Pass for re-backfill, and a stronger CleanEndIsComplete (Carried).
  • Regressions and tests — two regressions() cases (ranged and unranged) and two unit tests per language. The Copy::done pattern matches at end_error, bound_at_end and reopen are updated to Some(Ok(_)).

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit b728f0a Nov 18, 2026
6 checks passed
@kixelated
kixelated deleted the quest/m1/precut-fixes/resume-range-reader branch November 18, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant