Skip to content

fix(net): fail an idle copy whose route answers behind its cache - #8041

Merged
kixelated merged 10 commits into
moq-dev:mainfrom
michaelstuurman:fix/answer-behind-cache
Nov 8, 2026
Merged

kixelated merged 10 commits into
moq-dev:mainfrom
michaelstuurman:fix/answer-behind-cache

Conversation

@michaelstuurman

@michaelstuurman michaelstuurman commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem
An idle track copy that resumed on a route answer with no Largest hid its cache behind a live floor, while claim_sequence still refused the feed's re-sent groups as Duplicate. Live readers stalled until the feed passed the old newest group, forever for a one-group catalog.

Approach
Fail loud, as decided in #7973. When an idle copy resumes and its route's answer (lite-07 SUBSCRIBE_START or moq-transport SUBSCRIBE_OK) is behind its cache, set_live/set_live_ahead drop the cache and return Error::MalformedTrack. Behind means no Largest while the copy held an object, or a Largest before the newest object it held. What it held is snapshotted when the copy goes idle, after the groups in flight are displaced, so groups the next feed resends before its answer is judged do not count; the object is the moq-transport object ID where the group carries one, else the frame index. An empty group holds no object, so it is skipped. The subscribers fail the copy through their existing teardown: readers get the stream-scoped MALFORMED_TRACK, and the next subscribe builds a fresh copy that accepts the feed from group 0.

Our moq-transport publisher now answers the last held object as Largest when its newest group cannot be read (the _ arm of cached_edge in ietf/publisher.rs), instead of no Largest. That happens on a relay whose upstream reset its newest group while its copy stayed live; before, the downstream relay's lingering copy over the earlier groups would have failed with MALFORMED_TRACK. next stays past the newest group (latest + 1, object 0), so nothing strands. live_edge still prefers an upstream report only when it is past this cached Largest, and a Relative filter now counts from the held Largest rather than the unreadable group.

Two choices beyond the quest:

  • The cache is dropped before the copy fails. Otherwise an abort lets readers drain the stale groups first.
  • Lite before 07 has no Largest. There the first group header stands in (set_live_from) and is not judged, since a resend of the cached head would read as behind.

Impact

  • No public API change. A relay fails a resumed copy with MALFORMED_TRACK where it used to stall its readers.
  • Wire: a moq-transport publisher whose newest group is unreadable now sends the last held object as SUBSCRIBE_OK / TRACK_STATUS_OK Largest instead of omitting it, and a FETCH relative to Largest is served there instead of refused as unsupported.

Decisions

  • ✅ Fix the publisher's missing Largest in this PR (rejected: a follow-up). Maintainer, 2026-11-07 via /quest-complete: report the last held object, as the arm above already does through largest_before, keeping next past the newest group.

Alternatives

  • Trust the cache, or drop it silently: a cache that disagrees with its route is likely another track.
  • Judge only a missing Largest: misses a Largest before the cache.

Caused by

  • moq-dev/moq#4741 - hid a cache that a no-Largest answer disowned behind a live floor, while re-sent groups were still refused as duplicates.

Follow-ups

Validation

  • just ci via quest gate on 874bc8a: passed (10996 tests); again on f342773 after the review fixes (10998 tests). It first caught a_draft22_fetch_past_the_largest_object_is_refused, which expected an aborted newest group to leave Largest unknown; it now expects INVALID_RANGE past the held Largest.
  • The new relay-chain test (a_reset_newest_group_answers_the_group_before_*) fails with the publisher change reverted (the upper relay answers no Largest and the lingering copy fails) and passes with it; the unit test an_aborted_newest_group_leaves_the_earlier_largest likewise.

Models

Size: S
Implementer: Claude Code, Claude Opus 5.5, unknown effort
Implementer: Claude Code, Claude Opus 5.5, medium effort

Review

  • Reviewer: Codex, gpt-6.1-sol, medium effort (size M by default: the branch is on a fork, so quest review could not read this record and printed record skipped; this section is written by hand).
  • Session 01a1b86f-f1cb-7211-a304-2c32cffe4d5c, following session 01a1b85d-f3cb-7d62-a8b3-3a73b7dedbfd round 1.
  • Reviewed head f342773 against origin/main fe00d62: no findings.
  • Score 9/10: "Both reported defects are addressed with focused regression tests."
  • Round 1 (head 874bc8a, score 5/10) found two, both fixed in 4b508bb:
    • high rs/moq-net/src/model/track.rs:2901 Judge only content retained from the previous feed: on moq-transport the answer is judged after the next feed's head arrives, so a resent in-flight group read as held past the answer. Fixed: set_idle snapshots the held position after displacing groups in flight (Feed::Idle { held }), and judge() uses it.
    • medium rs/moq-net/src/model/track.rs:2907 Compare IETF Largest against object IDs rather than frame counts. Fixed: group::Handle::last_object() reads the wire's last object ID when present, as group::Consumer::last_object does for the publisher's answer.
  • Earlier: the contributor's Claude Code /code-review (Claude Opus 5.5, high) on a450141 fixed an empty newest group read as cached (b911760); merging the set_live variants and replacing eviction through evict_group were left for the reviewer and not raised by Codex.

Closes #7973. Implements quest/m1/answer-behind-cache.md, removed here.

(Written with Claude Opus 5.5)
(Written by Claude Opus 5.5 with medium)

michaelstuurman and others added 3 commits November 6, 2026 18:05
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@michaelstuurman

Copy link
Copy Markdown
Contributor Author

@kixelated The body lists one open decision: our moq-transport publisher answers with no Largest when its newest group is unreadable, which this change turns into a failure downstream. Fix it here or as a follow-up? Checked end to end against moqxr v0.4.15 (drafts 18 and 22): the resumed catalog fetch now fails with MALFORMED_TRACK instead of hanging. The next fresh copy sometimes still misses group 0, from a separate relay bug on a reused track alias: #8044.

(Written by Claude Opus 5.5)

kixelated and others added 4 commits November 7, 2026 15:17
…able

A moq-transport publisher whose newest group cannot be read, such as one its
upstream reset, answered with no Largest while it still held earlier groups.
With the behind-the-cache check, that failed a downstream relay's lingering
copy with MALFORMED_TRACK. Report the last held object instead, keeping the
subscription past the newest group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efused

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bject ID

The answer was judged against the cache at the time it arrived, which on
moq-transport already holds the next feed's resent groups, and by frame count,
which is not a sparse group's newest object ID. Snapshot the held position at
idle, using the wire's object ID where the group has one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 7, 2026 23:56
@coderabbitai

coderabbitai Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 17 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ecedad7e-4529-4244-9a12-0d2e54cf8c75
Commits

Reviewing files that changed from the base of the PR and between 7a272fc and 33ca70f.

Files selected for processing: 9

  • quest/m1/README.md
  • quest/m1/answer-behind-cache.md
  • rs/moq-net/src/ietf/publisher.rs
  • rs/moq-net/src/ietf/subscriber.rs
  • rs/moq-net/src/lite/subscriber.rs
  • rs/moq-net/src/model/group.rs
  • rs/moq-net/src/model/track.rs
  • rs/moq-net/tests/integration/answer_behind_cache.rs
  • rs/moq-net/tests/integration/main.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated

Copy link
Copy Markdown
Collaborator

@michaelstuurman Thanks for checking this end to end. The maintainer's decision on the open question: ✅ fix it in this PR. I pushed it here, along with two fixes the cross-vendor review found:

  • Publisher Largest (518cf3b, 874bc8a): when the newest group can't be read, the _ arm of cached_edge now answers the last held object through largest_before, the same as the arm above it. next stays at (latest + 1, 0), so nothing strands. live_edge still takes an upstream report only when it's past that Largest. A draft-22 FETCH past it is now refused with INVALID_RANGE, and a_draft22_fetch_past_the_largest_object_is_refused was updated to expect that. Tests: an_aborted_newest_group_leaves_the_earlier_largest, plus a three-hop test, a_reset_newest_group_answers_the_group_before_ietf{18,22}. In that test, the upper relay stays live while its upstream resets the newest group, and the lower relay's lingering copy resumes. It fails with the publisher change reverted.
  • Judge against the ended feed (4b508bb): on moq-transport the answer is judged only after the next feed's head arrives, so a group that was in flight at idle and gets resent could read as held past the answer and fail a healthy resume. set_idle now snapshots the held position after displacing in-flight groups, and judge() uses that snapshot.
  • Object IDs (same commit): Largest names a moq-transport object ID, which isn't frame_count - 1 in a sparse group. The new group::Handle::last_object() reads the wire's last object ID when the group has one.

just ci passes, and the Codex review's last round found nothing. The PR is now ready for review.

(Written by Claude Opus 5.5 with medium)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed f342773.

No actionable findings. Snapshotting the ended feed's held position after removing open groups avoids judging an answer against data that races ahead of it. Sparse moq-transport groups use object IDs rather than frame counts, and older lite versions retain the separate, non-judging first-group path. Clearing the cache before returning MalformedTrack fits the fail-loud policy without draining stale content. The publisher's fallback Largest preserves held history while keeping the next-object floor beyond the unreadable newest group.

Verification limits: GitHub-only static review of all nine changed files and surrounding live-edge and subscriber teardown logic. No tests, builds, relay interoperability or timing reproduction run. The same-name publisher restart and reused-alias races remain the documented separate limitations. Rechecked open/non-draft state, full head/base and published reviews immediately before posting.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Idle-feed snapshot — Feed::Idle carries held, the newest object the ended feed left, taken after open groups are displaced so the next feed's resent groups cannot race the answer.
  • Answer judging — TrackState::judge fails a resumed copy with MalformedTrack on a missing Largest or one before held, evicting the cache first; set_live/set_live_ahead now report it and set_live_from keeps the lite <07 first-header path unjudged.
  • Object-level comparison — group::Handle::last_object uses the wire's object ID where a group carries one, matching how the IETF subscriber builds Position from Largest.Object.
  • Teardown on both wires — ServeEnd::Invalid aborts the copy on moq-transport, and the lite GiveBack path does the same without escalating the session.
  • Publisher fallback — cached_edge answers largest_before(latest) when the newest group is unreadable, keeping next past that group, so a downstream lingering copy over earlier groups resumes instead of failing; the draft-22 fetch test now expects INVALID_RANGE past the held Largest.
  • Tests — model tests for behind/current answers, empty newest groups, the displacement race, and sparse object IDs; integration tests for the empty answer failing the lingering copy and the reset-newest-group relay chain.

I also ran the new tests (cargo test -p moq-net --lib --features async, cargo test -p moq-net --features async --test integration answer_behind) and mutation-checked the publisher change: reverting the _ arm to largest: None makes both a_reset_newest_group_answers_the_group_before_* tests fail with MalformedTrack, so the test genuinely pins the fix.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed c178fcd, incrementally from f342773 on the unchanged base.

No new actionable findings. Gating the sparse-object regression on async matches the feature gate on group::Producer::set_wire, which the test calls. The other idle-copy regressions remain enabled without that feature, and the production answer/cache behavior is unchanged. This is a focused fix for the no-async test build rather than a weakening of the object-ID assertion where its wire setup is available.

Verification limits: GitHub-only static review of the one-line update, complete regression and set_wire feature gate. No feature-matrix builds or tests run. Rechecked open/non-draft state, full head/base and published reviews immediately before posting.

kixelated and others added 2 commits November 7, 2026 16:40
main made Hop crate-private, so the answer_behind_cache test takes the
public origin::Config::new(NonZeroU64) like the other integration tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated added this pull request to the merge queue Nov 8, 2026
Merged via the queue into moq-dev:main with commit 544f144 Nov 8, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A lingering track refuses a restarted publisher's groups as duplicates

2 participants