Skip to content

fix(net): a relay copy fails loud when upstream's largest group goes backwards - #5057

Closed
Dryvnt wants to merge 3 commits into
moq-dev:mainfrom
Dryvnt:quest/m0/largest-regression
Closed

Dryvnt wants to merge 3 commits into
moq-dev:mainfrom
Dryvnt:quest/m0/largest-regression

Conversation

@Dryvnt

@Dryvnt Dryvnt commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Implements quest/m1/largest-regression.md (deleted here). Contributed by @Dryvnt; rebased onto main by the maintainer's agent.

Problem

A relay keeps a track's copy for the linger after its last reader leaves, and a returning reader re-subscribes it from the newest group it cached. Under a prefix claim the relay never sees a worker close an output, so when the worker serves the path again from group 0, the answer's largest group is below the cache. The copy took the answer as current and served the old output's group, then nothing until the new sequence passed it.

Approach

On versions whose answer carries the largest position (moq-lite 07, moq-transport), an idle copy on a route without an epoch takes a largest group below what it still caches as new content under the old name. It then:

A route with an epoch may be serving a replica that lags the copy, so its copy keeps waiting as before. moq-lite 05 and 06 carry no largest and keep waiting on the floor, as does an answer with no largest (a cold moq-transport relay).

Only a restart behind an unchanged route reaches a copy unannounced: since #5087 every route change, including a per-path winner change under a prefix pool, reaches downstream as a Restart that drops the cached copy. Closing the source is still needed, because #5087's sticky fronts let a new request join a front while its route entry still wins, and here the entry never changed.

Known false positive, accepted: an upstream relay answers from its own copy's cache rather than its upstream's largest, so a copy it re-created cold that holds only older groups under-reports. That needs the downstream copy to outlive the upstream's plus a concurrent lower-start subscriber upstream. It costs a spurious UNROUTABLE and a re-request, never stale data. Equal group numbers can't be told apart and still splice.

Rebase onto main

Tests

Mocked time. tests/largest_regression.rs restarts a claim worker's output within the linger on moq-lite 07 and moq-transport 14, 16 and 22: the returning reader gets UNROUTABLE, then the new output from group 0, never the old group. With the check disabled, every version gets the old group (re-verified on the rebased head). A same-epoch standby behind the cached copy keeps the copy and resumes it. A track.rs unit test covers the judgment and the withheld cache.

just check passes locally. just test interop --all passes except python -> c, which timed out under host load (load average over 100) and passes on rerun.

Impact

  • Public API: none. track::Producer::regresses and withhold_cache are crate-private.
  • Wire: no format change. A relay now resets a downstream subscription with UNROUTABLE in this case.

Alternatives

  • End the front on the copy's error instead of closing the source: a failed copy is the front's to replace, so it would re-query the same source and stall on the old floor.
  • Judge every route: a same-epoch standby that lags would cut viewers who should resume.

Decisions

  1. Relays further from the worker, which still hand a returning reader the old group:
    • ✅ A follow-up quest, quest/m1/refused-copy.md (maintainer, 2026-10-08: m1, beside its parents, not m0).
    • Extend this PR.
  2. The false positive from a relay under-reporting its largest:
    • ✅ Accept it with no follow-up (maintainer, 2026-10-08: ietf-cold-largest is not reopened).
    • Fold it into a relay-largest quest.
  3. A newcomer that requests the path after the source closes but before the front task retires the front (Codex, on a multithreaded runtime):
    • ✅ Accept it: it gets one error, as a request that joined just before the check would, never old groups. A reader that saw the error re-requests onto a fresh front, since it only sees the error through the front's verdict.
    • Track each front's serving source in the front table, so a request skips a front whose source closed.
  4. The error a regressed copy ends with:
    • ✅ Keep UNROUTABLE, as a superseded or failed epochless track gets.
    • A different error.

Follow-ups

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

@kixelated

Copy link
Copy Markdown
Collaborator

Rebase notes from the 2026-10-08 quest audit (#5058, #5063):

  • quest: audit the whole tree (2026-10-08) #5058 moved largest-regression and claim-epochs to m1: delete quest/m1/largest-regression.md and edit quest/m1/claim-epochs.md, not the m0 paths.
  • quest/m1/ietf-cold-largest.md is deleted on main: cold-relay Largest stays INVALID_RANGE (quest/m0/README.md). Decided 2026-10-08: it is not reopened, so accept the false-regression cost rather than relying on that quest.
  • Decided 2026-10-08: the new refused-copy quest goes in m1 beside its parents, not m0.

(Written by Claude Opus 5.5)

@kixelated
kixelated force-pushed the quest/m0/largest-regression branch from 8f1a4d8 to b655c54 Compare October 12, 2026 01:51
@kixelated

Copy link
Copy Markdown
Collaborator

Thanks @Dryvnt for this fix and the careful write-up. The regression judgment and the source-before-abort ordering held up unchanged through the rebase.

I rebased your two commits onto main (dropping the #5054 commits that merged), keeping your authorship:

(Written by Claude Opus 5.5)

@kixelated
kixelated force-pushed the quest/m0/largest-regression branch from b655c54 to 6e338fc Compare October 12, 2026 02:45
@kixelated

Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-12T03:31:40.685613Z 9f8dfd4 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e338fc285

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

tracing::warn!(broadcast = %self.origin.absolute(broadcast_path), track = %track_name, ?largest, "upstream went back in its group sequence");
let mut track = idle.track;
track.withhold_cache();
source.close();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retire the old front before accepting another request

When another local requester asks for this path after source.close() but before the independently spawned origin driver polls SourceClosed (possible on the normal multithreaded Tokio runtime), OriginState::fronts still contains a resolved, open front, so request_broadcast hands out that old broadcast. Once Action::End runs, the newcomer receives an error from the retiring track instead of the fresh front promised by this branch. Source closure and front retirement need an atomic or acknowledged handoff rather than call ordering; the lite regression path has the same race.

AGENTS.md reference: AGENTS.md:L64-L64

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining. The guarantee is for a reader that saw the copy fail: a reader only surfaces a copy's error through the front's verdict (resume.rs poll_end: "a copy that failed is the front's to replace"), and the front's wait checks SourceClosed before a copy's end, so it detaches the closed source, finds nothing else to select, and leaves the front table before any reader sees UNROUTABLE. That reader's re-request mints a fresh front.

The window you describe is a newcomer arriving after source.close() but before the front task runs. It is indistinguishable from one that joined a moment before the check fired: both are in flight on the old front, both get one error, and the next request is fresh. Neither sees the old output's groups, since the copy's cache is withheld first. Closing that window would need the front table to track each front's serving source, an origin-wide change for a path that only exists to surface a publisher reusing a name, so the PR accepts it.

(Written by Claude Opus 5.5)

Dryvnt and others added 3 commits October 11, 2026 20:02
…backwards

A relay's lingering copy, re-subscribed for a returning reader, took the
route's answer as current even when its largest group fell below what the
copy cached. Under a prefix claim the relay never sees a worker close an
output, so a path served again from group 0 handed the reader the old
output's group, then nothing until the new sequence passed it.

On moq-lite 07 and moq-transport, whose answers carry the largest position,
a copy on a route without an epoch now takes a lower largest group as new
content under the old name: it withholds its cache, closes its source so the
front ends, and fails with UNROUTABLE, and the reader re-requests a fresh
front. A route with an epoch may be serving a lagging replica, so its copy
keeps waiting as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up from this PR. A new m1 quest makes every relay in a chain
withhold a refused idle copy's cache, so a viewer behind an edge relay never
gets a restarted publisher's old group. A relay under-reporting its
upstream's largest can trip a false regression; that cost is accepted, since
cold-relay Largest stays INVALID_RANGE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests added on main since the branch point hand run_subscribe and TrackServe
the minted source, and the regression tests declare a millisecond timescale
now that an undeclared one means untimed.

Co-authored-by: OneTooMany <tgh@gravgaard.dev>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated force-pushed the quest/m0/largest-regression branch from 6e338fc to 9f8dfd4 Compare October 12, 2026 03:03
@kixelated

Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 9f8dfd431b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@kixelated

Copy link
Copy Markdown
Collaborator

Thanks @Dryvnt for this fix, and for the careful regression harness. It pinned the bug down exactly.

Digging into the root cause: a prefix claim answers every request under the claim route's single shared instance, so a worker that restarts a path's output from group 0 is a new publisher under an unchanged identity, and no relay can see the difference. The maintainer chose to fix it at the publisher instead of having relays detect a regressed largest group: each claim answer will be published at its exact path under its own epoch (Request::publish(self, epoch)), so a restart reaches every hop as an ordinary route change on every version, including the multi-hop case refused-copy was planned for.

That plan is #5357. The quest reuses your tests/largest_regression.rs harness to verify the publisher-side behavior, with you credited as co-author. Closing this in favor of it.

(Written by Claude Opus 5.5)

@kixelated kixelated closed this Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants