Repository navigation
Conversation
|
Rebase notes from the 2026-10-08 quest audit (#5058, #5063):
(Written by Claude Opus 5.5) |
8f1a4d8 to
b655c54
Compare
|
Thanks @Dryvnt for this fix and the careful write-up. The regression judgment and the source-before-abort ordering held up unchanged through the rebase. I rebased your two commits onto main (dropping the #5054 commits that merged), keeping your authorship:
(Written by Claude Opus 5.5) |
b655c54 to
6e338fc
Compare
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e338fc285
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| tracing::warn!(broadcast = %self.origin.absolute(broadcast_path), track = %track_name, ?largest, "upstream went back in its group sequence"); | ||
| let mut track = idle.track; | ||
| track.withhold_cache(); | ||
| source.close(); |
There was a problem hiding this comment.
Retire the old front before accepting another request
When another local requester asks for this path after source.close() but before the independently spawned origin driver polls SourceClosed (possible on the normal multithreaded Tokio runtime), OriginState::fronts still contains a resolved, open front, so request_broadcast hands out that old broadcast. Once Action::End runs, the newcomer receives an error from the retiring track instead of the fresh front promised by this branch. Source closure and front retirement need an atomic or acknowledged handoff rather than call ordering; the lite regression path has the same race.
AGENTS.md reference: AGENTS.md:L64-L64
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining. The guarantee is for a reader that saw the copy fail: a reader only surfaces a copy's error through the front's verdict (resume.rs poll_end: "a copy that failed is the front's to replace"), and the front's wait checks SourceClosed before a copy's end, so it detaches the closed source, finds nothing else to select, and leaves the front table before any reader sees UNROUTABLE. That reader's re-request mints a fresh front.
The window you describe is a newcomer arriving after source.close() but before the front task runs. It is indistinguishable from one that joined a moment before the check fired: both are in flight on the old front, both get one error, and the next request is fresh. Neither sees the old output's groups, since the copy's cache is withheld first. Closing that window would need the front table to track each front's serving source, an origin-wide change for a path that only exists to surface a publisher reusing a name, so the PR accepts it.
(Written by Claude Opus 5.5)
…backwards A relay's lingering copy, re-subscribed for a returning reader, took the route's answer as current even when its largest group fell below what the copy cached. Under a prefix claim the relay never sees a worker close an output, so a path served again from group 0 handed the reader the old output's group, then nothing until the new sequence passed it. On moq-lite 07 and moq-transport, whose answers carry the largest position, a copy on a route without an epoch now takes a lower largest group as new content under the old name: it withholds its cache, closes its source so the front ends, and fails with UNROUTABLE, and the reader re-requests a fresh front. A route with an epoch may be serving a lagging replica, so its copy keeps waiting as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up from this PR. A new m1 quest makes every relay in a chain withhold a refused idle copy's cache, so a viewer behind an edge relay never gets a restarted publisher's old group. A relay under-reporting its upstream's largest can trip a false regression; that cost is accepted, since cold-relay Largest stays INVALID_RANGE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests added on main since the branch point hand run_subscribe and TrackServe the minted source, and the regression tests declare a millisecond timescale now that an undeclared one means untimed. Co-authored-by: OneTooMany <tgh@gravgaard.dev> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6e338fc to
9f8dfd4
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Thanks @Dryvnt for this fix, and for the careful regression harness. It pinned the bug down exactly. Digging into the root cause: a prefix claim answers every request under the claim route's single shared instance, so a worker that restarts a path's output from group 0 is a new publisher under an unchanged identity, and no relay can see the difference. The maintainer chose to fix it at the publisher instead of having relays detect a regressed largest group: each claim answer will be published at its exact path under its own epoch ( That plan is #5357. The quest reuses your (Written by Claude Opus 5.5) |
Implements
quest/m1/largest-regression.md(deleted here). Contributed by @Dryvnt; rebased onto main by the maintainer's agent.Problem
A relay keeps a track's copy for the linger after its last reader leaves, and a returning reader re-subscribes it from the newest group it cached. Under a prefix claim the relay never sees a worker close an output, so when the worker serves the path again from group 0, the answer's largest group is below the cache. The copy took the answer as current and served the old output's group, then nothing until the new sequence passed it.
Approach
On versions whose answer carries the largest position (moq-lite 07, moq-transport), an idle copy on a route without an epoch takes a largest group below what it still caches as new content under the old name. It then:
UNROUTABLE, as an epochless track failing upstream now does (fix(net): an epochless track ends on an upstream error instead of re-splicing #5137), so the reader requests a fresh front.A route with an epoch may be serving a replica that lags the copy, so its copy keeps waiting as before. moq-lite 05 and 06 carry no largest and keep waiting on the floor, as does an answer with no largest (a cold moq-transport relay).
Only a restart behind an unchanged route reaches a copy unannounced: since #5087 every route change, including a per-path winner change under a prefix pool, reaches downstream as a
Restartthat drops the cached copy. Closing the source is still needed, because #5087's sticky fronts let a new request join a front while its route entry still wins, and here the entry never changed.Known false positive, accepted: an upstream relay answers from its own copy's cache rather than its upstream's largest, so a copy it re-created cold that holds only older groups under-reports. That needs the downstream copy to outlive the upstream's plus a concurrent lower-start subscriber upstream. It costs a spurious
UNROUTABLEand a re-request, never stale data. Equal group numbers can't be told apart and still splice.Rebase onto main
subscribe_once(moq-transport) andServeLoop(lite-07). The tests added since the branch point handrun_subscribeandTrackServethe minted source.largest-regressionmoved to m1, so this PR deletes that path. Claim epochs is gone from main (replaced by Announcement takeover), so no claim-epoch edit remains. Theietf-cold-largestfold is dropped: that quest is deleted and cold-relay Largest stays INVALID_RANGE.Tests
Mocked time.
tests/largest_regression.rsrestarts a claim worker's output within the linger on moq-lite 07 and moq-transport 14, 16 and 22: the returning reader getsUNROUTABLE, then the new output from group 0, never the old group. With the check disabled, every version gets the old group (re-verified on the rebased head). A same-epoch standby behind the cached copy keeps the copy and resumes it. Atrack.rsunit test covers the judgment and the withheld cache.just checkpasses locally.just test interop --allpasses exceptpython -> c, which timed out under host load (load average over 100) and passes on rerun.Impact
track::Producer::regressesandwithhold_cacheare crate-private.UNROUTABLEin this case.Alternatives
Decisions
quest/m1/refused-copy.md(maintainer, 2026-10-08: m1, beside its parents, not m0).ietf-cold-largestis not reopened).UNROUTABLE, as a superseded or failed epochless track gets.Follow-ups
quest/m1/refused-copy.md[M] (last commit): behind any chain of relays, a returning reader never gets a restarted publisher's old group. An idle copy refused before it was answered withholds its cache and ends, and no hop answers from a withheld cache, on lite-07 and moq-transport. Its facts are refreshed against main (no line numbers, supersede gone after feat(net)!: restart announce consumers on a replaced broadcast, keep subscriptions sticky #5087).(Written by Claude Opus 5.5)
🤖 Generated with Claude Code