From 2fc643a50b7dddcb92ec36dcd649e1878d136192 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 12 Oct 2026 08:08:10 -0700 Subject: [PATCH 1/5] quest(m1): plan per-answer epochs for claims, replacing largest-regression A claim worker that restarts a path's output under one instance is a publisher bug, so fix it at the publisher: Request::publish(epoch) answers each output at its exact path under its own epoch. The relay-side largest-regression check (#5057) is dropped. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 +- quest/m1/announce-takeover.md | 8 ++-- quest/m1/claim-answer-epochs.md | 71 +++++++++++++++++++++++++++++++++ quest/m1/largest-regression.md | 65 ------------------------------ 4 files changed, 77 insertions(+), 69 deletions(-) create mode 100644 quest/m1/claim-answer-epochs.md delete mode 100644 quest/m1/largest-regression.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 17b74156dcc..0bb28bffd77 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -48,7 +48,7 @@ session outcomes, and embedder refusals inside session outcomes. - [Interop close logs](/quest/m1/interop-close-log.md) - logging fix landed; delete once the required Interop run is green - [One max_age meaning](/quest/m1/cache-max-age.md) - a superseded group goes stale on wall clock since its successor arrived or on media time, whichever is first, in Rust and js/net; fixes the untimed failover stall - [FETCH max-delay](/quest/m1/fetch-max-delay.md) - historical readers obey publisher max-age and an optional tighter reader budget, including while holding a group -- [Upstream position regression](/quest/m1/largest-regression.md) - a relay copy that sees upstream's largest group go backwards on moq-transport or epochless lite-07 fails loud instead of serving the old instance's cache +- [Claim answer epochs](/quest/m1/claim-answer-epochs.md) - a claim answers each output by publishing it at its exact path under its own epoch, so a restarted output never reaches a returning reader as the old one's cache - [Untimed decisions](/quest/m1/untimed-decisions.md) - the maintainer decides whether moq-archive keeps refusing untimed tracks, and whether a malformed FETCH object ends its track - [FFI shape](/quest/m1/ffi-shape/README.md) - the bindings mirror Rust's layers: net at the root, then media, json, flate, audio, and video namespaces built from the handle below - [Publishing never invents a timestamp](/quest/m1/publish-timestamp.md) - no Rust or binding publish API fills in a timestamp; an untimed payload goes out untimed diff --git a/quest/m1/announce-takeover.md b/quest/m1/announce-takeover.md index 3f1b732b311..dfca44fc50e 100644 --- a/quest/m1/announce-takeover.md +++ b/quest/m1/announce-takeover.md @@ -17,7 +17,8 @@ IETF counterpart remains separate. Ordinary Start/Restart is a supported fallback, including the accepted cold-start reset before this lands. - A worker advertises `transcode/` without an epoch. Demand for a path starts - one job, which announces `transcode/path` with its own freshly minted epoch. + one job, which announces `transcode/path` with its own freshly minted epoch + through `Request::publish`, the claim-to-route operation. Lazy catalog references and initial requests need no anticipated epoch. Identical encoder settings do not make two workers the same instance. - TAKEOVER is START plus a predecessor Announce ID: it carries the child @@ -81,12 +82,13 @@ event, not only route-table checks. Measure route/readers fanout on both axes. Run `just check`, `just test interop --all`, and `just drafts check`; wire regressions and benchmarks into existing CI lanes. -Public API: a claim-to-independent-route operation and a `Takeover` event, -mirrored across languages and exposed bindings. Wire: lite-07-wip gains +Public API: a `Takeover` event, mirrored across languages and exposed +bindings, on top of `Request::publish`. Wire: lite-07-wip gains ANNOUNCE_TAKEOVER; published versions retain their framing. ## Required +- [Claim answer epochs](/quest/m1/claim-answer-epochs.md) - `Request::publish` answers a claim at the exact path under a fresh epoch, the operation TAKEOVER continues - [Routes and announces](/quest/m1/cluster-routing/routes.md) - shared routing layer and per-stream identity ## Related diff --git a/quest/m1/claim-answer-epochs.md b/quest/m1/claim-answer-epochs.md new file mode 100644 index 00000000000..33b7e285924 --- /dev/null +++ b/quest/m1/claim-answer-epochs.md @@ -0,0 +1,71 @@ +# [L] A claim answers each output as its own epoch + +## Goal + +A worker behind a prefix claim that restarts a path's output does so under a +new epoch, so every hop sees the change and drops its copy: a returning reader +never gets the old output's groups. The public claim API cannot serve two +outputs at one path under the same instance. Replaces the relay-side +regression check proposed in #5057. + +## Plan + +Facts as of main 90500867d: + +- A claim (`origin::Producer::dynamic`, JS `Producer.dynamic`, moq-ffi, moq-c + and the binding wrappers) answers with any broadcast via `Request::accept`. + Every answer resolves under the claim route's instance (its epoch, or the + entry's generation), shared by every path and every answer for the claim's + lifetime. Once an answer closes, the handler is asked again and the new + broadcast, restarting at group 0, resolves under the same instance; nothing + on any wire changes, so a lingering downstream copy splices old onto new. +- `Dynamic::update` to a new epoch, or a renewal, restarts the whole prefix, + resetting every sibling path's readers. Too coarse for a pool. +- Tracks already avoid this: a broadcast keeps each track name's sequence + across replacement producers. Claims have no equivalent. +- In repo, the production claim handlers are moq-net sessions forwarding a + peer's claim; they relay upstream numbering and keep working as is. + +Decided 2026-10-12: a request is answered by publishing, not by handing over +content, and the caller names the epoch: + + impl Request { + /// Serve the path as a new publisher instance, announced at exactly this path under `epoch`. + pub fn publish(self, epoch: Epoch) -> Result; + } + +The broadcast is announced at the requested path with the claim's route +metadata and that epoch, and requesters resolve through that route, never the +claim's instance. Closing it ends the route; the next request publishes +under a new epoch (`Epoch::mint()` per run, or a replica set's shared one). +Session forwarding keeps a crate-private accept. Rejected: renewing the +prefix on each restart (resets every sibling), and carrying sequences over to +the next answer (relabels a different job as a continuation). + +Look out for: + +- A downstream relay that resolved the path through its copy of the claim + before the exact announcement arrived holds a front under the claim's + instance. Check it can't be rejoined once another route won the path and + then ended; if it can, retire it per path, not by renewing the prefix. The + cold-start reset it implies is accepted until announce takeover lands. +- One exact announcement per live output: benchmark outputs against downstream + cursors. +- No producer-side refusal of a backwards sequence: a fresh broadcast has no + memory of the last run, and backfill below the newest group is legitimate. + +Verification: mocked time, reusing #5057's worker harness +(`tests/largest_regression.rs` on its branch, with Dryvnt credited as +co-author). A worker restarts `pool/p` within the linger, behind one relay +and behind an edge relay, on lite-05, 06, 07 and moq-transport 14, 16, 22: +the returning reader gets the new output from group 0, never the old group, +and sibling paths keep their readers. A same-epoch lagging replica still +resumes. JS mirrors the tests. `just test interop --all`. + +Public API: breaking change to how a claim request is answered, in Rust, JS, +moq-ffi, moq-c and every binding wrapper, plus the claim docs. Wire: none. + +## Related + +- [Announcement takeover](/quest/m1/announce-takeover.md) - its claim-to-route operation is `Request::publish`; TAKEOVER then removes the cold-start reset +- [#5057](https://github.com/moq-dev/moq/pull/5057) - the relay-side check this replaces diff --git a/quest/m1/largest-regression.md b/quest/m1/largest-regression.md deleted file mode 100644 index f645ecce759..00000000000 --- a/quest/m1/largest-regression.md +++ /dev/null @@ -1,65 +0,0 @@ -# [S] A relay copy fails loud when upstream's position goes backwards - -## Goal - -A relay copy that subscribes upstream again and hears that upstream's largest -group is below the newest group it has cached treats it as a reused name: the -copy ends with an error, its source closes so the front ends, and the readers -it was serving re-request. A new request arriving as the front ends gets a -fresh front without an error. Today the copy keeps its stale live floor (`set_live` in -`rs/moq-net/src/model/track.rs`), so a viewer returning within the linger to a -publisher that restarted at group 0 gets the old instance's cached group and -then nothing until the new sequence passes it. - -Covers lite-07 and moq-transport, whose answers carry the largest position. -lite-05 and 06 stay uncovered on purpose: their only hint is the first -group's sequence, and they keep waiting on the floor. - -## Plan - -Found while scoping idle fronts (2026-10-07): -under a prefix claim the relay never sees the worker close a broadcast, so a -restart at the same path reaches a lingering copy unannounced. A publisher -that restarts its group sequence under the same name is buggy, and this makes -that bug visible instead of a silent stall. A change of route can't cause -it: every route change, including a per-path winner change under a prefix -pool, reaches downstream as a `Restart` of the route, which unsets the cached -copy of every broadcast nested under it -(maintainer, 2026-10-08, in Restart). -Only a restart behind an unchanged route reaches a copy unannounced. - -Decided 2026-10-08: shrunk to the two answer-carrying wires and moved to m1. -The heuristic is unproven and risks a false failover, so it builds on the -front and route shapes Restart and idle fronts leave rather than ahead of -them. - -Where it lands: since #4914, a rejoining copy goes idle (`set_idle`) and the -route's answer reaches `set_live`, which holds readers in `Feed::Answered` -until the cache shows the answered position. An answer below what the cache -already shows goes straight to `Feed::Live` and keeps the old floor; that is -the case to end instead, when it qualifies below. - -A lower largest group alone doesn't prove a restart: a same-epoch resume onto -a replica that has fallen behind reports one legitimately, and must keep -waiting as today. The copy ends only when the answer comes from the route it -was last served from without an epoch, where no cross-route resume happens, -or names a different epoch than the copy holds. Check for any other -legitimate case before relying on that split. - -Whatever ends a front, it leaves the origin's front table before its broadcast -closes or anything fails, and `request()` treats a front whose broadcast closed -as gone. So a reader re-requesting after seeing the front's end mints a fresh -front. The copy's error must reach readers through that end, not an `Abort` -ahead of it, or a re-request can still join the old front. A request that -joined before the front decided to end is in flight, like a subscription. - -Verification: mocked time, a publisher restarting a path at group 0 within -the linger. The returning reader gets an error and then the new instance from -group 0, never the old group. A same-epoch standby behind the cached copy -keeps the copy and resumes it. - -Public API: none. Wire: none. - -## Related - -- [Announcement takeover](/quest/m1/announce-takeover.md) - proves continuity when a lazy job gains an independent announcement; a replacement never takes over its predecessor's content From 1ab53a809081999bb5529bab263fa50e589ad41f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 12 Oct 2026 08:15:15 -0700 Subject: [PATCH 2/5] quest: claim answers prepare before publishing and stay atomic with the claim Co-Authored-By: Claude Opus 5.5 --- quest/m1/claim-answer-epochs.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/quest/m1/claim-answer-epochs.md b/quest/m1/claim-answer-epochs.md index 33b7e285924..4181ca96c67 100644 --- a/quest/m1/claim-answer-epochs.md +++ b/quest/m1/claim-answer-epochs.md @@ -42,6 +42,24 @@ Session forwarding keeps a crate-private accept. Rejected: renewing the prefix on each restart (resets every sibling), and carrying sequences over to the next answer (relabels a different job as a continuation). +Open, for the maintainer: a requester that resolves through an announced but +still empty broadcast gets `NotFound` for its first track, so the handler +must create its tracks (or a `broadcast::Dynamic`) before anyone resolves. +`publish` as sketched announces too early. Options, keeping the name and the +explicit epoch: + +- A typed two-phase answer: `request.prepare(epoch)` returns an unannounced + handle to fill, and its `publish()` announces and resolves (recommended: + misuse can't be expressed). +- `request.publish(broadcast, epoch)` taking a broadcast from + `create_broadcast(request.path())`, refusing a wrong path or an announced + one at runtime. + +Required: validity and route insertion are atomic. A request the claim +released (an `update` to another instance) or abandoned (the claim dropped) +before the handler publishes is refused, never announced, as `accept` drops +it today. Regression tests cover both. + Look out for: - A downstream relay that resolved the path through its copy of the claim From d533c7b591df2951c6e33e13cdd466fa49ddd01b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 12 Oct 2026 08:20:11 -0700 Subject: [PATCH 3/5] quest: record the epoch'd claim choice for claim answer epochs Co-Authored-By: Claude Opus 5.5 --- quest/m1/claim-answer-epochs.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/quest/m1/claim-answer-epochs.md b/quest/m1/claim-answer-epochs.md index 4181ca96c67..93aa87c43b3 100644 --- a/quest/m1/claim-answer-epochs.md +++ b/quest/m1/claim-answer-epochs.md @@ -55,6 +55,21 @@ explicit epoch: `create_broadcast(request.path())`, refusing a wrong path or an announced one at runtime. +Open, for the maintainer: a claim announced with an epoch. A lite-07 +requester may pin a child path to the claim's epoch, which an answer under a +fresh epoch never satisfies. Options: + +- Public claims carry no epoch: `Producer::dynamic` refuses one (or its + route type has none), and a pinned request resolves only through an exact + route with that epoch (recommended: a claim is a capability, as announce + takeover already treats it). Sessions keep epochs on the peer routes they + land internally. +- An epoch'd claim keeps its identity: its answers publish under the claim's + epoch, and changing any output's content means `Dynamic::update` to a new + epoch for the whole prefix. + +Either way the regression matrix covers a pinned request. + Required: validity and route insertion are atomic. A request the claim released (an `update` to another instance) or abandoned (the claim dropped) before the handler publishes is refused, never announced, as `accept` drops From 0467fb51481ebe6d64e9a5e7bc754917bda02e68 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 12 Oct 2026 08:46:31 -0700 Subject: [PATCH 4/5] quest: decide a two-step claim answer and epochless public claims Co-Authored-By: Claude Opus 5.5 --- quest/m1/announce-takeover.md | 6 +-- quest/m1/claim-answer-epochs.md | 78 +++++++++++++++------------------ 2 files changed, 38 insertions(+), 46 deletions(-) diff --git a/quest/m1/announce-takeover.md b/quest/m1/announce-takeover.md index dfca44fc50e..490c6e911ac 100644 --- a/quest/m1/announce-takeover.md +++ b/quest/m1/announce-takeover.md @@ -18,7 +18,7 @@ fallback, including the accepted cold-start reset before this lands. - A worker advertises `transcode/` without an epoch. Demand for a path starts one job, which announces `transcode/path` with its own freshly minted epoch - through `Request::publish`, the claim-to-route operation. + through `Request::prepare` then `publish`, the claim-to-route operation. Lazy catalog references and initial requests need no anticipated epoch. Identical encoder settings do not make two workers the same instance. - TAKEOVER is START plus a predecessor Announce ID: it carries the child @@ -83,12 +83,12 @@ Run `just check`, `just test interop --all`, and `just drafts check`; wire regressions and benchmarks into existing CI lanes. Public API: a `Takeover` event, mirrored across languages and exposed -bindings, on top of `Request::publish`. Wire: lite-07-wip gains +bindings, on top of `Request::prepare` and `publish`. Wire: lite-07-wip gains ANNOUNCE_TAKEOVER; published versions retain their framing. ## Required -- [Claim answer epochs](/quest/m1/claim-answer-epochs.md) - `Request::publish` answers a claim at the exact path under a fresh epoch, the operation TAKEOVER continues +- [Claim answer epochs](/quest/m1/claim-answer-epochs.md) - `Request::prepare` then `publish` answers a claim at the exact path under a fresh epoch, the operation TAKEOVER continues - [Routes and announces](/quest/m1/cluster-routing/routes.md) - shared routing layer and per-stream identity ## Related diff --git a/quest/m1/claim-answer-epochs.md b/quest/m1/claim-answer-epochs.md index 93aa87c43b3..e1afe1a520c 100644 --- a/quest/m1/claim-answer-epochs.md +++ b/quest/m1/claim-answer-epochs.md @@ -27,53 +27,44 @@ Facts as of main 90500867d: peer's claim; they relay upstream numbering and keep working as is. Decided 2026-10-12: a request is answered by publishing, not by handing over -content, and the caller names the epoch: +content. The answer has two steps, so the broadcast holds its tracks (or a +`broadcast::Dynamic`) before anyone resolves through it, and the caller names +the epoch. Roughly: impl Request { - /// Serve the path as a new publisher instance, announced at exactly this path under `epoch`. - pub fn publish(self, epoch: Epoch) -> Result; + /// Start answering the path as a new publisher instance under `epoch`, unannounced until published. + pub fn prepare(self, epoch: Epoch) -> Result; } -The broadcast is announced at the requested path with the claim's route -metadata and that epoch, and requesters resolve through that route, never the -claim's instance. Closing it ends the route; the next request publishes -under a new epoch (`Epoch::mint()` per run, or a replica set's shared one). -Session forwarding keeps a crate-private accept. Rejected: renewing the -prefix on each restart (resets every sibling), and carrying sequences over to -the next answer (relabels a different job as a continuation). - -Open, for the maintainer: a requester that resolves through an announced but -still empty broadcast gets `NotFound` for its first track, so the handler -must create its tracks (or a `broadcast::Dynamic`) before anyone resolves. -`publish` as sketched announces too early. Options, keeping the name and the -explicit epoch: - -- A typed two-phase answer: `request.prepare(epoch)` returns an unannounced - handle to fill, and its `publish()` announces and resolves (recommended: - misuse can't be expressed). -- `request.publish(broadcast, epoch)` taking a broadcast from - `create_broadcast(request.path())`, refusing a wrong path or an announced - one at runtime. - -Open, for the maintainer: a claim announced with an epoch. A lite-07 -requester may pin a child path to the claim's epoch, which an answer under a -fresh epoch never satisfies. Options: - -- Public claims carry no epoch: `Producer::dynamic` refuses one (or its - route type has none), and a pinned request resolves only through an exact - route with that epoch (recommended: a claim is a capability, as announce - takeover already treats it). Sessions keep epochs on the peer routes they - land internally. -- An epoch'd claim keeps its identity: its answers publish under the claim's - epoch, and changing any output's content means `Dynamic::update` to a new - epoch for the whole prefix. - -Either way the regression matrix covers a pinned request. + impl Prepared { + /// Announce the filled broadcast at exactly the requested path and resolve every waiting requester. + pub fn publish(self) -> Result; + } + +`Prepared` (name open) gives access to the broadcast to fill and is the only +way to answer, so an announced empty answer can't be expressed. Dropping it +unpublished rejects the request, as dropping a `Request` does. The broadcast +is announced with the claim's route metadata and that epoch, and requesters +resolve through that route, never the claim's instance. Closing it ends the +route; the next request prepares under a new epoch (`Epoch::mint()` per run, +or a replica set's shared one). Session forwarding keeps a crate-private +accept. Rejected: a single `publish` that announces an empty broadcast, +renewing the prefix on each restart (resets every sibling), and carrying +sequences over to the next answer (relabels a different job as a +continuation). + +Decided 2026-10-12: public claims carry no epoch. A claim is a capability, +as announce takeover already treats it, so `Producer::dynamic` (JS and the +bindings alike) cannot take one; prefer a shape where the route has none over +a runtime refusal. A request pinned to an epoch resolves only through an +exact-path route with that epoch. Sessions keep epochs on the peer routes +they land internally. Migrate the in-repo callers and tests that claim with +an epoch. Required: validity and route insertion are atomic. A request the claim released (an `update` to another instance) or abandoned (the claim dropped) -before the handler publishes is refused, never announced, as `accept` drops -it today. Regression tests cover both. +before `publish` is refused, never announced, as `accept` drops it today. +Regression tests cover both, and a pinned request. Look out for: @@ -95,10 +86,11 @@ the returning reader gets the new output from group 0, never the old group, and sibling paths keep their readers. A same-epoch lagging replica still resumes. JS mirrors the tests. `just test interop --all`. -Public API: breaking change to how a claim request is answered, in Rust, JS, -moq-ffi, moq-c and every binding wrapper, plus the claim docs. Wire: none. +Public API: breaking. A claim request is answered through `prepare` and +`publish`, and a public claim takes no epoch, in Rust, JS, moq-ffi, moq-c and +every binding wrapper, plus the claim docs. Wire: none. ## Related -- [Announcement takeover](/quest/m1/announce-takeover.md) - its claim-to-route operation is `Request::publish`; TAKEOVER then removes the cold-start reset +- [Announcement takeover](/quest/m1/announce-takeover.md) - its claim-to-route operation is `Request::prepare` then `publish`; TAKEOVER then removes the cold-start reset - [#5057](https://github.com/moq-dev/moq/pull/5057) - the relay-side check this replaces From 51b33d4118e30327ba000984d066df00d9cafd9d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 12 Oct 2026 08:50:12 -0700 Subject: [PATCH 5/5] quest: scope claim answer epochs to our relays on epochless wires Co-Authored-By: Claude Opus 5.5 --- quest/m1/claim-answer-epochs.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/quest/m1/claim-answer-epochs.md b/quest/m1/claim-answer-epochs.md index e1afe1a520c..4a28eb1d08b 100644 --- a/quest/m1/claim-answer-epochs.md +++ b/quest/m1/claim-answer-epochs.md @@ -81,10 +81,15 @@ Look out for: Verification: mocked time, reusing #5057's worker harness (`tests/largest_regression.rs` on its branch, with Dryvnt credited as co-author). A worker restarts `pool/p` within the linger, behind one relay -and behind an edge relay, on lite-05, 06, 07 and moq-transport 14, 16, 22: -the returning reader gets the new output from group 0, never the old group, -and sibling paths keep their readers. A same-epoch lagging replica still -resumes. JS mirrors the tests. `just test interop --all`. +and behind an edge relay of ours, on lite-05, 06, 07 and moq-transport 14, +16, 22: the returning reader gets the new output from group 0, never the old +group, and sibling paths keep their readers. On the wires without an epoch +that rests on our relays retiring a copy on End then Start, as +`tests/restart.rs` shows; arbitrary older caching relays are out of scope, as +in [IETF epochs](/quest/m1/ietf-epochs.md), since base IETF object identity +survives a withdrawal. A same-epoch lagging replica still resumes, checked on +lite-07 only, the one wire here that carries the epoch. JS mirrors the tests. +`just test interop --all`. Public API: breaking. A claim request is answered through `prepare` and `publish`, and a public claim takes no epoch, in Rust, JS, moq-ffi, moq-c and @@ -93,4 +98,5 @@ every binding wrapper, plus the claim docs. Wire: none. ## Related - [Announcement takeover](/quest/m1/announce-takeover.md) - its claim-to-route operation is `Request::prepare` then `publish`; TAKEOVER then removes the cold-start reset +- [IETF epochs](/quest/m1/ietf-epochs.md) - carries the answer epoch over moq-transport, so same-epoch resume and replacement can be told apart there - [#5057](https://github.com/moq-dev/moq/pull/5057) - the relay-side check this replaces