diff --git a/quest/m1/credit-model.md b/quest/m1/credit-model.md new file mode 100644 index 0000000000..a0dc11b8e6 --- /dev/null +++ b/quest/m1/credit-model.md @@ -0,0 +1,70 @@ +# [L] A TLA+ model shows receive credit never deadlocks + +## Goal + +A TLA+ model of one connection's flow control, run by TLC in CI like the +other `rs/moq-net/verify` models, checks every ordering of arrivals, reads, +pauses, and evictions for two properties: + +- Safety: the credit every stream holds together (unread bytes, leased bytes + not yet released, unfinished frames, paused allowances) never exceeds the + connection window, and paused credit never exceeds its cap. +- Liveness: under weak fairness for the reader and the peer, every stream + eventually progresses, finishes, or is evicted, and a sender waiting for + stream credit eventually opens or gives up its group. + +It models the design as [#9210](https://github.com/moq-dev/moq/pull/9210) +leaves it, and the code matches the model, or the PR fixes it. + +## Plan + +Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control audit): + +- ✅ The cut waits on a model (rejected: restoring the overcommitted benchmark + cells as the evidence, which only samples orderings; cutting without it). + At least four credit bugs landed in the two weeks before the audit, two of + them hangs: [#8794](https://github.com/moq-dev/moq/pull/8794) (a paused + stream granted credit until it held the window), + [#8808](https://github.com/moq-dev/moq/pull/8808) (QMux leaked receive + credit on `release`), [#8774](https://github.com/moq-dev/moq/pull/8774) + (a dropped group never reached the reader), and + [#8607](https://github.com/moq-dev/moq/pull/8607) (a group waiting for + credit held a higher-ranked one behind it). +- ✅ The model comes before #9210, which merges once the model passes on its + design (rejected: #9210 first, or both in parallel). +- ✅ Scope: receive credit, pausing, and the sender's credit wait (rejected: + receive side only; a relay chain end to end, since pausing is per hop). + +What to model: + +- Receive, `rs/moq-quic/src/connection/streams/lease.rs`: the connection and + stream windows, leases and their release at a segment seal, and + `evict_if_stuck` with its two waits: an evicted reader that has not stopped + its stream (`evicted_held`), and data a reader has not looked at + (`trim_unlooked`). The second is the likeliest hang: it holds eviction off + for as long as any stream has data no reader looked at, so liveness depends + on moq-net always polling every stream it accepted, including ones not yet + accepted. +- Pausing, `rs/moq-net/src/transport/boxed.rs` and `model/pace.rs`: the + half-window cap, #9210's slot allowances (`cap / ((i + 1) * (i + 2))`) and + `set_window`, the pause only at a segment seal, and the read-on-and-shed + path past the cap. +- Send, `rs/moq-net/src/lite/publisher.rs` `TrackRun::opening` and `stuck`, + and `transport/open.rs`: a group waiting for stream credit, by order and + rank. + +Abstract bytes into a few units, and keep frames, streams, and slots small +enough for TLC to run in the merge gate, as the other models do; a larger +configuration can run nightly. Add a `verify-credit` skill beside +`verify-subscribe` and the others, naming the code each action maps to. + +A violation in main's code is fixed in this PR. A violation in #9210's +design goes to #9210 before it merges. + +Public API: none. Wire: none. + +## Related + +- [Paused-stream allowance](/quest/m1/paused-allowance.md) - the design this model checks before it merges +- [Overcommitted receive-window cells](/quest/m3/receive-windows-evicted.md) - the benchmark side of the same mechanism +- [Send-credit stall counters](/quest/m3/send-credit-stalls.md) - what an operator sees when the sender waits on credit diff --git a/quest/m1/js-frame-cap.md b/quest/m1/js-frame-cap.md new file mode 100644 index 0000000000..e4eff887f7 --- /dev/null +++ b/quest/m1/js-frame-cap.md @@ -0,0 +1,24 @@ +# [XS] @moq/net rejects a received frame over 32 MiB + +## Goal + +Both JS wire readers (`js/net/src/lite/group.ts` and +`js/net/src/ietf/object.ts`) reject a plain frame whose declared size passes +`MAX_FRAME_BYTES` at its header with `FrameTooLarge`, before reading any of +it, as Rust's `content.rs` admission does. A local writer is unchanged. + +## Plan + +Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control +audit): ✅ additive, not a cut blocker (rejected: blocking the cut; dropping +it). + +Today a received plain frame past one segment streams in at any size, and +`Consumer.readFrame()` (`#join` in `js/net/src/group.ts`) keeps every chunk +until it joins them, even after the group sheds them, and the browser +media path reads that way (`js/hang/src/container/consumer.ts`). Rust +relays already reject such frames, so only a browser connected directly to +a hostile peer is exposed. The 40 MiB tests in `group.test.ts` write +locally and stay. + +Behavior change, no `!`. Public API: none. Wire: none. diff --git a/quest/m1/paused-allowance.md b/quest/m1/paused-allowance.md new file mode 100644 index 0000000000..559567d8a6 --- /dev/null +++ b/quest/m1/paused-allowance.md @@ -0,0 +1,24 @@ +# [S] Any number of streams pause within half the connection window + +## Goal + +[#9210](https://github.com/moq-dev/moq/pull/9210) merges with its design +checked by the [credit model](/quest/m1/credit-model.md): each paused stream +holds a slot-sized allowance of credit instead of a stream window, so any +number of streams pause within half the connection window. #9210 deletes +this quest. + +## Plan + +Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control +audit): ✅ #9210 ships in this cut (rejected: landing after the cut to soak). +✅ It merges once the credit model passes on its design (rejected: merging +first and modeling the result). + +#9210's own design decisions, settled 2026-11-19, and its two open `Waiting` +items stay on the PR. When the model finds a violation in its design, fix it +there before merging. + +## Required + +- [Credit model](/quest/m1/credit-model.md) - checks the slot allowances and `set_window` before they ship diff --git a/quest/m1/relay-memory-default.md b/quest/m1/relay-memory-default.md new file mode 100644 index 0000000000..fe6add7713 --- /dev/null +++ b/quest/m1/relay-memory-default.md @@ -0,0 +1,31 @@ +# [S] A stock moq-relay bounds its cache memory + +## Goal + +moq-relay with no cache settings sizes its group cache to keep 10% of system +memory free (the cgroup limit when set), as `cache.headroom = "10%"` does +today, so an authenticated publisher can't grow the relay without bound by +opening tracks. Setting `capacity` or `headroom` explicitly still works, and +an operator can still choose an unbounded pool explicitly. + +## Plan + +Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control +audit): ✅ default `headroom` to 10% (rejected: a 50% `capacity`, which +ignores the rest of the process's memory; both). + +Receive credit bounds only what a connection holds in the transport and in +unfinished frames. A finished frame moves into the shared cache, which is +unbounded in bytes when neither knob is set (`rs/moq-relay/src/cache.rs`). +Each track's live group is exempt from the 30 s idle window and holds up to +its 32 MiB tail, so a publisher holds about 32 MiB per track it opens. Under +pool debt a group sheds its front sooner, so a byte budget bounds it. + +Decide how an operator spells "unbounded" without a magic value in the +setting, per AGENTS.md (for example an explicit `capacity = "100%"`, or +`headroom = "0%"`), and say it in the relay docs and the config reference. +Measure process RSS with several publishers opening many tracks against a +small headroom, with the default, so the bound is shown on the built binary. + +Behavior change, no `!`. Public API: the relay config default. Wire: none. +`## Caused by`: as old as the cache pool, no single PR. diff --git a/quest/m3/release/cut.md b/quest/m3/release/cut.md index 5c3fe9b367..9e1aeced74 100644 --- a/quest/m3/release/cut.md +++ b/quest/m3/release/cut.md @@ -231,6 +231,16 @@ Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): cut gate's "hand-written `moq-c`" below reads `libmoq`, and the two quests for a code-free `libmoq` stub were deleted. +Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control +audit): ✅ the cut waits on the [credit model](/quest/m1/credit-model.md), +[#9210](https://github.com/moq-dev/moq/pull/9210)'s +[paused-stream allowance](/quest/m1/paused-allowance.md), which merges once +the model passes on it, and a [bounded relay cache by +default](/quest/m1/relay-memory-default.md). Rejected: cutting with flow +control unverified; landing #9210 after the cut. The +[JS frame cap](/quest/m1/js-frame-cap.md) ships with the cut only if it +lands first. + On the reconciled candidate, regenerate the Nix build plan with `just rs cargo-nix` after the shared version bump. A dry run of the bump to 0.22.0 updated 52 packages and left Cargo.nix stale. Repeat the two-relay smoke with the candidate and the last published relay. Put each binary upstream @@ -249,4 +259,7 @@ check that Release remeasure opened or updated its pin PR, as RELEASE.md describ - [Cargo feature review](/quest/m3/release/feature-flags.md) - every Cargo feature was judged against the feature rules, and approved changes landed - [Interop and Video pass on main](/quest/m0/interop-green/README.md) - the cut PR's release gate runs `interop.yml`, which failed on `main` every day through 2026-11-12 - [Nightly passes on main](/quest/m0/nightly/README.md) - the release gate runs `features.yml`, red in the 2026-11-12 Nightly +- [Credit model](/quest/m1/credit-model.md) - receive credit, pausing, and the sender's credit wait never deadlock +- [Paused-stream allowance](/quest/m1/paused-allowance.md) - #9210 merges with its design checked +- [Relay memory default](/quest/m1/relay-memory-default.md) - a stock relay's cache is bounded - [Claim new npm names](/quest/m3/release/npm-first-publish.md) - trusted publishing can publish every new `@moq/*` package