From 88e631a2f7f683a37730c64be263bfab680f2d35 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 20 Nov 2026 16:31:31 -0800 Subject: [PATCH 1/3] chore(quest): claim quest/m1/credit-model.md Quest-Claim: 0dd4107cdbd239cd From cd803b3b2a5215b36fe2aa59324fa04c1778c9f5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 20 Nov 2026 20:00:34 -0800 Subject: [PATCH 2/3] verify: model receive and send credit with TLA+ ReceiveCredit.tla models one connection's receive credit, from moq-quic's leases and stuck-window eviction to moq-net's paced ingest, on main's design and on moq-dev/moq#9210's slot allowances. SendCredit.tla models a group's wait for stream credit in the session's line. Both check their safety rules and that every stream ends, by a progress measure TLC checks as fast as an invariant. #9210's claim that every pause is charged its slot's allowance is an open finding. The verify-credit skill maps each step to its code, and the credit configs run nightly on a leg of their own. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-credit/SKILL.md | 102 ++++ .github/workflows/nightly.yml | 14 +- quest/m1/credit-model.md | 70 --- quest/m1/paused-allowance.md | 18 +- quest/m3/release/cut.md | 3 +- rs/moq-net/verify/credit/ReceiveCredit.cfg | 25 + rs/moq-net/verify/credit/ReceiveCredit.tla | 511 ++++++++++++++++++ .../verify/credit/ReceiveCreditAllowance.cfg | 22 + .../verify/credit/ReceiveCreditGuards.cfg | 18 + .../verify/credit/ReceiveCreditMain.cfg | 24 + .../verify/credit/ReceiveCreditSegments.cfg | 24 + .../verify/credit/ReceiveCreditThree.cfg | 24 + .../verify/credit/ReceiveCreditWide.cfg | 24 + rs/moq-net/verify/credit/SendCredit.cfg | 17 + rs/moq-net/verify/credit/SendCredit.tla | 257 +++++++++ .../verify/credit/SendCreditPriority.cfg | 18 + .../verify/credit/SendCreditStreams.cfg | 16 + 17 files changed, 1103 insertions(+), 84 deletions(-) create mode 100644 .claude/skills/verify-credit/SKILL.md delete mode 100644 quest/m1/credit-model.md create mode 100644 rs/moq-net/verify/credit/ReceiveCredit.cfg create mode 100644 rs/moq-net/verify/credit/ReceiveCredit.tla create mode 100644 rs/moq-net/verify/credit/ReceiveCreditAllowance.cfg create mode 100644 rs/moq-net/verify/credit/ReceiveCreditGuards.cfg create mode 100644 rs/moq-net/verify/credit/ReceiveCreditMain.cfg create mode 100644 rs/moq-net/verify/credit/ReceiveCreditSegments.cfg create mode 100644 rs/moq-net/verify/credit/ReceiveCreditThree.cfg create mode 100644 rs/moq-net/verify/credit/ReceiveCreditWide.cfg create mode 100644 rs/moq-net/verify/credit/SendCredit.cfg create mode 100644 rs/moq-net/verify/credit/SendCredit.tla create mode 100644 rs/moq-net/verify/credit/SendCreditPriority.cfg create mode 100644 rs/moq-net/verify/credit/SendCreditStreams.cfg diff --git a/.claude/skills/verify-credit/SKILL.md b/.claude/skills/verify-credit/SKILL.md new file mode 100644 index 0000000000..3e29acb131 --- /dev/null +++ b/.claude/skills/verify-credit/SKILL.md @@ -0,0 +1,102 @@ +--- +name: verify-credit +description: Model-check receive and send credit (`rs/moq-quic/src/connection/streams/lease.rs` leases and `evict_if_stuck`, `rs/moq-net/src/coding/reader.rs` pacing, `transport/boxed.rs` pausing, `lite/publisher.rs` `TrackRun::opening`, `transport/open.rs`) with TLA+. Use when a change touches how a lease returns credit, `MAX_STREAM_DATA`, stuck-window eviction (`evicted_held`, `trim_unlooked`, `unlooked`), the pace gate or where ingest pauses a stream, the paused budget (`Budgeted::pause`, `reserve`, the half-window cap, slot allowances, `set_window`, `shape`), or how a group waits for stream credit (`Lines::make_room`, `give_way`, `shed`, `stuck`), or when a credit hang needs every ordering of arrivals, reads, pauses, and evictions checked rather than a random sweep. +--- + +# Verify credit + +Two models, one per direction of a connection's flow control. + +`rs/moq-net/verify/credit/ReceiveCredit.tla` models one connection's receive credit, from moq-quic's leases and stuck-window eviction to moq-net's ingest, which paces each group stream by its group's fastest reader. Bytes are units. The peer sends a unit once stream and connection credit allow, and it arrives in order. A reader leases what arrived and takes one chunk, at most to the next segment seal. It then hands the lease back: released at a seal, which returns the credit, or restored short of one, which keeps it. A grant is queued as moq-quic queues it, and the driver computes its limit when it builds the packet, at any later point (`Transmit`). + +Ingest checks the group's pace gate. Past the pace window, the stream reserves the credit it holds from the session's paused budget (half the connection window) and parks until the reader catches up, the reader leaves, or the stream is evicted. Refused a reservation, it reads on. `Shaped` selects moq-dev/moq#9210's design, `FALSE` main's: + +- #9210 gates only at a seal, charges a pause what the stream holds once shaped to its slot's allowance (`cap / ((i + 1)(i + 2))`), and shapes each stream's window to its group's room plus that allowance. +- Main gates on every turn and charges a stream window plus what the stream took but has not released. + +When the peer has no connection credit left, moq-quic evicts the lowest-ranked stream holding credit, newest first. It holds off while an evicted reader has not stopped its stream, or while some reader has data it has not looked at. A reader arrives late or leaves only under `Late` and `Leave`. + +`rs/moq-net/verify/credit/SendCredit.tla` models the sender's wait for stream credit. Each subscription's `TrackRun` hands its groups out one at a time, in any arrival order. One group per subscription waits for credit (`TrackRun::opening`) in the session's line, ranked as `open::Rank` orders it. Only the head asks the transport, and a waiting open resets the lowest-ranked stream it preempts (`Lines::make_room`). A group handed out while another waits replaces it if it outranks it (`give_way`), and is shed otherwise. While the waiting group is `stuck`, the cursor hands out only groups that may move it on. The receiver is abstract: a stream sent whole or reset is eventually freed, and ReceiveCredit.tla checks that it is. + +Each step is one poll of the code that handles one event, under the connection's lock where the code takes it. TLC checks every ordering under the bounds in each config. + +## Rules + +| Rule | Model | Holds when | Harm if broken | +| --- | --- | --- | --- | +| `Books` | Receive | Every unit arrived is held by one stream or was credited back. | Credit leaks, and the connection window shrinks for good. | +| `WithinWindow` | Receive | What all streams hold together (unread units, leased units not released, a frame partly in) stays within the connection window. | The peer is promised credit the receiver never returns: a flow-control error or a hang. | +| `PausedWithinCap` | Receive | Paused streams hold at most what they reserved, and that at most half the window. | Paused streams starve the rest of the connection's credit. | +| `evict`'s assertion | Receive | Eviction never picks a stream whose reader may still progress (`Assert` in `EvictLoop`). | A reader that would have finished loses its group. | +| `Progress` and `Ends` | Receive | Every stream is read whole or evicted, and stopped (`AllEnd`). | A stream holds credit forever: the connection hangs. | +| `WithinCredit` | Send | Never more streams open than the peer granted. | The peer closes the connection with STREAM_LIMIT_ERROR. | +| `Progress` and `Ends` | Send | Every group waiting for credit opens or is given up, and every stream ends. | A subscription stops sending behind a group that never opens. | + +TLC's own liveness check is about a thousand times slower here, so liveness is checked through safety. `Progress` is an action property: every step other than the final stutter raises a bounded measure, so every behavior ends stuttering in one state. `Ends` is an invariant: every state is done, or one of the system's steps is enabled there, which weak fairness takes. `Ends` reads each step's guard in place of ENABLED, and `Guarded` checks the two agree on a smaller config. The comment above `Lifecycle` in ReceiveCredit.tla gives the measure. + +#9210's claim that every stream pauses, charged its slot's allowance, is `EveryPauseFits` and `ChargedAllowance`, which ReceiveCreditAllowance.cfg checks as an open finding (see below). + +Each model has witnesses (`NeverPaused`, `NeverEvicted`, `NeverStuckWaiting`, `NeverPartial`, `NeverPausedInside`; `NeverPreempts`, `NeverWaits`, `NeverGivesWay`). Each one is expected to fail, which shows TLC reaches the state. Check one after changing a step's guard. + +## Run + +```sh +quest gate bash sh/tla.sh rs/moq-net/verify/credit +quest gate just rs tla moq-net/verify/credit/ReceiveCredit.tla -config rs/moq-net/verify/credit/ReceiveCreditMain.cfg +``` + +| Config | Covers | States | Time | +| --- | --- | --- | --- | +| `ReceiveCredit.cfg` | #9210, two streams of two one-segment frames, a window two streams exhaust, readers late or leaving | 813k | 21 s | +| `ReceiveCreditMain.cfg` | main's design, as above | 363k | 7 s | +| `ReceiveCreditSegments.cfg` | #9210, frames of two segments, so a reader arriving mid-frame closes the gate there | 1.59M | 33 s | +| `ReceiveCreditWide.cfg` | #9210, a wider window, where the two slots' allowances differ | 373k | 12 s | +| `ReceiveCreditThree.cfg` | #9210, three streams, readers from the start | 52.4M | 21 min | +| `ReceiveCreditGuards.cfg` | `Guarded`, on one-unit frames | 23.8k | 9 min | +| `ReceiveCreditAllowance.cfg` | open finding: #9210's allowance claims | fails | 4 s | +| `SendCredit.cfg` | two subscriptions, one newest first and one oldest first, one stream of credit | 89k | 6 s | +| `SendCreditPriority.cfg` | one subscription at a higher priority, so its opens preempt; also `Guarded` | 99k | 46 s | +| `SendCreditStreams.cfg` | two streams of credit, the orders swapped | 296k | 14 s | + +Every config passes but the open finding. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. + +The open finding: on #9210, `Stream::release` queues `MAX_STREAM_DATA` under the connection lock. The shrink to the slot's allowance (`shape`, or `set_window` in `pause`) takes the lock again later. A driver on another thread can build the packet in between, granting the old window. The pause is then charged up to a stream window rather than its allowance, and a later pause can be refused under the cap. That stream reads on and sheds, as on main. Neither the window nor the cap is broken. The config passes once the driver runs only between polls, or once the shrink happens in the release's own lock hold. Promote both rules into the other configs once #9210 does that. + +A second gap is inherent, so it is no rule: a group with no reader, whose writer ran past the pace window, has no room (`Pace::room` is None). Its stream is granted a whole stream window. A reader arriving later closes the gate, and the pause is charged that credit, up to a stream window. + +## Reading a failure + +Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the code: + +- the moq-quic half with the stuck-window tests in `connection/streams/state.rs` (`stuck`, `lease_out_holds_eviction_off`, `evicted_taken_bytes_keep_credit_until_stop`); +- an ordering against the packet builder with `tests/sim.rs`; +- ingest's half with `Credited` in `coding/reader.rs` tests; +- the send half with the `Opener` tests in `transport/open.rs` and `TrackRun`'s tests in `lite/publisher.rs`. + +In a receive trace, follow `pc`, `q[s]`'s `arr`, `taken`, `rel`, and `msd`, `charge`, and `slot` per stream. + +## Coverage + +Sync commit: `ef832d9bcc96e06e95a7e005f03f36c88ac6dc4c`; with `Shaped`, moq-dev/moq#9210 at `69dc410be`. Run `git log ef832d9bcc.. -- rs/moq-quic/src/connection/streams rs/moq-net/src/coding/reader.rs rs/moq-net/src/transport/boxed.rs rs/moq-net/src/transport/open.rs rs/moq-net/src/model/pace.rs rs/moq-net/src/model/frame.rs rs/moq-net/src/lite/publisher.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. + +| Code | Model | +| --- | --- | +| `rs/moq-quic/src/connection/streams/state.rs` `StreamsState::received` | `Send` | +| `rs/moq-quic/src/connection/streams/state.rs` `StreamsState::write_control_frames`; `rs/moq-quic/src/connection/streams/recv.rs` `Recv::max_stream_data`, `Recv::can_send_flow_control`, `Recv::sent_max_stream_data` (#9210: `Recv::set_window`, `Recv::credit`) | `Transmit`, `Limit`, `CanGrant` (`Credit`) | +| `rs/moq-quic/src/connection/streams/lease.rs` `poll_lease`, `restore`, `release`, `queue_max_stream_data`, `Leased::release` | `Lease`, `HandBack`, `Queue` | +| `rs/moq-quic/src/connection/streams/lease.rs` `evict_if_stuck`, `trim_unlooked`; `rs/moq-quic/src/connection/streams/state.rs` `StreamsState::ranks`, `StreamsState::unlooked`, `StreamsState::stuck_waiting`, `StreamsState::evicted_held` | `Pass`, `EvictLoop`, `Evict`, `Unlooked`, `Order`, `stuckWaiting` | +| `rs/moq-net/src/coding/reader.rs` `Reader::poll_read_frame`, `Reader::poll_admit`, `Reader::poll_pace` | `Step`, `Park`, `Write` | +| `rs/moq-net/src/model/frame.rs` `Producer::poll_ready`, `Producer::unsealed`; `rs/moq-net/src/model/pace.rs` `window`, `Pace::paced`, `Pace::ahead`, `Pace::took`, `Pace::park` (#9210: `Pace::room`) | `Paced`, `Unsealed`, `PaceWindow`, `Take` (`Room`) | +| `rs/moq-net/src/transport/boxed.rs` `Budgeted::window`, `Budgeted::pause`, `Budgeted::held`, `Budgeted::release`, `Shared::reserve` (#9210: `Pacing::cap`, `Pacing::allowance`, `Slots::take`, `Budgeted::shape`) | `PaceWindow`, `Charge`, `charge`, `Stop` (`Cap`, `Allowance`, `LowestFree`, `Shape`) | +| `rs/moq-net/src/lite/publisher.rs` `TrackRun::opening`, `GroupServe::poll_open`, `TrackRun::stuck`, `TrackRun::give_way`, `TrackRun::shed`, `TrackRun::poll_step` | `PollOpen`, `Stuck`, `Admitted`, `Cursor`, `Step` | +| `rs/moq-net/src/transport/open.rs` `Rank::preempts`, `Line::join`, `Line::leave`, `Line::wake_head`, `Line::resets`, `Lines::make_room`; `rs/moq-net/src/priority.rs` `in_group` | `Rank`, `Above`, `Join`, `Leave`, `Preempts`, `MakeRoom`, `InGroup` | + +What the model relies on in that code: + +- A stopped stream returns its paused credit and frees its slot (`Stop`), so a reader that never returns cannot pin the paused budget. +- Eviction passes over a reader that may still progress, and an evicted reader's taken units keep their credit until it stops (`evicted_held`). +- Only the head of the line asks the transport, and leaving the line wakes the next head (`Line::leave`). + +## Not covered + +Both models leave out loss, gaps, and reordering within a stream: every unit arrives in order. They also leave out rate caps, shrink debt, frame headers' bytes, and the peer resetting a stream on the receive model. Every stream ranks at one priority on the receive side, so eviction order is newest first. Relay chains are left out, since pausing is per hop. So are the front's shedding past a pause and a group's expiry. The send side has two priority levels (`High` and the rest). TLC's simulate mode crashes on ReceiveCredit.tla (an `ArrayIndexOutOfBoundsException` in TLC), so larger shapes run exhaustively, as ReceiveCreditThree.cfg does. A pass covers two streams, or three with readers held from the start, of two or three frames, and two subscriptions of three groups. diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 9ccfa2974d..4d62f118ba 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -488,17 +488,21 @@ jobs: strategy: fail-fast: false matrix: - # The largest model gets a runner of its own, as does the warm model - # (its configs take about an hour together); the `rest` leg takes - # every other config, so a new one is checked without editing this - # list. sh/gh/nightly.test.py fails on a path that no longer exists. + # The largest model gets a runner of its own, as do the warm model + # (its configs take about an hour together) and the credit models + # (about 35 minutes); the `rest` leg takes every other config, so a + # new one is checked without editing this list. A new leg goes last, + # since each leg's cache key holds its index. + # sh/gh/nightly.test.py fails on a path that no longer exists. include: - name: resume paths: rs/moq-net/verify/resume - name: warm paths: rs/moq-net/verify/warm - name: rest - paths: ":!rs/moq-net/verify/resume :!rs/moq-net/verify/warm" + paths: ":!rs/moq-net/verify/resume :!rs/moq-net/verify/warm :!rs/moq-net/verify/credit" + - name: credit + paths: rs/moq-net/verify/credit steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/quest/m1/credit-model.md b/quest/m1/credit-model.md deleted file mode 100644 index a0dc11b8e6..0000000000 --- a/quest/m1/credit-model.md +++ /dev/null @@ -1,70 +0,0 @@ -# [L] A TLA+ model shows receive credit never deadlocks - -## Goal - -A TLA+ model of one connection's flow control, run by TLC in CI like the -other `rs/moq-net/verify` models, checks every ordering of arrivals, reads, -pauses, and evictions for two properties: - -- Safety: the credit every stream holds together (unread bytes, leased bytes - not yet released, unfinished frames, paused allowances) never exceeds the - connection window, and paused credit never exceeds its cap. -- Liveness: under weak fairness for the reader and the peer, every stream - eventually progresses, finishes, or is evicted, and a sender waiting for - stream credit eventually opens or gives up its group. - -It models the design as [#9210](https://github.com/moq-dev/moq/pull/9210) -leaves it, and the code matches the model, or the PR fixes it. - -## Plan - -Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control audit): - -- ✅ The cut waits on a model (rejected: restoring the overcommitted benchmark - cells as the evidence, which only samples orderings; cutting without it). - At least four credit bugs landed in the two weeks before the audit, two of - them hangs: [#8794](https://github.com/moq-dev/moq/pull/8794) (a paused - stream granted credit until it held the window), - [#8808](https://github.com/moq-dev/moq/pull/8808) (QMux leaked receive - credit on `release`), [#8774](https://github.com/moq-dev/moq/pull/8774) - (a dropped group never reached the reader), and - [#8607](https://github.com/moq-dev/moq/pull/8607) (a group waiting for - credit held a higher-ranked one behind it). -- ✅ The model comes before #9210, which merges once the model passes on its - design (rejected: #9210 first, or both in parallel). -- ✅ Scope: receive credit, pausing, and the sender's credit wait (rejected: - receive side only; a relay chain end to end, since pausing is per hop). - -What to model: - -- Receive, `rs/moq-quic/src/connection/streams/lease.rs`: the connection and - stream windows, leases and their release at a segment seal, and - `evict_if_stuck` with its two waits: an evicted reader that has not stopped - its stream (`evicted_held`), and data a reader has not looked at - (`trim_unlooked`). The second is the likeliest hang: it holds eviction off - for as long as any stream has data no reader looked at, so liveness depends - on moq-net always polling every stream it accepted, including ones not yet - accepted. -- Pausing, `rs/moq-net/src/transport/boxed.rs` and `model/pace.rs`: the - half-window cap, #9210's slot allowances (`cap / ((i + 1) * (i + 2))`) and - `set_window`, the pause only at a segment seal, and the read-on-and-shed - path past the cap. -- Send, `rs/moq-net/src/lite/publisher.rs` `TrackRun::opening` and `stuck`, - and `transport/open.rs`: a group waiting for stream credit, by order and - rank. - -Abstract bytes into a few units, and keep frames, streams, and slots small -enough for TLC to run in the merge gate, as the other models do; a larger -configuration can run nightly. Add a `verify-credit` skill beside -`verify-subscribe` and the others, naming the code each action maps to. - -A violation in main's code is fixed in this PR. A violation in #9210's -design goes to #9210 before it merges. - -Public API: none. Wire: none. - -## Related - -- [Paused-stream allowance](/quest/m1/paused-allowance.md) - the design this model checks before it merges -- [Overcommitted receive-window cells](/quest/m3/receive-windows-evicted.md) - the benchmark side of the same mechanism -- [Send-credit stall counters](/quest/m3/send-credit-stalls.md) - what an operator sees when the sender waits on credit diff --git a/quest/m1/paused-allowance.md b/quest/m1/paused-allowance.md index 559567d8a6..dd614444a7 100644 --- a/quest/m1/paused-allowance.md +++ b/quest/m1/paused-allowance.md @@ -3,10 +3,10 @@ ## Goal [#9210](https://github.com/moq-dev/moq/pull/9210) merges with its design -checked by the [credit model](/quest/m1/credit-model.md): each paused stream -holds a slot-sized allowance of credit instead of a stream window, so any -number of streams pause within half the connection window. #9210 deletes -this quest. +checked by the credit model (`rs/moq-net/verify/credit`, the `verify-credit` +skill): each paused stream holds a slot-sized allowance of credit instead of a +stream window, so any number of streams pause within half the connection +window. #9210 deletes this quest. ## Plan @@ -19,6 +19,10 @@ first and modeling the result). items stay on the PR. When the model finds a violation in its design, fix it there before merging. -## Required - -- [Credit model](/quest/m1/credit-model.md) - checks the slot allowances and `set_window` before they ship +The model passes on #9210 at 69dc410be: the window, the cap, and every stream +ending hold (`verify-credit`). Its open finding, `ReceiveCreditAllowance.cfg`, +is #9210's own claim that every pause is charged its slot's allowance. A grant +queued by `Stream::release` can go out before the next lock shrinks the window, +so a pause is charged up to a stream window and a later one can be refused +under the cap. Shrinking in the release's own lock hold passes it; then +promote `EveryPauseFits` and `ChargedAllowance` into the other configs. diff --git a/quest/m3/release/cut.md b/quest/m3/release/cut.md index 9e1aeced74..50847eb783 100644 --- a/quest/m3/release/cut.md +++ b/quest/m3/release/cut.md @@ -232,7 +232,7 @@ Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): quests for a code-free `libmoq` stub were deleted. Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control -audit): ✅ the cut waits on the [credit model](/quest/m1/credit-model.md), +audit): ✅ the cut waits on the credit model (`verify-credit`), [#9210](https://github.com/moq-dev/moq/pull/9210)'s [paused-stream allowance](/quest/m1/paused-allowance.md), which merges once the model passes on it, and a [bounded relay cache by @@ -259,7 +259,6 @@ check that Release remeasure opened or updated its pin PR, as RELEASE.md describ - [Cargo feature review](/quest/m3/release/feature-flags.md) - every Cargo feature was judged against the feature rules, and approved changes landed - [Interop and Video pass on main](/quest/m0/interop-green/README.md) - the cut PR's release gate runs `interop.yml`, which failed on `main` every day through 2026-11-12 - [Nightly passes on main](/quest/m0/nightly/README.md) - the release gate runs `features.yml`, red in the 2026-11-12 Nightly -- [Credit model](/quest/m1/credit-model.md) - receive credit, pausing, and the sender's credit wait never deadlock - [Paused-stream allowance](/quest/m1/paused-allowance.md) - #9210 merges with its design checked - [Relay memory default](/quest/m1/relay-memory-default.md) - a stock relay's cache is bounded - [Claim new npm names](/quest/m3/release/npm-first-publish.md) - trusted publishing can publish every new `@moq/*` package diff --git a/rs/moq-net/verify/credit/ReceiveCredit.cfg b/rs/moq-net/verify/credit/ReceiveCredit.cfg new file mode 100644 index 0000000000..dfee29c56f --- /dev/null +++ b/rs/moq-net/verify/credit/ReceiveCredit.cfg @@ -0,0 +1,25 @@ +\* moq-dev/moq#9210's design on two streams of two frames, a frame one segment, under a +\* window that two streams exhaust: units arrive in every order, readers arrive late or leave, +\* and the driver sends grants whenever it likes. +CONSTANTS + N = 2 + Frames = 2 + FrameLen = 2 + Seg = 2 + W = 4 + SW = 2 + Ahead = 2 + Late = TRUE + Leave = TRUE + Shaped = TRUE + +SPECIFICATION Spec + +INVARIANTS + Books + WithinWindow + PausedWithinCap + Ends + +PROPERTIES + Progress diff --git a/rs/moq-net/verify/credit/ReceiveCredit.tla b/rs/moq-net/verify/credit/ReceiveCredit.tla new file mode 100644 index 0000000000..a9af493464 --- /dev/null +++ b/rs/moq-net/verify/credit/ReceiveCredit.tla @@ -0,0 +1,511 @@ +---------------------------- MODULE ReceiveCredit ---------------------------- +(***************************************************************************) +(* One connection's receive credit, between moq-quic's leases and stuck- *) +(* window eviction (rs/moq-quic/src/connection/streams/lease.rs) and *) +(* moq-net's ingest, which paces each group stream by its group's fastest *) +(* reader (rs/moq-net/src/coding/reader.rs, transport/boxed.rs, *) +(* model/pace.rs). *) +(* *) +(* Bytes are units. The peer sends a unit on a stream once its stream and *) +(* connection credit allow; it arrives at once and in order, so no gap *) +(* holds a reader. Every stream carries one group of `Frames` frames of *) +(* `FrameLen` units, sealed every `Seg` units. A reader leases what *) +(* arrived, takes one chunk of it (at most to the next seal, or the whole *) +(* rest of the frame), and hands the lease back: released at a seal, which *) +(* returns the credit of every unit taken, restored short of one, which *) +(* keeps it. Credit returns to the peer at once. *) +(* *) +(* Ingest checks its group's pace gate at every seal: the writer may run *) +(* at most the pace window past the group's fastest reader. Past it, the *) +(* stream pauses: it reserves the credit it holds from the session's *) +(* paused budget, half the connection window, and is not read until the *) +(* reader catches up, the reader leaves, or the stream is evicted. Refused *) +(* a reservation, it reads on. Ingest shapes each stream's window to the *) +(* room its group has left plus the allowance of the stream's slot, as *) +(* moq-dev/moq#9210 designs it. *) +(* *) +(* When the peer has no connection credit left, moq-quic evicts the lowest *) +(* ranked stream holding credit, unless an evicted reader has yet to stop *) +(* its stream or some reader has data it has not looked at. *) +(* *) +(* Each step is one poll of the code that handles one event, under the *) +(* connection's lock where the code takes it. A task parks until an event *) +(* it registered for wakes it. *) +(***************************************************************************) +EXTENDS Integers, FiniteSets, TLC + +CONSTANTS + N, \* streams, numbered in the order the peer opens them + Frames, \* frames per stream + FrameLen, \* units per frame + Seg, \* units per segment: a frame's credit returns at each seal + W, \* the connection receive window + SW, \* the stream receive window + Ahead, \* `pace::AHEAD`, the most a writer runs ahead of its fastest reader + Late, \* a group's first reader may arrive after its writer started + Leave, \* a group's last reader may leave it + Shaped \* moq-dev/moq#9210's design; FALSE for main's + +ASSUME FrameLen % Seg = 0 /\ N >= 1 /\ Frames >= 1 /\ Seg >= 1 /\ SW >= 1 /\ W >= 1 + +Streams == 1..N +None == -1 +Min(a, b) == IF a < b THEN a ELSE b +Max(a, b) == IF a > b THEN a ELSE b + +RECURSIVE SumTo(_, _) +SumTo(f(_), n) == IF n = 0 THEN 0 ELSE f(n) + SumTo(f, n - 1) + +Len == Frames * FrameLen + +\* `Budgeted::window`: `pace::window`, the stream window capped at AHEAD and the connection +\* window; on a transport that shapes credit, plus a stream window, capped at AHEAD. +PaceWindow == + LET window == Min(Min(SW, Ahead), W) IN + IF Shaped THEN Min(window + SW, Ahead) ELSE window + +\* `Pacing::cap`: what the session's paused streams may hold together. +Cap == W \div 2 + +\* `Pacing::allowance`: the credit a stream in `slot` may hold past its group's room. +Allowance(slot) == Min(Cap \div ((slot + 1) * (slot + 2)), SW) + +\* `StreamsState::ranks`: the lowest priority first, then the newest. Every stream ranks at one +\* priority here, so the newest goes first. +Order == [i \in Streams |-> N + 1 - i] + +VARIABLES + \* moq-quic, one record per receive stream (`Recv` and its `Leased`) + q, + \* moq-quic, the connection: credit returned to the peer, and `stuck_waiting` + credited, + stuckWaiting, + \* moq-net ingest, per stream: where its task is, and whether a wake is pending + pc, \* "new" (not accepted), "loop" (reading), "out" (a lease in hand), "write" + \* (the chunk not yet in the frame), "paused", or "done" (stopped) + woken, + charge, \* the paused credit it reserved (`Budgeted::held`), or None + slot, \* its slot among the session's shaped streams, or None + \* the group + wr, \* units its writer wrote + rd, \* where the last frame its fastest reader took ends + held, \* whether a reader holds it: "late" (none yet), "held", or "gone" + \* history + refused \* some pause was refused + +vars == <> + +(***************************************************************************) +(* A receive stream in moq-quic. *) +(* arr: units arrived (the stream's `end`) *) +(* filled: units moved into the lease (`assembler.bytes_read`) *) +(* taken: units the reader took and handed the lease back over *) +(* rel: units whose credit the reader released *) +(* hand: units taken from a lease still out *) +(* msd: the stream limit the peer was sent (`sent_max_stream_data`) *) +(* due: a `MAX_STREAM_DATA` is queued (`pending.max_stream_data`) *) +(* win: the window `set_window` last set, SW until then *) +(* want, ready, leased (`filled`), ranked, unl (in `unlooked`), out: *) +(* `Leased`'s flags *) +(* rdr: "reading", "evicted", or "stopped" (`Reader`) *) +(***************************************************************************) +Fresh == [arr |-> 0, filled |-> 0, taken |-> 0, rel |-> 0, hand |-> 0, msd |-> SW, due |-> FALSE, + win |-> SW, want |-> FALSE, ready |-> FALSE, leased |-> FALSE, ranked |-> FALSE, + unl |-> FALSE, out |-> FALSE, rdr |-> "reading"] + +Unread(r) == r.arr - r.filled + +\* `Leased::held`: leased units not yet released, taken or not. +Held(r) == r.filled - r.rel + +\* The connection credit a stream withholds from the peer. +Hold(r) == CASE r.rdr = "reading" -> Unread(r) + Held(r) + [] r.rdr = "evicted" -> Held(r) + [] r.rdr = "stopped" -> 0 + +DataRecvd(qq) == SumTo(LAMBDA s: qq[s].arr, N) +EvictedHeld(qq) == SumTo(LAMBDA s: IF qq[s].rdr = "evicted" THEN Held(qq[s]) ELSE 0, N) +\* The peer has no connection credit left. +Stuck(qq, cr) == W + cr <= DataRecvd(qq) + +\* `unlooked`: a reader that may progress without new credit. +Unlooked(r) == r.rdr = "reading" + /\ ((Unread(r) > 0 /\ (~r.leased \/ r.want)) \/ r.ready \/ r.out) + +\* `Recv::can_send_flow_control`: the FIN, which rides the last unit, has not arrived. +CanGrant(r) == r.arr < Len /\ r.rdr = "reading" + +\* `Recv::max_stream_data`: credit runs from what the reader took, a window past it, or the +\* unit a waiting reader wants. +Limit(r) == Max(r.taken + Max(Min(r.win, SW), IF r.want THEN 1 ELSE 0), r.msd) + +\* `queue_max_stream_data`: queue a `MAX_STREAM_DATA` once the limit grew. The packet that +\* carries it computes the limit again (`Transmit`). +Queue(r) == IF CanGrant(r) /\ Limit(r) > r.msd THEN [r EXCEPT !.due = TRUE] ELSE r + +\* `Recv::credit` (moq-dev/moq#9210): what arrived and was not released, and what the peer +\* may still send; capped at the final size once the FIN arrived. +Credit(r) == IF r.rdr = "reading" + THEN (IF r.arr = Len THEN Min(Limit(r), Len) ELSE Limit(r)) - r.rel + ELSE Held(r) + +(***************************************************************************) +(* `StreamsState::evict_if_stuck`, one atomic pass under the lock. It *) +(* returns every outcome the order of `unlooked`, a hash set, allows: the *) +(* new streams, credit returned, `stuck_waiting`, and the streams woken. *) +(***************************************************************************) +Evict(qq, s) == + [qq EXCEPT ![s].filled = qq[s].taken, ![s].want = FALSE, ![s].ready = FALSE, + ![s].rdr = "evicted", ![s].unl = FALSE] + +RECURSIVE EvictLoop(_, _, _, _) +EvictLoop(qq, cr, woke, i) == + IF i > N \/ ~Stuck(qq, cr) \/ EvictedHeld(qq) > 0 + THEN [q |-> qq, cr |-> cr, woke |-> woke] + ELSE LET s == Order[i] IN + IF ~qq[s].ranked THEN EvictLoop(qq, cr, woke, i + 1) + ELSE LET q1 == [qq EXCEPT ![s].ranked = FALSE] IN + IF q1[s].rdr # "reading" \/ Hold(q1[s]) = 0 + THEN EvictLoop(q1, cr, woke, i) + \* `evict`'s debug assertion: the trim found no such reader. + ELSE IF Assert(~Unlooked(q1[s]), <<"evicted a stream whose reader may progress", s>>) + THEN EvictLoop(Evict(q1, s), cr + q1[s].arr - q1[s].taken, woke \cup {s}, i) + ELSE [q |-> qq, cr |-> cr, woke |-> woke] + +Pass(qq, cr) == + LET found == {s \in Streams : qq[s].unl /\ Unlooked(qq[s])} IN + IF ~Stuck(qq, cr) \/ EvictedHeld(qq) > 0 + THEN {[q |-> qq, cr |-> cr, sw |-> FALSE, woke |-> {}]} + ELSE IF found # {} + THEN {[q |-> qq, cr |-> cr, sw |-> qq[s].out, woke |-> {}] : s \in found} + ELSE LET trimmed == [s \in Streams |-> [qq[s] EXCEPT !.unl = FALSE]] + e == EvictLoop(trimmed, cr, {}, 1) + IN {[q |-> e.q, cr |-> e.cr, sw |-> FALSE, woke |-> e.woke]} + +\* No pass: the books as `qq` and `cr` leave them. +Unpassed(qq, cr) == {[q |-> qq, cr |-> cr, sw |-> stuckWaiting, woke |-> {}]} + +\* Wake the tasks parked on a stream's reads (`Wait::Read`): a lease waiting for data, or a +\* paused stream watching for a reset. +Wake(w, ss) == [s \in Streams |-> w[s] \/ (s \in ss /\ pc[s] \in {"loop", "paused"})] + +\* Apply a pass's outcome `p` on top of a step that leaves `w` as the wakes. +Settle(p, w) == + /\ q' = p.q + /\ credited' = p.cr + /\ stuckWaiting' = p.sw + /\ woken' = Wake(w, p.woke) + +(***************************************************************************) +(* moq-net ingest. *) +(***************************************************************************) +\* The slot `Slots::take` hands out: the lowest no live stream holds. +LowestFree(sl) == CHOOSE i \in 0..N : i \notin {sl[t] : t \in Streams} + /\ \A j \in 0..(i - 1) : j \in {sl[t] : t \in Streams} + +SlotOf(s) == IF slot[s] = None THEN LowestFree(slot) ELSE slot[s] + +PausedTotal(ch) == SumTo(LAMBDA s: IF ch[s] = None THEN 0 ELSE ch[s], N) + +\* `Pace::room`, then `saturating_sub(frame)`: what the writer may still take, `w` units in, +\* before the next frame's gate holds it, or None once nothing holds it back. +Room(s, w) == + LET lead == w - rd[s] + room == IF lead <= PaceWindow THEN PaceWindow - lead + ELSE IF held[s] = "held" THEN 0 ELSE None + IN IF room = None THEN None ELSE Max(room - FrameLen, 0) + +\* `Budgeted::shape`: let the peer send what the group has `room` for, at least the `unsealed` +\* rest of the segment, plus the slot's allowance. Main shapes nothing. +Shape(r, sl, room, unsealed) == + LET a == Allowance(sl) + target == IF room = None THEN SW ELSE Min(Max(room, unsealed) + a, SW) + IN IF Shaped /\ (target < r.win \/ target - r.win >= a \div 2) + THEN Queue([r EXCEPT !.win = target]) + ELSE r + +\* The pace gate, `w` units in: at a frame's start the admit gate, counting the whole next +\* frame; inside one, `Frame::poll_ready`, counting what of it is in. Open once no reader holds +\* the group (`poll_unwanted`). +Paced(s, w) == + LET inside == w % FrameLen + committed == w - inside + partial == IF inside = 0 THEN FrameLen ELSE inside + IN held[s] = "held" /\ rd[s] < committed /\ committed + partial - rd[s] > PaceWindow + +\* `Frame::unsealed` once `w` units of a frame are in: the rest of the segment being written. +Unsealed(w) == + LET inside == w % FrameLen IN + IF inside = 0 /\ w > 0 THEN 0 ELSE Seg - (inside % Seg) + +\* `Budgeted::pause`'s charge: on #9210, the credit the stream holds once shaped to its +\* allowance; on main, a stream window and the units taken but not released. +Charge(r) == IF Shaped THEN Min(Credit(r), W) ELSE Min(SW + r.taken - r.rel, W) + +Accept(s) == + /\ pc[s] = "new" + /\ q[s].arr > 0 + /\ pc' = [pc EXCEPT ![s] = "loop"] + /\ woken' = [woken EXCEPT ![s] = TRUE] + /\ UNCHANGED <> + +\* The stream ends: a reader that read it whole stops it, as does one evicted. The session drops +\* its stream, returning any paused credit and its slot. +Stop(s) == + LET r == q[s] + cr == credited + Held(r) + (IF r.rdr = "reading" THEN Unread(r) ELSE 0) + q1 == [q EXCEPT ![s] = [Fresh EXCEPT !.rdr = "stopped", !.arr = r.arr, !.msd = r.msd]] + IN /\ \E p \in Pass(q1, cr) : Settle(p, [woken EXCEPT ![s] = FALSE]) + /\ pc' = [pc EXCEPT ![s] = "done"] + /\ charge' = [charge EXCEPT ![s] = None] + /\ slot' = [slot EXCEPT ![s] = None] + /\ wr' = [wr EXCEPT ![s] = 0] + /\ rd' = [rd EXCEPT ![s] = 0] + /\ held' = [held EXCEPT ![s] = "gone"] + /\ UNCHANGED refused + +\* `Stream::poll_lease` for one unit, from `r`: hand out what arrived, or wait for it. +Lease(s, r, sl, ch) == + LET filled == [r EXCEPT !.filled = r.arr, !.unl = FALSE, !.leased = TRUE, !.want = FALSE, + !.ready = FALSE] + f == Queue([filled EXCEPT !.ranked = filled.ranked \/ Held(filled) > 0]) + len == f.filled - f.taken + frameRem == FrameLen - (f.taken % FrameLen) + segRem == Seg - (f.taken % Seg) + IN /\ slot' = [slot EXCEPT ![s] = sl] + /\ charge' = [charge EXCEPT ![s] = ch] + /\ IF len > 0 + THEN \E chunk \in 1..len : + LET take == IF chunk >= frameRem THEN frameRem ELSE Min(chunk, segRem) IN + /\ q' = [q EXCEPT ![s] = [f EXCEPT !.out = TRUE, !.hand = take, !.unl = TRUE]] + /\ pc' = [pc EXCEPT ![s] = "out"] + /\ UNCHANGED <> + ELSE LET waiting == Queue([f EXCEPT !.want = TRUE, !.unl = TRUE]) IN + \E p \in Pass([q EXCEPT ![s] = waiting], credited) : + \* Parked, unless the pass evicted the stream itself. + /\ Settle(p, [woken EXCEPT ![s] = p.q[s].rdr = "evicted"]) + /\ pc' = [pc EXCEPT ![s] = "loop"] + +\* Park on the pace gate and the stream's reset (`poll_pace`). +Park(s) == + /\ pc' = [pc EXCEPT ![s] = "paused"] + /\ woken' = [woken EXCEPT ![s] = FALSE] + +\* One turn of `Reader::poll_read_frame`'s loop, or of the frame admission before it. #9210 +\* checks the gate only at a seal, so a paused stream holds no segment's credit; main checks it +\* every turn. +Step(s) == + LET r == q[s] + sealed == r.rel = r.taken + sl == SlotOf(s) + IN + /\ pc[s] \in {"loop", "paused"} + /\ woken[s] + /\ IF r.rdr = "evicted" \/ r.taken = Len + THEN Stop(s) + ELSE IF (sealed \/ ~Shaped) /\ Paced(s, wr[s]) + THEN \* `Budgeted::pause`: reserve the credit the stream holds, or read on. + IF charge[s] # None + THEN Park(s) /\ UNCHANGED <> + ELSE LET shrunk == IF Shaped THEN Queue([r EXCEPT !.win = Allowance(sl)]) ELSE r + ch == Charge(shrunk) + IN IF PausedTotal(charge) + ch <= Cap + THEN /\ Park(s) + /\ q' = [q EXCEPT ![s] = shrunk] + /\ charge' = [charge EXCEPT ![s] = ch] + /\ slot' = [slot EXCEPT ![s] = IF Shaped THEN sl ELSE None] + /\ UNCHANGED <> + ELSE /\ Lease(s, shrunk, IF Shaped THEN sl ELSE None, None) + /\ refused' = TRUE + /\ UNCHANGED <> + ELSE IF sealed \/ ~Shaped + THEN \* The gate is open: resume, and shape the window, at a frame's start for the + \* admission and then for the frame, whose first segment is whole. + LET room == Room(s, wr[s]) + admitted == IF wr[s] % FrameLen = 0 THEN Shape(r, sl, room, 0) ELSE r + IN /\ Lease(s, Shape(admitted, sl, room, Seg), IF Shaped THEN sl ELSE None, None) + /\ UNCHANGED <> + ELSE Lease(s, r, slot[s], charge[s]) /\ UNCHANGED <> + +\* Hand the lease back (`Lease::release` at a seal, a drop short of one, which restores it). +HandBack(s) == + LET r == q[s] + tk == r.taken + r.hand + back == Queue([r EXCEPT !.out = FALSE, !.taken = tk, !.hand = 0]) + released == [back EXCEPT !.rel = tk] + passes == IF tk % Seg = 0 THEN Pass([q EXCEPT ![s] = released], credited + tk - r.rel) + ELSE IF stuckWaiting THEN Pass([q EXCEPT ![s] = back], credited) + ELSE Unpassed([q EXCEPT ![s] = back], credited) + IN /\ pc[s] = "out" + /\ \E p \in passes : Settle(p, woken) + /\ pc' = [pc EXCEPT ![s] = "write"] + /\ UNCHANGED <> + +\* Write the chunk into the frame, committing it once whole, and shape the window to the room +\* left. The lock is taken again, so the driver may have sent a grant since the hand back. +Write(s) == + LET sl == SlotOf(s) + w == q[s].taken + IN /\ pc[s] = "write" + /\ q' = [q EXCEPT ![s] = Shape(q[s], sl, Room(s, w), Unsealed(w))] + /\ wr' = [wr EXCEPT ![s] = w] + /\ pc' = [pc EXCEPT ![s] = "loop"] + /\ slot' = [slot EXCEPT ![s] = IF Shaped THEN sl ELSE None] + /\ UNCHANGED <> + +(***************************************************************************) +(* The driver, the peer, and the group's reader. *) +(***************************************************************************) +\* The driver builds a packet carrying the queued `MAX_STREAM_DATA`, computing its limit now. +Transmit(s) == + /\ q[s].due + /\ q' = [q EXCEPT ![s].due = FALSE, + ![s].msd = IF CanGrant(q[s]) THEN Limit(q[s]) ELSE q[s].msd] + /\ UNCHANGED <> + +\* The peer sends one unit, which arrives: `StreamsState::received`. +Send(s) == + LET r == q[s] + reading == r.rdr = "reading" + arrived == IF reading + THEN [r EXCEPT !.arr = r.arr + 1, !.ready = r.ready \/ r.want, !.want = FALSE, + !.ranked = r.ranked \/ r.leased, !.unl = TRUE] + ELSE [r EXCEPT !.arr = r.arr + 1] + cr == IF reading THEN credited ELSE credited + 1 + IN /\ r.rdr # "stopped" + /\ r.arr < Len + /\ r.arr < r.msd + /\ DataRecvd(q) < W + credited + /\ \E p \in Pass([q EXCEPT ![s] = arrived], cr) : + Settle(p, IF reading THEN Wake(woken, {s}) ELSE woken) + /\ UNCHANGED <> + +\* The group's reader takes the next committed frame, waking a paused writer. +Take(s) == + /\ pc[s] # "done" + /\ held[s] = "held" + /\ rd[s] + FrameLen <= wr[s] - (wr[s] % FrameLen) + /\ rd' = [rd EXCEPT ![s] = rd[s] + FrameLen] + /\ woken' = [woken EXCEPT ![s] = woken[s] \/ pc[s] = "paused"] + /\ UNCHANGED <> + +\* The group's first reader arrives, starting where readers got even. +Arrive(s) == + /\ pc[s] # "done" + /\ held[s] = "late" + /\ held' = [held EXCEPT ![s] = "held"] + /\ UNCHANGED <> + +\* The group's last reader leaves, which the writer's pace gate watches for. +Drop(s) == + /\ Leave + /\ pc[s] # "done" + /\ held[s] = "held" + /\ held' = [held EXCEPT ![s] = "gone"] + /\ woken' = [woken EXCEPT ![s] = woken[s] \/ pc[s] = "paused"] + /\ UNCHANGED <> + +Done == \A s \in Streams : pc[s] = "done" + +Init == + /\ q = [s \in Streams |-> Fresh] + /\ credited = 0 + /\ stuckWaiting = FALSE + /\ pc = [s \in Streams |-> "new"] + /\ woken = [s \in Streams |-> FALSE] + /\ charge = [s \in Streams |-> None] + /\ slot = [s \in Streams |-> None] + /\ wr = [s \in Streams |-> 0] + /\ rd = [s \in Streams |-> 0] + /\ held \in [Streams -> IF Late THEN {"late", "held"} ELSE {"held"}] + /\ refused = FALSE + +\* The steps weak fairness covers: all but a reader arriving or leaving, which may never happen. +SystemOf(s) == Send(s) \/ Transmit(s) \/ Accept(s) \/ Step(s) \/ HandBack(s) \/ Write(s) \/ Take(s) +System == \E s \in Streams : SystemOf(s) + +Next == + \/ System + \/ \E s \in Streams : Arrive(s) \/ Drop(s) + \/ Done /\ UNCHANGED vars + +Spec == Init /\ [][Next]_vars /\ \A s \in Streams : WF_vars(SystemOf(s)) + +(***************************************************************************) +(* Rules. *) +(***************************************************************************) +\* The books balance: every unit arrived is held by one stream or was credited back. +Books == DataRecvd(q) = credited + SumTo(LAMBDA s: Hold(q[s]), N) + +\* What every stream holds together (unread units, leased units not released, the frame partly +\* in) stays within the connection window. +WithinWindow == SumTo(LAMBDA s: Hold(q[s]), N) <= W + +\* Paused streams hold at most what they reserved, and that at most half the window. +PausedWithinCap == + /\ PausedTotal(charge) <= Cap + /\ \A s \in Streams : charge[s] # None => Credit(q[s]) <= charge[s] + +\* moq-dev/moq#9210's claims: every stream pauses, charged its slot's allowance. +EveryPauseFits == ~refused +ChargedAllowance == \A s \in Streams : charge[s] # None => charge[s] <= Allowance(slot[s]) + +\* Every stream is eventually read whole or evicted, and stopped. +AllEnd == \A s \in Streams : <>(pc[s] = "done") + +(***************************************************************************) +(* `AllEnd` by two safety checks, which TLC runs as fast as the rest: its *) +(* liveness search evaluates every step's enabling at every state, a *) +(* thousand times slower here. *) +(* *) +(* `Progress`: every step other than the final stutter raises a bounded *) +(* measure: first the units arrived, filled, taken (counted twice, so a *) +(* chunk handed back outweighs the hand it leaves), in hand, released, and *) +(* written, the frames read, the grants sent, and each stream's and *) +(* reader's lifecycle; then, with those equal, the tasks parked; then the *) +(* grants not queued. So every behavior ends stuttering in one state, and *) +(* weak fairness leaves it only where no step of the system is enabled. *) +(* `Ends`: every such state has every stream done. *) +(***************************************************************************) +Lifecycle(s) == + (CASE pc[s] = "new" -> 0 [] pc[s] = "done" -> 2 [] OTHER -> 1) + \* An eviction discards the leased units, so it outweighs them. + + (CASE q[s].rdr = "reading" -> 0 [] q[s].rdr = "evicted" -> Len + 1 [] OTHER -> 2 * Len + 2) + + (CASE held[s] = "late" -> 0 [] held[s] = "held" -> 1 [] OTHER -> 2) + +\* A done stream counts above anything a live one reaches, since `Stop` clears its books. +Counted == SumTo(LAMBDA s: IF pc[s] = "done" THEN 12 * Len + SW + 6 + ELSE q[s].arr + q[s].filled + 2 * q[s].taken + q[s].hand + q[s].rel + + wr[s] + q[s].msd + rd[s] + Lifecycle(s), N) +Parked == Cardinality({s \in Streams : ~woken[s]}) +Undue == Cardinality({s \in Streams : ~q[s].due}) + +Progress == + [][\/ Done + \/ Counted' > Counted + \/ Counted' = Counted /\ Parked' > Parked + \/ Counted' = Counted /\ Parked' = Parked /\ Undue' > Undue]_vars + +\* Where some step of the system is enabled: each step's guard, since the rest of every step +\* always has a successor (a pass always has an outcome, and a lease a chunk to take). +\* `Guarded` checks the two agree. +Enabled(s) == + \/ q[s].rdr # "stopped" /\ q[s].arr < Len /\ q[s].arr < q[s].msd /\ DataRecvd(q) < W + credited + \/ q[s].due + \/ pc[s] = "new" /\ q[s].arr > 0 + \/ pc[s] \in {"loop", "paused"} /\ woken[s] + \/ pc[s] \in {"out", "write"} + \/ pc[s] # "done" /\ held[s] = "held" /\ rd[s] + FrameLen <= wr[s] - (wr[s] % FrameLen) + +Guarded == (\E s \in Streams : Enabled(s)) = ENABLED System + +Ends == Done \/ \E s \in Streams : Enabled(s) + +\* Witnesses, each expected to fail: TLC reaching them shows the model is not vacuous. +NeverPaused == \A s \in Streams : pc[s] # "paused" +NeverEvicted == \A s \in Streams : q[s].rdr # "evicted" +NeverStuckWaiting == ~stuckWaiting +NeverPartial == \A s \in Streams : q[s].taken = q[s].rel +NeverPausedInside == \A s \in Streams : pc[s] = "paused" => wr[s] % FrameLen = 0 +============================================================================= diff --git a/rs/moq-net/verify/credit/ReceiveCreditAllowance.cfg b/rs/moq-net/verify/credit/ReceiveCreditAllowance.cfg new file mode 100644 index 0000000000..3373bf3e53 --- /dev/null +++ b/rs/moq-net/verify/credit/ReceiveCreditAllowance.cfg @@ -0,0 +1,22 @@ +\* Open finding: moq-dev/moq#9210 charges a pause more than its slot's allowance, and so can refuse +\* one under the cap, when the driver sends the grant a hand back queued before the reader's next +\* lock shrinks the window. Readers hold each group from the start, which leaves only that race: +\* with the driver held to the gaps between polls, or the shrink made in the hand back's own +\* lock, this config passes. Promote both rules once #9210 shrinks in that lock. +CONSTANTS + N = 2 + Frames = 2 + FrameLen = 2 + Seg = 2 + W = 4 + SW = 2 + Ahead = 2 + Late = FALSE + Leave = FALSE + Shaped = TRUE + +SPECIFICATION Spec + +INVARIANTS + EveryPauseFits + ChargedAllowance diff --git a/rs/moq-net/verify/credit/ReceiveCreditGuards.cfg b/rs/moq-net/verify/credit/ReceiveCreditGuards.cfg new file mode 100644 index 0000000000..ed32bdb2db --- /dev/null +++ b/rs/moq-net/verify/credit/ReceiveCreditGuards.cfg @@ -0,0 +1,18 @@ +\* `Ends` reads each step's guard in place of TLC's ENABLED, which is a thousand times slower: +\* this checks the two agree, on a smaller model. +CONSTANTS + N = 2 + Frames = 2 + FrameLen = 1 + Seg = 1 + W = 2 + SW = 1 + Ahead = 1 + Late = TRUE + Leave = TRUE + Shaped = TRUE + +SPECIFICATION Spec + +INVARIANTS + Guarded diff --git a/rs/moq-net/verify/credit/ReceiveCreditMain.cfg b/rs/moq-net/verify/credit/ReceiveCreditMain.cfg new file mode 100644 index 0000000000..362e3da23e --- /dev/null +++ b/rs/moq-net/verify/credit/ReceiveCreditMain.cfg @@ -0,0 +1,24 @@ +\* Main's design, as ReceiveCredit.cfg: a paused stream reserves a stream window and what it +\* took but has not released, and pauses at any point. +CONSTANTS + N = 2 + Frames = 2 + FrameLen = 2 + Seg = 2 + W = 4 + SW = 2 + Ahead = 2 + Late = TRUE + Leave = TRUE + Shaped = FALSE + +SPECIFICATION Spec + +INVARIANTS + Books + WithinWindow + PausedWithinCap + Ends + +PROPERTIES + Progress diff --git a/rs/moq-net/verify/credit/ReceiveCreditSegments.cfg b/rs/moq-net/verify/credit/ReceiveCreditSegments.cfg new file mode 100644 index 0000000000..6736b6073e --- /dev/null +++ b/rs/moq-net/verify/credit/ReceiveCreditSegments.cfg @@ -0,0 +1,24 @@ +\* moq-dev/moq#9210's design with frames of two segments, so a frame's first segment returns +\* its credit before the frame ends, and a reader arriving mid-frame closes the gate there. +CONSTANTS + N = 2 + Frames = 2 + FrameLen = 4 + Seg = 2 + W = 4 + SW = 2 + Ahead = 2 + Late = TRUE + Leave = FALSE + Shaped = TRUE + +SPECIFICATION Spec + +INVARIANTS + Books + WithinWindow + PausedWithinCap + Ends + +PROPERTIES + Progress diff --git a/rs/moq-net/verify/credit/ReceiveCreditThree.cfg b/rs/moq-net/verify/credit/ReceiveCreditThree.cfg new file mode 100644 index 0000000000..8eb913ab14 --- /dev/null +++ b/rs/moq-net/verify/credit/ReceiveCreditThree.cfg @@ -0,0 +1,24 @@ +\* moq-dev/moq#9210's design on three streams, so a third slot's allowance and the eviction +\* order among three matter; readers hold each group from the start and stay. +CONSTANTS + N = 3 + Frames = 2 + FrameLen = 2 + Seg = 2 + W = 4 + SW = 2 + Ahead = 2 + Late = FALSE + Leave = FALSE + Shaped = TRUE + +SPECIFICATION Spec + +INVARIANTS + Books + WithinWindow + PausedWithinCap + Ends + +PROPERTIES + Progress diff --git a/rs/moq-net/verify/credit/ReceiveCreditWide.cfg b/rs/moq-net/verify/credit/ReceiveCreditWide.cfg new file mode 100644 index 0000000000..188a9d62d2 --- /dev/null +++ b/rs/moq-net/verify/credit/ReceiveCreditWide.cfg @@ -0,0 +1,24 @@ +\* moq-dev/moq#9210's design under a wider window, where the slots' allowances differ (2 and 0), +\* with frames of two one-unit segments. +CONSTANTS + N = 2 + Frames = 3 + FrameLen = 2 + Seg = 1 + W = 8 + SW = 3 + Ahead = 4 + Late = FALSE + Leave = FALSE + Shaped = TRUE + +SPECIFICATION Spec + +INVARIANTS + Books + WithinWindow + PausedWithinCap + Ends + +PROPERTIES + Progress diff --git a/rs/moq-net/verify/credit/SendCredit.cfg b/rs/moq-net/verify/credit/SendCredit.cfg new file mode 100644 index 0000000000..41b122edb6 --- /dev/null +++ b/rs/moq-net/verify/credit/SendCredit.cfg @@ -0,0 +1,17 @@ +\* Two subscriptions at one priority, one newest first and one oldest first, three groups each, +\* arriving in every order, over one stream of credit. +CONSTANTS + Subs = {1, 2} + G = 3 + High = {} + Asc = {2} + MaxStreams = 1 + +SPECIFICATION Spec + +INVARIANTS + WithinCredit + Ends + +PROPERTIES + Progress diff --git a/rs/moq-net/verify/credit/SendCredit.tla b/rs/moq-net/verify/credit/SendCredit.tla new file mode 100644 index 0000000000..d8864fa0da --- /dev/null +++ b/rs/moq-net/verify/credit/SendCredit.tla @@ -0,0 +1,257 @@ +----------------------------- MODULE SendCredit ----------------------------- +(***************************************************************************) +(* The sender's wait for stream credit: each lite subscription's *) +(* `TrackRun` (rs/moq-net/src/lite/publisher.rs) hands its groups out one *) +(* at a time, in any order they arrive, and a group waits for the peer's *) +(* stream credit in its session's line (rs/moq-net/src/transport/open.rs) *) +(* before its stream opens. *) +(* *) +(* The peer grants `MaxStreams` concurrent streams, and one more each time *) +(* it frees one. One group per subscription waits (`TrackRun::opening`). *) +(* The line ranks the opens as the transport sends their streams: the *) +(* subscription's priority, then the turn its send group last took credit *) +(* at, the oldest first, then the group's rank in its subscription *) +(* (`priority::in_group`), the newest first when descending. Only the head *) +(* of the line asks the transport for credit. A waiting open resets the *) +(* lowest-ranked open stream it preempts (`Lines::make_room`): one of a *) +(* lower-priority subscription, or an older one of its own, never a tied *) +(* subscription's. The first opens in line are owed the resets whose *) +(* credit has yet to return, and make no more. *) +(* *) +(* A group handed out while another waits replaces it if it outranks it *) +(* (`give_way`), and is shed otherwise. While the waiting group ranks *) +(* below every stream its subscription serves (`stuck`), the cursor hands *) +(* out only groups that outrank it; otherwise none. *) +(* *) +(* The receiver is abstract: a stream sent whole or reset is eventually *) +(* freed, returning its credit; ReceiveCredit.tla checks that every stream *) +(* it receives ends. No group expires, so a waiting group must open or be *) +(* replaced without the clock's help. *) +(***************************************************************************) +EXTENDS Integers, FiniteSets, TLC + +CONSTANTS + Subs, \* subscriptions on the session + G, \* groups per subscription, arriving in any order + High, \* the subscriptions at a higher send group priority than the rest + Asc, \* the subscriptions whose groups go oldest first; the rest go newest first + MaxStreams \* the peer's initial stream credit + +ASSUME MaxStreams >= 1 /\ G >= 1 /\ High \subseteq Subs /\ Asc \subseteq Subs + +Prio == [s \in Subs |-> IF s \in High THEN 1 ELSE 0] + +Pairs == Subs \X (1..G) + +\* `priority::in_group`: a group's rank within its subscription. +InGroup(s, k) == IF s \in Asc THEN -k ELSE k + +\* Whether rank `a` orders above rank `b`: `open::Rank`'s order, greater first. +Above(a, b) == + \/ a[1] > b[1] + \/ a[1] = b[1] /\ (a[2] > b[2] \/ (a[2] = b[2] /\ a[3] > b[3])) + +(***************************************************************************) +(* The whole state, one record, so a poll's parts compose as functions. *) +(* st: each group: "cached", "opening", "serving" (its stream *) +(* listed), "fin" (sent whole, unlisted), "reset" (preempted), *) +(* "closed" (freed by the peer), or "given" (up, never opened) *) +(* opening: per subscription, the group waiting for credit, or 0 *) +(* awake: per subscription, its `TrackRun` has a wake pending *) +(* asked: per subscription, its open waits on the transport's credit *) +(* rank, arrival: per subscription, its open's place in line, rank <<>> *) +(* when it has none *) +(* arrivals: the line's arrival counter *) +(* resets: `Line::resets` *) +(* turn: per subscription, when its send group last took credit *) +(* turns: the last turn handed out *) +(* opened, closed: streams opened, and freed by the peer *) +(***************************************************************************) +VARIABLE v + +vars == <> + +\* An open or stream's rank, as `open::Rank` orders it. +Rank(x, s, k) == <> + +Placed(x) == {s \in Subs : x.rank[s] # <<>>} + +\* Whether `s` goes before `t` in line: the higher rank, then the earlier arrival. +Before(x, s, t) == + \/ Above(x.rank[s], x.rank[t]) + \/ x.rank[s] = x.rank[t] /\ x.arrival[s] < x.arrival[t] + +Position(x, s) == Cardinality({t \in Placed(x) : Before(x, t, s)}) + 1 + +IsHead(x, s) == s \in Placed(x) /\ Position(x, s) = 1 + +Credit(x) == x.opened < MaxStreams + x.closed + +Wake(x, ss) == [x EXCEPT !.awake = [s \in Subs |-> x.awake[s] \/ s \in ss]] + +\* `Line::leave`: wake whoever is now first, if `s` was, and owe no more resets than opens. +Leave(x, s) == + LET y == [x EXCEPT !.rank[s] = <<>>, !.arrival[s] = 0] + heads == {t \in Placed(y) : IsHead(y, t)} + z == [y EXCEPT !.resets = IF x.resets < Cardinality(Placed(y)) + THEN x.resets ELSE Cardinality(Placed(y))] + IN IF IsHead(x, s) THEN Wake(z, heads) ELSE z + +\* Whether an open of `s`'s group `k` may reset `t`'s stream of group `j`: a lower-priority +\* subscription's, or an older one of its own; a tied subscription's turn differs. +Preempts(s, k, t, j) == Prio[s] > Prio[t] \/ (s = t /\ InGroup(s, k) > InGroup(t, j)) + +Listed(x) == {p \in Pairs : x.st[p] = "serving"} + +\* The lowest-ranked of the listed streams `ls`, or of `s`'s own. +Lowest(x, ls) == CHOOSE p \in ls : \A o \in ls : o = p \/ Above(Rank(x, o[1], o[2]), Rank(x, p[1], p[2])) + +\* The stream an open of `s`'s group `k` resets from `ls`, or <<>>. +Candidate(x, s, k, ls) == + IF ls = {} THEN <<>> + ELSE LET low == Lowest(x, ls) + own == {p \in ls : p[1] = s} + IN IF Preempts(s, k, low[1], low[2]) THEN low + ELSE IF own # {} /\ Preempts(s, k, s, Lowest(x, own)[2]) THEN Lowest(x, own) + ELSE <<>> + +\* `Lines::make_room`: each open in line not owed a reset resets the lowest stream it preempts. +RECURSIVE MakeRoom(_, _, _) +MakeRoom(x, owed, ls) == + IF owed = {} THEN x + ELSE LET s == CHOOSE o \in owed : \A t \in owed : o = t \/ Before(x, o, t) + c == Candidate(x, s, x.opening[s], ls) + IN IF c = <<>> THEN MakeRoom(x, owed \ {s}, ls) + ELSE MakeRoom(Wake([x EXCEPT !.st[c] = "reset", !.resets = x.resets + 1], {c[1]}), + owed \ {s}, ls \ {c}) + +Room(x) == MakeRoom(x, {s \in Placed(x) : Position(x, s) > x.resets}, Listed(x)) + +\* The open takes credit: its group's stream is listed, and its send group takes a turn. +Open(x, s) == + LET k == x.opening[s] IN + [x EXCEPT !.st[s, k] = "serving", !.opening[s] = 0, !.opened = x.opened + 1, + !.turns = x.turns + 1, !.turn[s] = x.turns + 1] + +Join(x, s) == + [x EXCEPT !.rank[s] = Rank(x, s, x.opening[s]), !.arrival[s] = x.arrivals, + !.arrivals = x.arrivals + 1] + +\* `Open::poll` for `s`'s waiting group. +PollOpen(x, s) == + IF x.opening[s] = 0 THEN x + ELSE IF x.rank[s] = <<>> /\ Placed(x) = {} + THEN \* Nobody waits: take credit if there is any, and join the line only without it. + IF Credit(x) THEN Open(x, s) + ELSE Room([Join(x, s) EXCEPT !.asked[s] = TRUE]) + ELSE LET y == IF x.rank[s] = <<>> THEN Join(x, s) ELSE x IN + IF ~IsHead(y, s) THEN Room(y) + ELSE IF Credit(y) + THEN Leave([Open(y, s) EXCEPT !.resets = IF y.resets > 0 THEN y.resets - 1 ELSE 0], s) + ELSE Room([y EXCEPT !.asked[s] = TRUE]) + +\* `TrackRun::stuck`: the waiting group ranks below every stream the subscription serves. +Stuck(x, s) == + LET k == x.opening[s] + held == {j \in 1..G : x.st[s, j] = "serving"} + IN k # 0 /\ held # {} /\ \A j \in held : InGroup(s, j) > InGroup(s, k) + +\* Whether the cursor may hand out `s`'s group `k`: any while none waits; while one waits and is +\* stuck, any when descending, since newest first sheds the backlog, and only one that outranks +\* it when ascending; otherwise none. +Admitted(x, s, k) == + LET w == x.opening[s] IN + w = 0 \/ (Stuck(x, s) /\ (s \notin Asc \/ k < w)) + +\* The cursor hands out group `k`: shed if it ranks below the waiting group, or replacing it +\* (`give_way`) to wait itself. +Cursor(x, s, k) == + LET w == x.opening[s] IN + IF w # 0 /\ InGroup(s, k) < InGroup(s, w) + THEN [x EXCEPT !.st[s, k] = "given"] + ELSE LET z == IF w = 0 THEN x + ELSE LET g == [x EXCEPT !.st[s, w] = "given", !.opening[s] = 0] IN + IF g.rank[s] = <<>> THEN g ELSE Leave(g, s) + IN [z EXCEPT !.st[s, k] = "opening", !.opening[s] = k] + +\* One pass of `TrackRun::poll_step`: the waiting open, then the cursor, which hands out groups +\* in the order they arrived, any order here. A pass that changes nothing parks the task. +Step(s) == + /\ v.awake[s] + /\ LET y == PollOpen(v, s) + ready == {k \in 1..G : y.st[s, k] = "cached" /\ Admitted(y, s, k)} + IN IF ready = {} + THEN v' = IF y = v THEN [v EXCEPT !.awake[s] = FALSE] ELSE y + ELSE \E k \in ready : v' = Cursor(y, s, k) + +\* A serving stream is sent whole and leaves the list; its task wakes the `TrackRun`. +Finish(p) == + /\ v.st[p] = "serving" + /\ v' = Wake([v EXCEPT !.st[p] = "fin"], {p[1]}) + +\* The peer frees a stream sent whole or reset, granting one more; the opens waiting on the +\* transport wake. +Free(p) == + /\ v.st[p] \in {"fin", "reset"} + /\ LET waiting == {s \in Subs : v.asked[s]} IN + v' = Wake([v EXCEPT !.st[p] = "closed", !.closed = v.closed + 1, + !.asked = [s \in Subs |-> FALSE]], waiting) + +Init == + v = [st |-> [p \in Pairs |-> "cached"], + opening |-> [s \in Subs |-> 0], + awake |-> [s \in Subs |-> TRUE], + asked |-> [s \in Subs |-> FALSE], + rank |-> [s \in Subs |-> <<>>], + arrival |-> [s \in Subs |-> 0], + arrivals |-> 0, + resets |-> 0, + turn |-> [s \in Subs |-> 0], + turns |-> 0, + opened |-> 0, + closed |-> 0] + +Done == \A p \in Pairs : v.st[p] \in {"closed", "given"} + +System == (\E s \in Subs : Step(s)) \/ (\E p \in Pairs : Finish(p) \/ Free(p)) + +Next == System \/ (Done /\ UNCHANGED vars) + +Spec == Init /\ [][Next]_vars /\ WF_vars(System) + +(***************************************************************************) +(* Rules. *) +(***************************************************************************) +\* Never more streams open than the peer granted. +WithinCredit == v.opened <= MaxStreams + v.closed + +\* Every group handed out waits for credit until its stream opens or it is given up, and every +\* stream ends: as in ReceiveCredit.tla, by `Progress` and `Ends`. +StRank(x) == CASE x = "cached" -> 0 [] x = "opening" -> 1 [] x = "serving" -> 2 + [] x \in {"fin", "reset"} -> 3 [] OTHER -> 4 + +RECURSIVE SumSet(_, _) +SumSet(f(_), S) == IF S = {} THEN 0 + ELSE LET e == CHOOSE e \in S : TRUE IN f(e) + SumSet(f, S \ {e}) + +\* First the groups' states (twice, so leaving the line for a stream outweighs the place) and +\* the places taken; then the opens asking the transport, and the tasks parked. +Counted == 2 * SumSet(LAMBDA p: StRank(v.st[p]), Pairs) + Cardinality(Placed(v)) +Waits == Cardinality({s \in Subs : v.asked[s]}) + Cardinality({s \in Subs : ~v.awake[s]}) + +Progress == + [][\/ Done + \/ Counted' > Counted + \/ Counted' = Counted /\ Waits' > Waits]_vars + +Enabled == (\E s \in Subs : v.awake[s]) \/ (\E p \in Pairs : v.st[p] \in {"serving", "fin", "reset"}) + +Guarded == Enabled = ENABLED System + +Ends == Done \/ Enabled + +\* Witnesses, each expected to fail. +NeverPreempts == \A p \in Pairs : v.st[p] # "reset" +NeverWaits == \A s \in Subs : v.rank[s] = <<>> +NeverGivesWay == \A p \in Pairs : v.st[p] # "given" +============================================================================= diff --git a/rs/moq-net/verify/credit/SendCreditPriority.cfg b/rs/moq-net/verify/credit/SendCreditPriority.cfg new file mode 100644 index 0000000000..35fbe6b10a --- /dev/null +++ b/rs/moq-net/verify/credit/SendCreditPriority.cfg @@ -0,0 +1,18 @@ +\* As SendCredit.cfg, with the newest-first subscription at a higher priority, so its opens +\* preempt the other's streams; it also checks `Ends` reads the steps' guards right. +CONSTANTS + Subs = {1, 2} + G = 3 + High = {1} + Asc = {2} + MaxStreams = 1 + +SPECIFICATION Spec + +INVARIANTS + WithinCredit + Ends + Guarded + +PROPERTIES + Progress diff --git a/rs/moq-net/verify/credit/SendCreditStreams.cfg b/rs/moq-net/verify/credit/SendCreditStreams.cfg new file mode 100644 index 0000000000..257425e0e6 --- /dev/null +++ b/rs/moq-net/verify/credit/SendCreditStreams.cfg @@ -0,0 +1,16 @@ +\* As SendCredit.cfg over two streams of credit, with the orders swapped. +CONSTANTS + Subs = {1, 2} + G = 3 + High = {} + Asc = {1} + MaxStreams = 2 + +SPECIFICATION Spec + +INVARIANTS + WithinCredit + Ends + +PROPERTIES + Progress From f51bd09f3462b3f079fe4dd037da70b43fada8ca Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 20 Nov 2026 20:23:57 -0800 Subject: [PATCH 3/3] verify: model groups whose producers never finish in SendCredit Every serving stream used to finish under fairness, so the send model's liveness never needed give_way or the stuck admission: removing either still passed. A live group now keeps its credit until a higher-ranked open resets it, and TopDelivered checks each subscription's highest-ranked group reaches the peer or waits only on streams it may not reset. Both mutants break it in SendCreditLive.cfg. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-credit/SKILL.md | 11 ++-- rs/moq-net/verify/credit/SendCredit.cfg | 3 + rs/moq-net/verify/credit/SendCredit.tla | 60 ++++++++++++++----- rs/moq-net/verify/credit/SendCreditLive.cfg | 21 +++++++ .../verify/credit/SendCreditPriority.cfg | 3 + .../verify/credit/SendCreditStreams.cfg | 3 + 6 files changed, 83 insertions(+), 18 deletions(-) create mode 100644 rs/moq-net/verify/credit/SendCreditLive.cfg diff --git a/.claude/skills/verify-credit/SKILL.md b/.claude/skills/verify-credit/SKILL.md index 3e29acb131..fe6b0bee32 100644 --- a/.claude/skills/verify-credit/SKILL.md +++ b/.claude/skills/verify-credit/SKILL.md @@ -16,7 +16,7 @@ Ingest checks the group's pace gate. Past the pace window, the stream reserves t When the peer has no connection credit left, moq-quic evicts the lowest-ranked stream holding credit, newest first. It holds off while an evicted reader has not stopped its stream, or while some reader has data it has not looked at. A reader arrives late or leaves only under `Late` and `Leave`. -`rs/moq-net/verify/credit/SendCredit.tla` models the sender's wait for stream credit. Each subscription's `TrackRun` hands its groups out one at a time, in any arrival order. One group per subscription waits for credit (`TrackRun::opening`) in the session's line, ranked as `open::Rank` orders it. Only the head asks the transport, and a waiting open resets the lowest-ranked stream it preempts (`Lines::make_room`). A group handed out while another waits replaces it if it outranks it (`give_way`), and is shed otherwise. While the waiting group is `stuck`, the cursor hands out only groups that may move it on. The receiver is abstract: a stream sent whole or reset is eventually freed, and ReceiveCredit.tla checks that it is. +`rs/moq-net/verify/credit/SendCredit.tla` models the sender's wait for stream credit. Each subscription's `TrackRun` hands its groups out one at a time, in any arrival order. One group per subscription waits for credit (`TrackRun::opening`) in the session's line, ranked as `open::Rank` orders it. Only the head asks the transport, and a waiting open resets the lowest-ranked stream it preempts (`Lines::make_room`). A group handed out while another waits replaces it if it outranks it (`give_way`), and is shed otherwise. While the waiting group is `stuck`, the cursor hands out only groups that may move it on. The receiver is abstract: a stream sent whole or reset is eventually freed, and ReceiveCredit.tla checks that it is. A group in `LiveSubs` at `LiveGroup` is one whose producer never finishes, so its stream keeps its credit until a higher-ranked open resets it, as in `rs/moq-net/tests/integration/stream_credit.rs`. Each step is one poll of the code that handles one event, under the connection's lock where the code takes it. TLC checks every ordering under the bounds in each config. @@ -30,13 +30,14 @@ Each step is one poll of the code that handles one event, under the connection's | `evict`'s assertion | Receive | Eviction never picks a stream whose reader may still progress (`Assert` in `EvictLoop`). | A reader that would have finished loses its group. | | `Progress` and `Ends` | Receive | Every stream is read whole or evicted, and stopped (`AllEnd`). | A stream holds credit forever: the connection hangs. | | `WithinCredit` | Send | Never more streams open than the peer granted. | The peer closes the connection with STREAM_LIMIT_ERROR. | -| `Progress` and `Ends` | Send | Every group waiting for credit opens or is given up, and every stream ends. | A subscription stops sending behind a group that never opens. | +| `Progress` and `Ends` | Send | Every behavior ends quiet, and once quiet every group was sent and freed or given up, is a live group's stream, or waits only on live streams it may not reset. | A subscription stops sending behind a group that never opens. | +| `TopDelivered` | Send | Once quiet, each subscription's highest-ranked group reached the peer, was reset for a higher priority, or waits only on live streams it may not reset. | A waiting group holds back a newer one that could reset an unfinished stream: the stall #8607 fixed. | TLC's own liveness check is about a thousand times slower here, so liveness is checked through safety. `Progress` is an action property: every step other than the final stutter raises a bounded measure, so every behavior ends stuttering in one state. `Ends` is an invariant: every state is done, or one of the system's steps is enabled there, which weak fairness takes. `Ends` reads each step's guard in place of ENABLED, and `Guarded` checks the two agree on a smaller config. The comment above `Lifecycle` in ReceiveCredit.tla gives the measure. #9210's claim that every stream pauses, charged its slot's allowance, is `EveryPauseFits` and `ChargedAllowance`, which ReceiveCreditAllowance.cfg checks as an open finding (see below). -Each model has witnesses (`NeverPaused`, `NeverEvicted`, `NeverStuckWaiting`, `NeverPartial`, `NeverPausedInside`; `NeverPreempts`, `NeverWaits`, `NeverGivesWay`). Each one is expected to fail, which shows TLC reaches the state. Check one after changing a step's guard. +Each model has witnesses (`NeverPaused`, `NeverEvicted`, `NeverStuckWaiting`, `NeverPartial`, `NeverPausedInside`; `NeverPreempts`, `NeverWaits`, `NeverGivesWay`, `NeverBlocked`). Each one is expected to fail, which shows TLC reaches the state. Check one after changing a step's guard. ## Run @@ -57,6 +58,7 @@ quest gate just rs tla moq-net/verify/credit/ReceiveCredit.tla -config rs/moq-ne | `SendCredit.cfg` | two subscriptions, one newest first and one oldest first, one stream of credit | 89k | 6 s | | `SendCreditPriority.cfg` | one subscription at a higher priority, so its opens preempt; also `Guarded` | 99k | 46 s | | `SendCreditStreams.cfg` | two streams of credit, the orders swapped | 296k | 14 s | +| `SendCreditLive.cfg` | a middle group of each subscription never finishes | 41.6k | 3 s | Every config passes but the open finding. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. @@ -96,7 +98,8 @@ What the model relies on in that code: - A stopped stream returns its paused credit and frees its slot (`Stop`), so a reader that never returns cannot pin the paused budget. - Eviction passes over a reader that may still progress, and an evicted reader's taken units keep their credit until it stops (`evicted_held`). - Only the head of the line asks the transport, and leaving the line wakes the next head (`Line::leave`). +- A waiting group gives way to one that outranks it (`give_way`), and while it is `stuck` the cursor hands such a group out. Without either, `SendCreditLive.cfg` breaks `TopDelivered`, which the configs without a live group miss. ## Not covered -Both models leave out loss, gaps, and reordering within a stream: every unit arrives in order. They also leave out rate caps, shrink debt, frame headers' bytes, and the peer resetting a stream on the receive model. Every stream ranks at one priority on the receive side, so eviction order is newest first. Relay chains are left out, since pausing is per hop. So are the front's shedding past a pause and a group's expiry. The send side has two priority levels (`High` and the rest). TLC's simulate mode crashes on ReceiveCredit.tla (an `ArrayIndexOutOfBoundsException` in TLC), so larger shapes run exhaustively, as ReceiveCreditThree.cfg does. A pass covers two streams, or three with readers held from the start, of two or three frames, and two subscriptions of three groups. +Both models leave out loss, gaps, and reordering within a stream: every unit arrives in order. They also leave out rate caps, shrink debt, frame headers' bytes, and the peer resetting a stream on the receive model. Every stream ranks at one priority on the receive side, so eviction order is newest first. Relay chains are left out, since pausing is per hop. So are the front's shedding past a pause and a group's expiry. The send side has two priority levels (`High` and the rest). TLC's simulate mode crashes on ReceiveCredit.tla (an `ArrayIndexOutOfBoundsException` in TLC), so larger shapes run exhaustively, as ReceiveCreditThree.cfg does. A pass covers two streams, or three with readers held from the start, of two or three frames, and two subscriptions of three groups, one of each live in `SendCreditLive.cfg`. diff --git a/rs/moq-net/verify/credit/SendCredit.cfg b/rs/moq-net/verify/credit/SendCredit.cfg index 41b122edb6..74ad47de9f 100644 --- a/rs/moq-net/verify/credit/SendCredit.cfg +++ b/rs/moq-net/verify/credit/SendCredit.cfg @@ -6,12 +6,15 @@ CONSTANTS High = {} Asc = {2} MaxStreams = 1 + LiveSubs = {} + LiveGroup = 1 SPECIFICATION Spec INVARIANTS WithinCredit Ends + TopDelivered PROPERTIES Progress diff --git a/rs/moq-net/verify/credit/SendCredit.tla b/rs/moq-net/verify/credit/SendCredit.tla index d8864fa0da..9622dc73d6 100644 --- a/rs/moq-net/verify/credit/SendCredit.tla +++ b/rs/moq-net/verify/credit/SendCredit.tla @@ -15,7 +15,7 @@ (* of the line asks the transport for credit. A waiting open resets the *) (* lowest-ranked open stream it preempts (`Lines::make_room`): one of a *) (* lower-priority subscription, or an older one of its own, never a tied *) -(* subscription's. The first opens in line are owed the resets whose *) +(* subscription's. The first opens in line are owed the resets whose *) (* credit has yet to return, and make no more. *) (* *) (* A group handed out while another waits replaces it if it outranks it *) @@ -25,8 +25,10 @@ (* *) (* The receiver is abstract: a stream sent whole or reset is eventually *) (* freed, returning its credit; ReceiveCredit.tla checks that every stream *) -(* it receives ends. No group expires, so a waiting group must open or be *) -(* replaced without the clock's help. *) +(* it receives ends. A `Live` group's producer never finishes, so its *) +(* stream holds its credit until a higher-ranked open resets it. No group *) +(* expires, so a waiting group must open or be replaced without the *) +(* clock's help, unless only streams it may not reset hold the credit. *) (***************************************************************************) EXTENDS Integers, FiniteSets, TLC @@ -35,14 +37,19 @@ CONSTANTS G, \* groups per subscription, arriving in any order High, \* the subscriptions at a higher send group priority than the rest Asc, \* the subscriptions whose groups go oldest first; the rest go newest first - MaxStreams \* the peer's initial stream credit + MaxStreams, \* the peer's initial stream credit + LiveSubs, \* the subscriptions with a group whose producer never finishes + LiveGroup \* that group ASSUME MaxStreams >= 1 /\ G >= 1 /\ High \subseteq Subs /\ Asc \subseteq Subs +ASSUME LiveSubs \subseteq Subs /\ LiveGroup \in 1..G Prio == [s \in Subs |-> IF s \in High THEN 1 ELSE 0] Pairs == Subs \X (1..G) +Live == LiveSubs \X {LiveGroup} + \* `priority::in_group`: a group's rank within its subscription. InGroup(s, k) == IF s \in Asc THEN -k ELSE k @@ -184,9 +191,11 @@ Step(s) == THEN v' = IF y = v THEN [v EXCEPT !.awake[s] = FALSE] ELSE y ELSE \E k \in ready : v' = Cursor(y, s, k) -\* A serving stream is sent whole and leaves the list; its task wakes the `TrackRun`. +\* A serving stream is sent whole and leaves the list; its task wakes the `TrackRun`. A live +\* group's producer never finishes it. Finish(p) == /\ v.st[p] = "serving" + /\ p \notin Live /\ v' = Wake([v EXCEPT !.st[p] = "fin"], {p[1]}) \* The peer frees a stream sent whole or reset, granting one more; the opens waiting on the @@ -211,11 +220,15 @@ Init == opened |-> 0, closed |-> 0] -Done == \A p \in Pairs : v.st[p] \in {"closed", "given"} - System == (\E s \in Subs : Step(s)) \/ (\E p \in Pairs : Finish(p) \/ Free(p)) -Next == System \/ (Done /\ UNCHANGED vars) +\* Where some step of the system is enabled: each step's guard. `Guarded` checks they agree. +Enabled == \/ \E s \in Subs : v.awake[s] + \/ \E p \in Pairs : v.st[p] \in {"fin", "reset"} \/ (v.st[p] = "serving" /\ p \notin Live) + +Quiet == ~Enabled + +Next == System \/ (Quiet /\ UNCHANGED vars) Spec == Init /\ [][Next]_vars /\ WF_vars(System) @@ -225,8 +238,8 @@ Spec == Init /\ [][Next]_vars /\ WF_vars(System) \* Never more streams open than the peer granted. WithinCredit == v.opened <= MaxStreams + v.closed -\* Every group handed out waits for credit until its stream opens or it is given up, and every -\* stream ends: as in ReceiveCredit.tla, by `Progress` and `Ends`. +\* Every behavior ends quiet, as in ReceiveCredit.tla, by `Progress`; `Ends` and `TopDelivered` +\* say what a quiet state may leave undone. StRank(x) == CASE x = "cached" -> 0 [] x = "opening" -> 1 [] x = "serving" -> 2 [] x \in {"fin", "reset"} -> 3 [] OTHER -> 4 @@ -240,18 +253,37 @@ Counted == 2 * SumSet(LAMBDA p: StRank(v.st[p]), Pairs) + Cardinality(Placed(v)) Waits == Cardinality({s \in Subs : v.asked[s]}) + Cardinality({s \in Subs : ~v.awake[s]}) Progress == - [][\/ Done + [][\/ Quiet \/ Counted' > Counted \/ Counted' = Counted /\ Waits' > Waits]_vars -Enabled == (\E s \in Subs : v.awake[s]) \/ (\E p \in Pairs : v.st[p] \in {"serving", "fin", "reset"}) - Guarded == Enabled = ENABLED System -Ends == Done \/ Enabled +\* `s`'s group `k` waits for credit only live streams it may not reset hold: none is left, and +\* `make_room` finds no stream for it. +Blocked(s, k) == ~Credit(v) /\ Candidate(v, s, k, Listed(v)) = <<>> + +\* Once quiet, every group was sent and freed or given up, is a live group's stream, or waits, +\* blocked, or stays cached behind its subscription's blocked open: nothing is left to send +\* that credit could carry. +Ends == + Quiet => \A p \in Pairs : + \/ v.st[p] \in {"closed", "given"} + \/ v.st[p] = "serving" /\ p \in Live + \/ v.st[p] = "opening" /\ Blocked(p[1], p[2]) + \/ v.st[p] = "cached" /\ v.opening[p[1]] # 0 /\ Blocked(p[1], v.opening[p[1]]) + +\* Once quiet, each subscription's highest-ranked group reached the peer, was reset for a higher +\* priority, or is blocked: a waiting group never holds back one that could reset a live stream. +TopDelivered == + Quiet => \A s \in Subs : + LET k == CHOOSE k \in 1..G : \A j \in 1..G : InGroup(s, k) >= InGroup(s, j) IN + \/ v.st[s, k] \in {"serving", "fin", "reset", "closed"} + \/ v.st[s, k] \in {"cached", "opening"} /\ Blocked(s, k) \* Witnesses, each expected to fail. NeverPreempts == \A p \in Pairs : v.st[p] # "reset" NeverWaits == \A s \in Subs : v.rank[s] = <<>> NeverGivesWay == \A p \in Pairs : v.st[p] # "given" +NeverBlocked == ~(Quiet /\ \E s \in Subs : v.opening[s] # 0) ============================================================================= diff --git a/rs/moq-net/verify/credit/SendCreditLive.cfg b/rs/moq-net/verify/credit/SendCreditLive.cfg new file mode 100644 index 0000000000..91462f2a32 --- /dev/null +++ b/rs/moq-net/verify/credit/SendCreditLive.cfg @@ -0,0 +1,21 @@ +\* As SendCredit.cfg, with a group of each subscription whose producer never finishes, so its +\* stream holds credit until a higher-ranked open of its own subscription resets it, and a +\* tied subscription's live stream can block the other's opens for good. +CONSTANTS + Subs = {1, 2} + G = 3 + High = {} + Asc = {2} + MaxStreams = 1 + LiveSubs = {1, 2} + LiveGroup = 2 + +SPECIFICATION Spec + +INVARIANTS + WithinCredit + Ends + TopDelivered + +PROPERTIES + Progress diff --git a/rs/moq-net/verify/credit/SendCreditPriority.cfg b/rs/moq-net/verify/credit/SendCreditPriority.cfg index 35fbe6b10a..f10beab21b 100644 --- a/rs/moq-net/verify/credit/SendCreditPriority.cfg +++ b/rs/moq-net/verify/credit/SendCreditPriority.cfg @@ -6,12 +6,15 @@ CONSTANTS High = {1} Asc = {2} MaxStreams = 1 + LiveSubs = {} + LiveGroup = 1 SPECIFICATION Spec INVARIANTS WithinCredit Ends + TopDelivered Guarded PROPERTIES diff --git a/rs/moq-net/verify/credit/SendCreditStreams.cfg b/rs/moq-net/verify/credit/SendCreditStreams.cfg index 257425e0e6..f93a1b8114 100644 --- a/rs/moq-net/verify/credit/SendCreditStreams.cfg +++ b/rs/moq-net/verify/credit/SendCreditStreams.cfg @@ -5,12 +5,15 @@ CONSTANTS High = {} Asc = {1} MaxStreams = 2 + LiveSubs = {} + LiveGroup = 1 SPECIFICATION Spec INVARIANTS WithinCredit Ends + TopDelivered PROPERTIES Progress