diff --git a/quest/m1/bbr3-references/README.md b/quest/m1/bbr3-references/README.md new file mode 100644 index 00000000000..1d2e20496ae --- /dev/null +++ b/quest/m1/bbr3-references/README.md @@ -0,0 +1,49 @@ +# BBR3 matches its references + +## Goal + +moq-congestion's BBR3 matches its references wherever it has not +deliberately diverged: each bug a reference comparison confirmed is fixed +with a regression test on the seeded, bufferbloat-shaped simulation, the +controller follows draft-ietf-ccwg-bbr-07, and `rs/moq-quic/README.md`'s +ledger names the reference commits checked and every deliberate divergence. + +Out of scope: differences only measurement could settle (when ProbeBW_UP and +DOWN end, minimum loss events per round, the initial window, clamping samples +by receive timestamps, a CE ratio instead of one CE), since the 2026-11-09 +deletion audit ([moq-dev/moq#8423](https://github.com/moq-dev/moq/pull/8423)) +pruned the study that would measure them; app-limited bandwidth aging, which +the [probe epic](/quest/m3/probe/README.md) owns; and quinn `main` commits, +which the [quinn sync](/quest/m3/vendor-sync/quinn.md) owns. + +## Plan + +Decided 2026-11-14 (maintainer, /quest-plan): ✅ fix the confirmed bugs and +adopt draft-07 (rejected: bugs only, skipping draft-07's ProbeRTT rework; +also revive the pruned measurement study). + +Found by a 2026-11-14 comparison against +[Google Linux BBRv3 `90210de4`](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c) +(unchanged since 2025-03), [draft-07](https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-07.html) +(2026-11-02), and [QUICHE BBR3 `51a950db`](https://github.com/google/quiche/blob/51a950dba2100da7e7977aa97759470507d1ebb8/quiche/quic/core/congestion_control/bbr3_sender.cc). +[quinn#2481](https://github.com/quinn-rs/quinn/pull/2481)'s latest +force-push (`0ebaf7e2`) only rebased; its BBR3 is the `01b8848d` already +synced. + +Constraint from the maintainer: no fix here reverts a MoQ change for +application-limited senders (ProbeRTT skip, app-limited bandwidth aging, +skipping app-limited rounds in plateau counting, the bandwidth probe's +headroom). A reference behavior that conflicts with one keeps ours. + +This README's own work, after the children: update the BBR3 entries in +`rs/moq-quic/README.md` to name draft-07 and the three reference commits, +and list the divergences recorded only in code comments today (ProbeRTT +skip, app-limited aging, the one-packet send quantum floor, loss state saved +once per recovery episode, Linux's growth check) beside classic ECN. Record the reference heads there too, so each +[quinn sync](/quest/m3/vendor-sync/quinn.md) rechecks them with quinn#2481. + +## Related + +- [Startup loss counting](/quest/m2/bbr3-startup-loss.md) - draft-07's per-packet loss events, a breaking change before the cut +- [Probe up to a target bitrate](/quest/m3/probe/README.md) - owns app-limited aging and probe headroom +- [Realtime QUIC](/quest/m4/quic-realtime.md) - the bake-off for algorithms other than BBR3 diff --git a/quest/m1/bbr3-references/growth.md b/quest/m1/bbr3-references/growth.md new file mode 100644 index 00000000000..0dc99217da1 --- /dev/null +++ b/quest/m1/bbr3-references/growth.md @@ -0,0 +1,24 @@ +# [S] BBR3 recognizes bandwidth growth on any ACK + +## Goal + +Under ACK aggregation, as on Wi-Fi and cellular uplinks, Startup and +ProbeBW_UP end on a real plateau rather than on three low samples that +happened to land on round starts. + +## Plan + +Decided 2026-11-14 (maintainer, /quest-plan): adopt Linux's check, provided +it keeps every MoQ change for application-limited senders. The maintainer +answered "I don't know", with that constraint, so the pick follows the +recommendation; reopen it in review if that reading is wrong. + +`check_full_bw_reached` (`rs/moq-congestion/src/bbr3/mod.rs`) returns unless +`round_start`, as draft-06 and -07 gate it. Linux checks every +non-application-limited ACK for 1.25x growth and resets the baseline on +any, counting plateau rounds only at round boundaries +([tcp_bbr.c:1924-1948](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L1924-L1948)). +Move only `full_bw_count += 1` and the `app_limited_rounds` bookkeeping +under the round-start gate; application-limited samples still neither reset +nor advance the plateau. Test bursty, aggregated ACKs and a capacity that +rises mid-Startup, alongside the existing app-limited tests. diff --git a/quest/m1/bbr3-references/idle-aggregation.md b/quest/m1/bbr3-references/idle-aggregation.md new file mode 100644 index 00000000000..4ab05c63538 --- /dev/null +++ b/quest/m1/bbr3-references/idle-aggregation.md @@ -0,0 +1,23 @@ +# [XS] BBR3's ACK aggregation estimate stays bounded across idle + +## Goal + +A sender that goes idle between bursts, as media does when the RTT is +shorter than the frame interval, keeps `extra_acked` and the window it adds +bounded by the aggregation it actually sees. + +## Plan + +`handle_restart_from_idle` (`rs/moq-congestion/src/bbr3/mod.rs`) resets +`extra_acked_interval_start` but not `extra_acked_delivered`, so each +resumed burst counts every earlier epoch's bytes as aggregation. The cap at +cwnd is circular, since `set_cwnd` grows cwnd toward `2·BDP + extra_acked`. +Simulated in the 2026-11-14 audit: 8 Mbit/s media at 30 fps over 20 ms grew +cwnd to 11 MB in 18 s (74 KB with the fix). Draft-06's +`BBRHandleRestartFromIdle` has the same omission; Linux zeroes +`ack_epoch_acked` there ([tcp_bbr.c:540-542](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L540-L542)). + +Zero `extra_acked_delivered` with the interval start, and test that bursts +separated by idle gaps keep `extra_acked` and cwnd bounded. Caused by +[moq-dev/moq#4921](https://github.com/moq-dev/moq/pull/4921), which ported +the draft's omission. diff --git a/quest/m1/bbr3-references/loss-order.md b/quest/m1/bbr3-references/loss-order.md new file mode 100644 index 00000000000..00246d89763 --- /dev/null +++ b/quest/m1/bbr3-references/loss-order.md @@ -0,0 +1,27 @@ +# [M] Controllers hear an ACK's losses before the ACK + +## Goal + +moq-quic reports the packets an ACK declares lost before it reports the ACK, +as RFC 9002's OnAckReceived orders them, so a congestion controller's loss +response takes effect on the same ACK, for every controller. + +## Plan + +Decided 2026-11-14 (maintainer, /quest-plan): ✅ the transport reports losses +before the ACK (rejected: only refresh BBR3's pacing and cwnd at the end of +`on_loss`, which leaves the double cut). + +Today `Connection::on_ack_received` (`rs/moq-quic/src/connection/mod.rs`) +calls `Controller::on_ack` and then `detect_lost_packets`, an order inherited +from quinn. With BBR3, a loss that ends ProbeBW_UP leaves pacing at 1.25x +and UP's window until the next ACK, since `on_loss` recomputes neither, and +losses found by the ACK that closes a loss round open a second round, so +DOWN, CRUISE, or Drain can cut the short-term model twice where Linux cuts +once ([tcp_bbr.c:2176-2201](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L2176-L2201)). + +Change the order for all controllers (Cubic, SCReAM, and GCC included), +document it on `Controller`, and drop BBR3's note on the old order around +`on_spurious_loss`. A loss timer that fires with no ACK still needs BBR3's +`on_loss` to refresh pacing and cwnd. Test both effects. No single PR caused +it: the order is quinn's. diff --git a/quest/m1/bbr3-references/probe-rtt.md b/quest/m1/bbr3-references/probe-rtt.md new file mode 100644 index 00000000000..f3ad54d07fb --- /dev/null +++ b/quest/m1/bbr3-references/probe-rtt.md @@ -0,0 +1,43 @@ +# [M] BBR3 schedules ProbeRTT as draft-07 does + +## Goal + +BBR3 enters ProbeRTT when its 10 s `min_rtt` filter expires, as +[draft-07](https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-07.html#section-5.3.4) +does, halving ProbeRTT's cost for a backlogged sender, while an +application-limited sender still skips it and still learns a base RTT that +rose. + +## Plan + +draft-07 drops the 5 s `ProbeRTTInterval` and the `probe_rtt_min_delay`, +`probe_rtt_min_stamp`, and `probe_rtt_expired` fields, adds +`BBR.probe_rtt_cwnd` (half the BDP from the old `min_rtt`) to +`ProbeRTTCwnd()`, calls `SaveCwnd()` before `EnterProbeRTT()` (ours saves +after, so `prior_cwnd` only ratchets up), and replaces `ack_phase` with +`is_bw_probe_stopping` and `is_bw_probe_sample` with `check_probe_loss`. +Adopt all of it, citing -07 anchors throughout `rs/moq-congestion/src/bbr3/`. +QUICHE BBR3 also uses a 10 s period. + +The application-limited ProbeRTT skip stays. Decided 2026-11-14 (agent pick +after the maintainer answered "I don't know", for review), keeping the +skip's benefit for media: + +- ✅ Spurious-loss undo restores `probe_rtt_skip_congested` to its value + when the episode began, since a loss proven spurious was not congestion, + while a CE or real loss earlier in the interval still denies the skip. + Save it with the other undo state in `enter_recovery`; restarting the + interval clears the saved value too. +- ✅ A skipped expiry adopts the expired window's lowest RTT sample as + `min_rtt` instead of only refreshing its stamp, so a base RTT that rose + is learned without running ProbeRTT; an application-limited sender's + samples carry little self-induced queue. +- Rejected: QUICHE BBRv1's kBBR7 guard (run ProbeRTT when RTT drifted past + 12.5%), which would put ProbeRTT's half-BDP cap back on media; and denying + the skip only on loss above the 2% threshold, which needs measurement. + +The cut-short probe advance on ProbeRTT entry relies on the removed +PROBE_STARTING and PROBE_FEEDBACK phases; express it with +`check_probe_loss || is_bw_probe_stopping`. Test the 10 s schedule, the +save order, a base-RTT rise while application-limited, and a spurious loss +during a skip. diff --git a/quest/m1/bbr3-references/transitions.md b/quest/m1/bbr3-references/transitions.md new file mode 100644 index 00000000000..90cdbe98e9e --- /dev/null +++ b/quest/m1/bbr3-references/transitions.md @@ -0,0 +1,27 @@ +# [XS] BBR3 clears congestion signals in Drain and floors its Reno probe time + +## Goal + +Leaving Startup charges a loss or CE mark once, and a small-BDP bulk flow +probes no more often than Linux's minimum of a few rounds. + +## Plan + +- `enter_drain` (`rs/moq-congestion/src/bbr3/mod.rs`) does not reset the + congestion signals as Linux does on Drain entry + ([tcp_bbr.c:1963](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L1963)), + so the CE that ends Startup also cuts `bw_shortterm` and + `inflight_shortterm` one loss round later in Drain, contradicting the + ledger's once-per-episode CE response. Call `reset_congestion_signals()`. +- `is_reno_coexistence_probe_time` divides `min(BDP, cwnd)` by the MSS, + rounding down, with no floor: at 1 Mbit/s and 20 ms it probes every one or + two rounds, and DOWN can end before draining the probe's queue. Linux's + target goes through `bbr_inflight`, floored by the quantization budget + ([tcp_bbr.c:1010-1015](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L1010-L1015)). + Floor the BDP at `max(bdp, offload budget, min_pipe_cwnd)`, round up, + then cap it at cwnd as Linux's `bbr_target_inflight` does, so a cwnd cut + by congestion still shortens the wait. + Application-limited flows are unaffected, since their UP lasts until loss. + +Test each. Caused by [moq-dev/moq#4921](https://github.com/moq-dev/moq/pull/4921), +which ported both from draft-06. diff --git a/quest/m2/bbr3-startup-loss.md b/quest/m2/bbr3-startup-loss.md new file mode 100644 index 00000000000..1f85c1385ee --- /dev/null +++ b/quest/m2/bbr3-startup-loss.md @@ -0,0 +1,34 @@ +# [S] BBR3 counts Startup loss events per lost packet + +## Goal + +An application-limited media sender in Startup whose bottleneck shrinks +leaves Startup on high loss, as draft-07 specifies, and moq-congestion's +`Lost` drops the `new_burst` field that only the old counting needed. + +## Plan + +`note_loss` (`rs/moq-congestion/src/bbr3/mod.rs`) counts discontiguous lost +ranges, as draft-06 read, from `Lost::new_burst`, which moq-quic computes. +[draft-07](https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-07.html#section-5.3.1.3) +counts every lost packet and resets the count in +`AdvanceLatestDeliverySignals`, `ResetCongestionSignals`, and spurious-loss +undo; ours resets it in none of the last two. A media sender bursting about +14 packets per frame into a tail-drop link that shrank (50 to 3 Mbit/s) +loses one run per frame, one to three events per round, never six, and its +application-limited rounds block the plateau exit, so Startup never ends +and keyframe tails keep dropping. + +Count per lost packet, reset where draft-07 does, remove `Lost::new_burst` +and its computation in moq-quic, and replace the even 1-in-25 drop test with +a tail-drop bottleneck test. + +In m2, apart from [its epic](/quest/m1/bbr3-references/README.md): +removing `Lost::new_burst` changes moq-congestion's public API, which the +cut publishes for the first time. Decided 2026-11-15 (maintainer, +/quest-complete): ✅ keep it in m2 (rejected: m3, keeping the API). +Public API: removes `Lost::new_burst`. Wire: none. + +## Related + +- [BBR3 matches its references](/quest/m1/bbr3-references/README.md) - the rest of the comparison's fixes diff --git a/quest/m3/vendor-sync/quinn.md b/quest/m3/vendor-sync/quinn.md index 299b2b2c023..1eb0570fe35 100644 --- a/quest/m3/vendor-sync/quinn.md +++ b/quest/m3/vendor-sync/quinn.md @@ -23,6 +23,16 @@ and [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481)'s BBR3 through all recorded in `rs/moq-quic/README.md`'s `Upstream` section with every disposition. Start there, and record the new heads in the same place. +Checked 2026-11-14 (/quest-plan): quinn#2481 force-pushed to +[`0ebaf7e2`](https://github.com/quinn-rs/quinn/pull/2481/commits/0ebaf7e2c9932e432ef5e7caf916291b02d8277e), +a rebase onto `c7b46a96` whose BBR3 commits match `01b8848d`, so it has +nothing to port; recorded in the ledger. Of the nine `main` commits since `aea1e538`, check +[`aa61ce99`](https://github.com/quinn-rs/quinn/commit/aa61ce996b97b21c63731a84fac1346c2d70852b) (late +ACK-only packets are not spurious losses) against BBR3's spurious-loss undo +first. The [BBR3 references epic](/quest/m1/bbr3-references/README.md) adds +Google Linux BBRv3, draft-ietf-ccwg-bbr, and QUICHE BBR3 heads to the ledger; +recheck them with quinn#2481 each run. + Learned: a fix missing from the ledger is often already held, since the fork reworked the same code (`stop` under `RESET_STREAM_AT`, the stale-accept check under `handshake_timeout`, socket setup in `moq_sock::bind`), so check diff --git a/rs/moq-quic/README.md b/rs/moq-quic/README.md index 81a64319ab5..6c2b1b035e6 100644 --- a/rs/moq-quic/README.md +++ b/rs/moq-quic/README.md @@ -56,7 +56,7 @@ Changes on top of the upstream commit, besides the renames: - **Close reasons**: a local close reports `connection::Error::LocallyClosed` with its code and reason from `poll_closed` and `error()`, a handshake that idles out reports `HandshakeTimeout`, and close reasons are strings, lossily decoded. - **No early data**: TLS never offers or accepts 0-RTT, and early packets are discarded without decryption. The early keys, remembered transport parameters, stream rollback, and early-session driver APIs are removed. Session tickets still resume 1-RTT handshakes. MoQ's CONNECT and authorization already wait for the handshake. - **BBR3**, from [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481) at [`55f74c0d`](https://github.com/quinn-rs/quinn/pull/2481/commits/55f74c0dd0738b5ad7d7676c0b2869a547783f83) (open upstream), rebased onto the fork point with authorship kept. It replaces quinn's BBR and is the default controller. It carries the BBR correctness fixes moq-dev/noq shipped in `moq-noq` 1.3.1, and with them the `Controller` changes BBR needs: per-packet send and loss callbacks, `on_cwnd_limited`, `on_app_limited`, `on_ack_frequency_update`, and controller-owned pacing and burst size. `Metrics` rates are bytes per second. -- **BBR3 upstream refresh**: Checked against the force-pushed [`01b8848d`](https://github.com/quinn-rs/quinn/pull/2481/commits/01b8848dcb321eb17ad4d88dd6f84f28f45a0d6a) on 2026-10-31: carry startup's per-round ACK aggregation maximum, restore the saved window once when recovery ends, preserve it through recovery into ProbeRTT, saturate probe-bound growth, and stop a probe at the previous loss bound before accelerating again on clean feedback. Spurious-loss undo also restores that probe history, and an idle ACK renews application-limited marking even when it crosses the old marker. The two-cycle bandwidth filter and transport-owned packet snapshots already cover its other fixes. Upstream's bandwidth metric conversion to bits/s does not apply: this fork's controller metrics use bytes/s. +- **BBR3 upstream refresh**: Checked against the force-pushed [`01b8848d`](https://github.com/quinn-rs/quinn/pull/2481/commits/01b8848dcb321eb17ad4d88dd6f84f28f45a0d6a) on 2026-10-31: carry startup's per-round ACK aggregation maximum, restore the saved window once when recovery ends, preserve it through recovery into ProbeRTT, saturate probe-bound growth, and stop a probe at the previous loss bound before accelerating again on clean feedback. Spurious-loss undo also restores that probe history, and an idle ACK renews application-limited marking even when it crosses the old marker. The two-cycle bandwidth filter and transport-owned packet snapshots already cover its other fixes. Upstream's bandwidth metric conversion to bits/s does not apply: this fork's controller metrics use bytes/s. Rechecked against the force-pushed [`0ebaf7e2`](https://github.com/quinn-rs/quinn/pull/2481/commits/0ebaf7e2c9932e432ef5e7caf916291b02d8277e) on 2026-11-14: a rebase onto quinn `main` at `c7b46a96` whose BBR3 commits match `01b8848d`, so nothing to port. - **BBR3 loss samples**: a lost packet supplies its own inflight, loss, and application-limited state without borrowing or overwriting the ACK sample, including before the first ACK observation. This follows draft-06 and Google Linux BBRv3; upstream submission waits on quinn#2481 for the same callback adaptation as packet retirement. - **BBR3 packet retirement**: per-space packet lookup and round-indexed expiry retire completed snapshots incrementally, preserving the extra ACK batch of metadata for late loss callbacks. The bounded indexes handle sparse packet numbers without walking the live flight. Upstream submission is deferred while quinn#2481 remains open: this fork also carries the packet-space and loss callback adaptations, so a port needs to target that implementation after its shape settles. - **Classic ECN for BBR3**, ported from [moq-dev/noq#12](https://github.com/moq-dev/noq/pull/12): CE exits Startup, stops a bandwidth probe, or lowers the short-term model, once per recovery episode, instead of counting as a loss.