Repository navigation
quest: plan BBR3 fixes from a reference comparison #8792
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| # BBR3 matches its references | ||
|
|
||
| ## Goal | ||
|
|
||
| moq-congestion's BBR3 matches its references wherever it has not | ||
| deliberately diverged: each bug a reference comparison confirmed is fixed | ||
| with a regression test on the seeded, bufferbloat-shaped simulation, the | ||
| controller follows draft-ietf-ccwg-bbr-07, and `rs/moq-quic/README.md`'s | ||
| ledger names the reference commits checked and every deliberate divergence. | ||
|
|
||
| Out of scope: differences only measurement could settle (when ProbeBW_UP and | ||
| DOWN end, minimum loss events per round, the initial window, clamping samples | ||
| by receive timestamps, a CE ratio instead of one CE), since the 2026-11-09 | ||
| deletion audit ([moq-dev/moq#8423](https://github.com/moq-dev/moq/pull/8423)) | ||
| pruned the study that would measure them; app-limited bandwidth aging, which | ||
| the [probe epic](/quest/m3/probe/README.md) owns; and quinn `main` commits, | ||
| which the [quinn sync](/quest/m3/vendor-sync/quinn.md) owns. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-14 (maintainer, /quest-plan): ✅ fix the confirmed bugs and | ||
| adopt draft-07 (rejected: bugs only, skipping draft-07's ProbeRTT rework; | ||
| also revive the pruned measurement study). | ||
|
|
||
| Found by a 2026-11-14 comparison against | ||
| [Google Linux BBRv3 `90210de4`](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c) | ||
| (unchanged since 2025-03), [draft-07](https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-07.html) | ||
| (2026-11-02), and [QUICHE BBR3 `51a950db`](https://github.com/google/quiche/blob/51a950dba2100da7e7977aa97759470507d1ebb8/quiche/quic/core/congestion_control/bbr3_sender.cc). | ||
| [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481)'s latest | ||
| force-push (`0ebaf7e2`) only rebased; its BBR3 is the `01b8848d` already | ||
| synced. | ||
|
|
||
| Constraint from the maintainer: no fix here reverts a MoQ change for | ||
| application-limited senders (ProbeRTT skip, app-limited bandwidth aging, | ||
| skipping app-limited rounds in plateau counting, the bandwidth probe's | ||
| headroom). A reference behavior that conflicts with one keeps ours. | ||
|
|
||
| This README's own work, after the children: update the BBR3 entries in | ||
| `rs/moq-quic/README.md` to name draft-07 and the three reference commits, | ||
| and list the divergences recorded only in code comments today (ProbeRTT | ||
| skip, app-limited aging, the one-packet send quantum floor, loss state saved | ||
| once per recovery episode, Linux's growth check) beside classic ECN. Record the reference heads there too, so each | ||
| [quinn sync](/quest/m3/vendor-sync/quinn.md) rechecks them with quinn#2481. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Startup loss counting](/quest/m2/bbr3-startup-loss.md) - draft-07's per-packet loss events, a breaking change before the cut | ||
| - [Probe up to a target bitrate](/quest/m3/probe/README.md) - owns app-limited aging and probe headroom | ||
| - [Realtime QUIC](/quest/m4/quic-realtime.md) - the bake-off for algorithms other than BBR3 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # [S] BBR3 recognizes bandwidth growth on any ACK | ||
|
|
||
| ## Goal | ||
|
|
||
| Under ACK aggregation, as on Wi-Fi and cellular uplinks, Startup and | ||
| ProbeBW_UP end on a real plateau rather than on three low samples that | ||
| happened to land on round starts. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-14 (maintainer, /quest-plan): adopt Linux's check, provided | ||
| it keeps every MoQ change for application-limited senders. The maintainer | ||
| answered "I don't know", with that constraint, so the pick follows the | ||
| recommendation; reopen it in review if that reading is wrong. | ||
|
|
||
| `check_full_bw_reached` (`rs/moq-congestion/src/bbr3/mod.rs`) returns unless | ||
| `round_start`, as draft-06 and -07 gate it. Linux checks every | ||
| non-application-limited ACK for 1.25x growth and resets the baseline on | ||
| any, counting plateau rounds only at round boundaries | ||
| ([tcp_bbr.c:1924-1948](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L1924-L1948)). | ||
| Move only `full_bw_count += 1` and the `app_limited_rounds` bookkeeping | ||
| under the round-start gate; application-limited samples still neither reset | ||
| nor advance the plateau. Test bursty, aggregated ACKs and a capacity that | ||
| rises mid-Startup, alongside the existing app-limited tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # [XS] BBR3's ACK aggregation estimate stays bounded across idle | ||
|
|
||
| ## Goal | ||
|
|
||
| A sender that goes idle between bursts, as media does when the RTT is | ||
| shorter than the frame interval, keeps `extra_acked` and the window it adds | ||
| bounded by the aggregation it actually sees. | ||
|
|
||
| ## Plan | ||
|
|
||
| `handle_restart_from_idle` (`rs/moq-congestion/src/bbr3/mod.rs`) resets | ||
| `extra_acked_interval_start` but not `extra_acked_delivered`, so each | ||
| resumed burst counts every earlier epoch's bytes as aggregation. The cap at | ||
| cwnd is circular, since `set_cwnd` grows cwnd toward `2·BDP + extra_acked`. | ||
| Simulated in the 2026-11-14 audit: 8 Mbit/s media at 30 fps over 20 ms grew | ||
| cwnd to 11 MB in 18 s (74 KB with the fix). Draft-06's | ||
| `BBRHandleRestartFromIdle` has the same omission; Linux zeroes | ||
| `ack_epoch_acked` there ([tcp_bbr.c:540-542](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L540-L542)). | ||
|
|
||
| Zero `extra_acked_delivered` with the interval start, and test that bursts | ||
| separated by idle gaps keep `extra_acked` and cwnd bounded. Caused by | ||
| [moq-dev/moq#4921](https://github.com/moq-dev/moq/pull/4921), which ported | ||
| the draft's omission. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # [M] Controllers hear an ACK's losses before the ACK | ||
|
|
||
| ## Goal | ||
|
|
||
| moq-quic reports the packets an ACK declares lost before it reports the ACK, | ||
| as RFC 9002's OnAckReceived orders them, so a congestion controller's loss | ||
| response takes effect on the same ACK, for every controller. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-14 (maintainer, /quest-plan): ✅ the transport reports losses | ||
| before the ACK (rejected: only refresh BBR3's pacing and cwnd at the end of | ||
| `on_loss`, which leaves the double cut). | ||
|
|
||
| Today `Connection::on_ack_received` (`rs/moq-quic/src/connection/mod.rs`) | ||
| calls `Controller::on_ack` and then `detect_lost_packets`, an order inherited | ||
| from quinn. With BBR3, a loss that ends ProbeBW_UP leaves pacing at 1.25x | ||
| and UP's window until the next ACK, since `on_loss` recomputes neither, and | ||
| losses found by the ACK that closes a loss round open a second round, so | ||
| DOWN, CRUISE, or Drain can cut the short-term model twice where Linux cuts | ||
| once ([tcp_bbr.c:2176-2201](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L2176-L2201)). | ||
|
|
||
| Change the order for all controllers (Cubic, SCReAM, and GCC included), | ||
| document it on `Controller`, and drop BBR3's note on the old order around | ||
| `on_spurious_loss`. A loss timer that fires with no ACK still needs BBR3's | ||
| `on_loss` to refresh pacing and cwnd. Test both effects. No single PR caused | ||
| it: the order is quinn's. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # [M] BBR3 schedules ProbeRTT as draft-07 does | ||
|
|
||
| ## Goal | ||
|
|
||
| BBR3 enters ProbeRTT when its 10 s `min_rtt` filter expires, as | ||
| [draft-07](https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-07.html#section-5.3.4) | ||
| does, halving ProbeRTT's cost for a backlogged sender, while an | ||
| application-limited sender still skips it and still learns a base RTT that | ||
| rose. | ||
|
|
||
| ## Plan | ||
|
|
||
| draft-07 drops the 5 s `ProbeRTTInterval` and the `probe_rtt_min_delay`, | ||
| `probe_rtt_min_stamp`, and `probe_rtt_expired` fields, adds | ||
| `BBR.probe_rtt_cwnd` (half the BDP from the old `min_rtt`) to | ||
| `ProbeRTTCwnd()`, calls `SaveCwnd()` before `EnterProbeRTT()` (ours saves | ||
| after, so `prior_cwnd` only ratchets up), and replaces `ack_phase` with | ||
| `is_bw_probe_stopping` and `is_bw_probe_sample` with `check_probe_loss`. | ||
| Adopt all of it, citing -07 anchors throughout `rs/moq-congestion/src/bbr3/`. | ||
| QUICHE BBR3 also uses a 10 s period. | ||
|
|
||
| The application-limited ProbeRTT skip stays. Decided 2026-11-14 (agent pick | ||
| after the maintainer answered "I don't know", for review), keeping the | ||
| skip's benefit for media: | ||
|
|
||
| - ✅ Spurious-loss undo restores `probe_rtt_skip_congested` to its value | ||
| when the episode began, since a loss proven spurious was not congestion, | ||
| while a CE or real loss earlier in the interval still denies the skip. | ||
| Save it with the other undo state in `enter_recovery`; restarting the | ||
| interval clears the saved value too. | ||
| - ✅ A skipped expiry adopts the expired window's lowest RTT sample as | ||
| `min_rtt` instead of only refreshing its stamp, so a base RTT that rose | ||
| is learned without running ProbeRTT; an application-limited sender's | ||
| samples carry little self-induced queue. | ||
| - Rejected: QUICHE BBRv1's kBBR7 guard (run ProbeRTT when RTT drifted past | ||
| 12.5%), which would put ProbeRTT's half-BDP cap back on media; and denying | ||
| the skip only on loss above the 2% threshold, which needs measurement. | ||
|
|
||
| The cut-short probe advance on ProbeRTT entry relies on the removed | ||
| PROBE_STARTING and PROBE_FEEDBACK phases; express it with | ||
| `check_probe_loss || is_bw_probe_stopping`. Test the 10 s schedule, the | ||
| save order, a base-RTT rise while application-limited, and a spurious loss | ||
| during a skip. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # [XS] BBR3 clears congestion signals in Drain and floors its Reno probe time | ||
|
|
||
| ## Goal | ||
|
|
||
| Leaving Startup charges a loss or CE mark once, and a small-BDP bulk flow | ||
| probes no more often than Linux's minimum of a few rounds. | ||
|
|
||
| ## Plan | ||
|
|
||
| - `enter_drain` (`rs/moq-congestion/src/bbr3/mod.rs`) does not reset the | ||
| congestion signals as Linux does on Drain entry | ||
| ([tcp_bbr.c:1963](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L1963)), | ||
| so the CE that ends Startup also cuts `bw_shortterm` and | ||
| `inflight_shortterm` one loss round later in Drain, contradicting the | ||
| ledger's once-per-episode CE response. Call `reset_congestion_signals()`. | ||
| - `is_reno_coexistence_probe_time` divides `min(BDP, cwnd)` by the MSS, | ||
| rounding down, with no floor: at 1 Mbit/s and 20 ms it probes every one or | ||
| two rounds, and DOWN can end before draining the probe's queue. Linux's | ||
| target goes through `bbr_inflight`, floored by the quantization budget | ||
| ([tcp_bbr.c:1010-1015](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L1010-L1015)). | ||
| Floor the BDP at `max(bdp, offload budget, min_pipe_cwnd)`, round up, | ||
| then cap it at cwnd as Linux's `bbr_target_inflight` does, so a cwnd cut | ||
| by congestion still shortens the wait. | ||
| Application-limited flows are unaffected, since their UP lasts until loss. | ||
|
|
||
| Test each. Caused by [moq-dev/moq#4921](https://github.com/moq-dev/moq/pull/4921), | ||
| which ported both from draft-06. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| # [S] BBR3 counts Startup loss events per lost packet | ||
|
|
||
| ## Goal | ||
|
|
||
| An application-limited media sender in Startup whose bottleneck shrinks | ||
| leaves Startup on high loss, as draft-07 specifies, and moq-congestion's | ||
| `Lost` drops the `new_burst` field that only the old counting needed. | ||
|
|
||
| ## Plan | ||
|
|
||
| `note_loss` (`rs/moq-congestion/src/bbr3/mod.rs`) counts discontiguous lost | ||
| ranges, as draft-06 read, from `Lost::new_burst`, which moq-quic computes. | ||
| [draft-07](https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-07.html#section-5.3.1.3) | ||
| counts every lost packet and resets the count in | ||
| `AdvanceLatestDeliverySignals`, `ResetCongestionSignals`, and spurious-loss | ||
| undo; ours resets it in none of the last two. A media sender bursting about | ||
| 14 packets per frame into a tail-drop link that shrank (50 to 3 Mbit/s) | ||
| loses one run per frame, one to three events per round, never six, and its | ||
| application-limited rounds block the plateau exit, so Startup never ends | ||
| and keyframe tails keep dropping. | ||
|
|
||
| Count per lost packet, reset where draft-07 does, remove `Lost::new_burst` | ||
| and its computation in moq-quic, and replace the even 1-in-25 drop test with | ||
| a tail-drop bottleneck test. | ||
|
|
||
| In m2, apart from [its epic](/quest/m1/bbr3-references/README.md): | ||
| removing `Lost::new_burst` changes moq-congestion's public API, which the | ||
| cut publishes for the first time. Decided 2026-11-15 (maintainer, | ||
| /quest-complete): ✅ keep it in m2 (rejected: m3, keeping the API). | ||
| Public API: removes `Lost::new_burst`. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [BBR3 matches its references](/quest/m1/bbr3-references/README.md) - the rest of the comparison's fixes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.