Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions quest/m1/bbr3-references/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# BBR3 matches its references

## Goal

moq-congestion's BBR3 matches its references wherever it has not
deliberately diverged: each bug a reference comparison confirmed is fixed
with a regression test on the seeded, bufferbloat-shaped simulation, the
controller follows draft-ietf-ccwg-bbr-07, and `rs/moq-quic/README.md`'s
ledger names the reference commits checked and every deliberate divergence.

Out of scope: differences only measurement could settle (when ProbeBW_UP and
DOWN end, minimum loss events per round, the initial window, clamping samples
by receive timestamps, a CE ratio instead of one CE), since the 2026-11-09
deletion audit ([moq-dev/moq#8423](https://github.com/moq-dev/moq/pull/8423))
pruned the study that would measure them; app-limited bandwidth aging, which
the [probe epic](/quest/m3/probe/README.md) owns; and quinn `main` commits,
which the [quinn sync](/quest/m3/vendor-sync/quinn.md) owns.

## Plan

Decided 2026-11-14 (maintainer, /quest-plan): ✅ fix the confirmed bugs and
adopt draft-07 (rejected: bugs only, skipping draft-07's ProbeRTT rework;
also revive the pruned measurement study).

Found by a 2026-11-14 comparison against
[Google Linux BBRv3 `90210de4`](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c)
(unchanged since 2025-03), [draft-07](https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-07.html)
(2026-11-02), and [QUICHE BBR3 `51a950db`](https://github.com/google/quiche/blob/51a950dba2100da7e7977aa97759470507d1ebb8/quiche/quic/core/congestion_control/bbr3_sender.cc).
[quinn#2481](https://github.com/quinn-rs/quinn/pull/2481)'s latest
force-push (`0ebaf7e2`) only rebased; its BBR3 is the `01b8848d` already
synced.

Constraint from the maintainer: no fix here reverts a MoQ change for
application-limited senders (ProbeRTT skip, app-limited bandwidth aging,
skipping app-limited rounds in plateau counting, the bandwidth probe's
headroom). A reference behavior that conflicts with one keeps ours.

This README's own work, after the children: update the BBR3 entries in
`rs/moq-quic/README.md` to name draft-07 and the three reference commits,
and list the divergences recorded only in code comments today (ProbeRTT
skip, app-limited aging, the one-packet send quantum floor, loss state saved
once per recovery episode, Linux's growth check) beside classic ECN. Record the reference heads there too, so each
[quinn sync](/quest/m3/vendor-sync/quinn.md) rechecks them with quinn#2481.

## Related

- [Startup loss counting](/quest/m2/bbr3-startup-loss.md) - draft-07's per-packet loss events, a breaking change before the cut
- [Probe up to a target bitrate](/quest/m3/probe/README.md) - owns app-limited aging and probe headroom
- [Realtime QUIC](/quest/m4/quic-realtime.md) - the bake-off for algorithms other than BBR3
24 changes: 24 additions & 0 deletions quest/m1/bbr3-references/growth.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# [S] BBR3 recognizes bandwidth growth on any ACK

## Goal

Under ACK aggregation, as on Wi-Fi and cellular uplinks, Startup and
ProbeBW_UP end on a real plateau rather than on three low samples that
happened to land on round starts.

## Plan

Decided 2026-11-14 (maintainer, /quest-plan): adopt Linux's check, provided
it keeps every MoQ change for application-limited senders. The maintainer
answered "I don't know", with that constraint, so the pick follows the
recommendation; reopen it in review if that reading is wrong.

`check_full_bw_reached` (`rs/moq-congestion/src/bbr3/mod.rs`) returns unless
`round_start`, as draft-06 and -07 gate it. Linux checks every
non-application-limited ACK for 1.25x growth and resets the baseline on
any, counting plateau rounds only at round boundaries
([tcp_bbr.c:1924-1948](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L1924-L1948)).
Move only `full_bw_count += 1` and the `app_limited_rounds` bookkeeping
under the round-start gate; application-limited samples still neither reset
nor advance the plateau. Test bursty, aggregated ACKs and a capacity that
rises mid-Startup, alongside the existing app-limited tests.
23 changes: 23 additions & 0 deletions quest/m1/bbr3-references/idle-aggregation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# [XS] BBR3's ACK aggregation estimate stays bounded across idle

## Goal

A sender that goes idle between bursts, as media does when the RTT is
shorter than the frame interval, keeps `extra_acked` and the window it adds
bounded by the aggregation it actually sees.

## Plan

`handle_restart_from_idle` (`rs/moq-congestion/src/bbr3/mod.rs`) resets
`extra_acked_interval_start` but not `extra_acked_delivered`, so each
resumed burst counts every earlier epoch's bytes as aggregation. The cap at
cwnd is circular, since `set_cwnd` grows cwnd toward `2·BDP + extra_acked`.
Simulated in the 2026-11-14 audit: 8 Mbit/s media at 30 fps over 20 ms grew
cwnd to 11 MB in 18 s (74 KB with the fix). Draft-06's
`BBRHandleRestartFromIdle` has the same omission; Linux zeroes
`ack_epoch_acked` there ([tcp_bbr.c:540-542](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L540-L542)).

Zero `extra_acked_delivered` with the interval start, and test that bursts
separated by idle gaps keep `extra_acked` and cwnd bounded. Caused by
[moq-dev/moq#4921](https://github.com/moq-dev/moq/pull/4921), which ported
the draft's omission.
27 changes: 27 additions & 0 deletions quest/m1/bbr3-references/loss-order.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# [M] Controllers hear an ACK's losses before the ACK

## Goal

moq-quic reports the packets an ACK declares lost before it reports the ACK,
as RFC 9002's OnAckReceived orders them, so a congestion controller's loss
response takes effect on the same ACK, for every controller.

## Plan

Decided 2026-11-14 (maintainer, /quest-plan): ✅ the transport reports losses
before the ACK (rejected: only refresh BBR3's pacing and cwnd at the end of
`on_loss`, which leaves the double cut).

Today `Connection::on_ack_received` (`rs/moq-quic/src/connection/mod.rs`)
calls `Controller::on_ack` and then `detect_lost_packets`, an order inherited
from quinn. With BBR3, a loss that ends ProbeBW_UP leaves pacing at 1.25x
and UP's window until the next ACK, since `on_loss` recomputes neither, and
losses found by the ACK that closes a loss round open a second round, so
DOWN, CRUISE, or Drain can cut the short-term model twice where Linux cuts
once ([tcp_bbr.c:2176-2201](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L2176-L2201)).

Change the order for all controllers (Cubic, SCReAM, and GCC included),
document it on `Controller`, and drop BBR3's note on the old order around
`on_spurious_loss`. A loss timer that fires with no ACK still needs BBR3's
`on_loss` to refresh pacing and cwnd. Test both effects. No single PR caused
it: the order is quinn's.
43 changes: 43 additions & 0 deletions quest/m1/bbr3-references/probe-rtt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# [M] BBR3 schedules ProbeRTT as draft-07 does

## Goal

BBR3 enters ProbeRTT when its 10 s `min_rtt` filter expires, as
[draft-07](https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-07.html#section-5.3.4)
does, halving ProbeRTT's cost for a backlogged sender, while an
application-limited sender still skips it and still learns a base RTT that
rose.

## Plan

draft-07 drops the 5 s `ProbeRTTInterval` and the `probe_rtt_min_delay`,
`probe_rtt_min_stamp`, and `probe_rtt_expired` fields, adds
`BBR.probe_rtt_cwnd` (half the BDP from the old `min_rtt`) to
`ProbeRTTCwnd()`, calls `SaveCwnd()` before `EnterProbeRTT()` (ours saves
after, so `prior_cwnd` only ratchets up), and replaces `ack_phase` with
`is_bw_probe_stopping` and `is_bw_probe_sample` with `check_probe_loss`.
Adopt all of it, citing -07 anchors throughout `rs/moq-congestion/src/bbr3/`.
QUICHE BBR3 also uses a 10 s period.

The application-limited ProbeRTT skip stays. Decided 2026-11-14 (agent pick
after the maintainer answered "I don't know", for review), keeping the
skip's benefit for media:

- ✅ Spurious-loss undo restores `probe_rtt_skip_congested` to its value
when the episode began, since a loss proven spurious was not congestion,
while a CE or real loss earlier in the interval still denies the skip.
Save it with the other undo state in `enter_recovery`; restarting the
interval clears the saved value too.
- ✅ A skipped expiry adopts the expired window's lowest RTT sample as
`min_rtt` instead of only refreshing its stamp, so a base RTT that rose
is learned without running ProbeRTT; an application-limited sender's
samples carry little self-induced queue.
- Rejected: QUICHE BBRv1's kBBR7 guard (run ProbeRTT when RTT drifted past
12.5%), which would put ProbeRTT's half-BDP cap back on media; and denying
the skip only on loss above the 2% threshold, which needs measurement.

The cut-short probe advance on ProbeRTT entry relies on the removed
PROBE_STARTING and PROBE_FEEDBACK phases; express it with
`check_probe_loss || is_bw_probe_stopping`. Test the 10 s schedule, the
save order, a base-RTT rise while application-limited, and a spurious loss
during a skip.
27 changes: 27 additions & 0 deletions quest/m1/bbr3-references/transitions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# [XS] BBR3 clears congestion signals in Drain and floors its Reno probe time

## Goal

Leaving Startup charges a loss or CE mark once, and a small-BDP bulk flow
probes no more often than Linux's minimum of a few rounds.

## Plan

- `enter_drain` (`rs/moq-congestion/src/bbr3/mod.rs`) does not reset the
congestion signals as Linux does on Drain entry
([tcp_bbr.c:1963](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L1963)),
so the CE that ends Startup also cuts `bw_shortterm` and
`inflight_shortterm` one loss round later in Drain, contradicting the
ledger's once-per-episode CE response. Call `reset_congestion_signals()`.
- `is_reno_coexistence_probe_time` divides `min(BDP, cwnd)` by the MSS,
rounding down, with no floor: at 1 Mbit/s and 20 ms it probes every one or
two rounds, and DOWN can end before draining the probe's queue. Linux's
target goes through `bbr_inflight`, floored by the quantization budget
([tcp_bbr.c:1010-1015](https://github.com/google/bbr/blob/90210de4b779d40496dee0b89081780eeddf2a60/net/ipv4/tcp_bbr.c#L1010-L1015)).
Floor the BDP at `max(bdp, offload budget, min_pipe_cwnd)`, round up,
then cap it at cwnd as Linux's `bbr_target_inflight` does, so a cwnd cut
by congestion still shortens the wait.
Application-limited flows are unaffected, since their UP lasts until loss.

Test each. Caused by [moq-dev/moq#4921](https://github.com/moq-dev/moq/pull/4921),
which ported both from draft-06.
34 changes: 34 additions & 0 deletions quest/m2/bbr3-startup-loss.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# [S] BBR3 counts Startup loss events per lost packet

## Goal

An application-limited media sender in Startup whose bottleneck shrinks
leaves Startup on high loss, as draft-07 specifies, and moq-congestion's
`Lost` drops the `new_burst` field that only the old counting needed.

## Plan

`note_loss` (`rs/moq-congestion/src/bbr3/mod.rs`) counts discontiguous lost
ranges, as draft-06 read, from `Lost::new_burst`, which moq-quic computes.
[draft-07](https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-07.html#section-5.3.1.3)
counts every lost packet and resets the count in
`AdvanceLatestDeliverySignals`, `ResetCongestionSignals`, and spurious-loss
undo; ours resets it in none of the last two. A media sender bursting about
14 packets per frame into a tail-drop link that shrank (50 to 3 Mbit/s)
loses one run per frame, one to three events per round, never six, and its
application-limited rounds block the plateau exit, so Startup never ends
and keyframe tails keep dropping.

Count per lost packet, reset where draft-07 does, remove `Lost::new_burst`
and its computation in moq-quic, and replace the even 1-in-25 drop test with
a tail-drop bottleneck test.

In m2, apart from [its epic](/quest/m1/bbr3-references/README.md):
removing `Lost::new_burst` changes moq-congestion's public API, which the
cut publishes for the first time. Decided 2026-11-15 (maintainer,
/quest-complete): ✅ keep it in m2 (rejected: m3, keeping the API).
Public API: removes `Lost::new_burst`. Wire: none.

## Related

- [BBR3 matches its references](/quest/m1/bbr3-references/README.md) - the rest of the comparison's fixes
10 changes: 10 additions & 0 deletions quest/m3/vendor-sync/quinn.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,16 @@ and [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481)'s BBR3 through
all recorded in `rs/moq-quic/README.md`'s `Upstream` section with every
disposition. Start there, and record the new heads in the same place.

Checked 2026-11-14 (/quest-plan): quinn#2481 force-pushed to
[`0ebaf7e2`](https://github.com/quinn-rs/quinn/pull/2481/commits/0ebaf7e2c9932e432ef5e7caf916291b02d8277e),
Comment thread
kixelated marked this conversation as resolved.
Resolved
a rebase onto `c7b46a96` whose BBR3 commits match `01b8848d`, so it has
nothing to port; recorded in the ledger. Of the nine `main` commits since `aea1e538`, check
[`aa61ce99`](https://github.com/quinn-rs/quinn/commit/aa61ce996b97b21c63731a84fac1346c2d70852b) (late
ACK-only packets are not spurious losses) against BBR3's spurious-loss undo
first. The [BBR3 references epic](/quest/m1/bbr3-references/README.md) adds
Google Linux BBRv3, draft-ietf-ccwg-bbr, and QUICHE BBR3 heads to the ledger;
recheck them with quinn#2481 each run.

Learned: a fix missing from the ledger is often already held, since the fork
reworked the same code (`stop` under `RESET_STREAM_AT`, the stale-accept
check under `handshake_timeout`, socket setup in `moq_sock::bind`), so check
Expand Down
2 changes: 1 addition & 1 deletion rs/moq-quic/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Changes on top of the upstream commit, besides the renames:
- **Close reasons**: a local close reports `connection::Error::LocallyClosed` with its code and reason from `poll_closed` and `error()`, a handshake that idles out reports `HandshakeTimeout`, and close reasons are strings, lossily decoded.
- **No early data**: TLS never offers or accepts 0-RTT, and early packets are discarded without decryption. The early keys, remembered transport parameters, stream rollback, and early-session driver APIs are removed. Session tickets still resume 1-RTT handshakes. MoQ's CONNECT and authorization already wait for the handshake.
- **BBR3**, from [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481) at [`55f74c0d`](https://github.com/quinn-rs/quinn/pull/2481/commits/55f74c0dd0738b5ad7d7676c0b2869a547783f83) (open upstream), rebased onto the fork point with authorship kept. It replaces quinn's BBR and is the default controller. It carries the BBR correctness fixes moq-dev/noq shipped in `moq-noq` 1.3.1, and with them the `Controller` changes BBR needs: per-packet send and loss callbacks, `on_cwnd_limited`, `on_app_limited`, `on_ack_frequency_update`, and controller-owned pacing and burst size. `Metrics` rates are bytes per second.
- **BBR3 upstream refresh**: Checked against the force-pushed [`01b8848d`](https://github.com/quinn-rs/quinn/pull/2481/commits/01b8848dcb321eb17ad4d88dd6f84f28f45a0d6a) on 2026-10-31: carry startup's per-round ACK aggregation maximum, restore the saved window once when recovery ends, preserve it through recovery into ProbeRTT, saturate probe-bound growth, and stop a probe at the previous loss bound before accelerating again on clean feedback. Spurious-loss undo also restores that probe history, and an idle ACK renews application-limited marking even when it crosses the old marker. The two-cycle bandwidth filter and transport-owned packet snapshots already cover its other fixes. Upstream's bandwidth metric conversion to bits/s does not apply: this fork's controller metrics use bytes/s.
- **BBR3 upstream refresh**: Checked against the force-pushed [`01b8848d`](https://github.com/quinn-rs/quinn/pull/2481/commits/01b8848dcb321eb17ad4d88dd6f84f28f45a0d6a) on 2026-10-31: carry startup's per-round ACK aggregation maximum, restore the saved window once when recovery ends, preserve it through recovery into ProbeRTT, saturate probe-bound growth, and stop a probe at the previous loss bound before accelerating again on clean feedback. Spurious-loss undo also restores that probe history, and an idle ACK renews application-limited marking even when it crosses the old marker. The two-cycle bandwidth filter and transport-owned packet snapshots already cover its other fixes. Upstream's bandwidth metric conversion to bits/s does not apply: this fork's controller metrics use bytes/s. Rechecked against the force-pushed [`0ebaf7e2`](https://github.com/quinn-rs/quinn/pull/2481/commits/0ebaf7e2c9932e432ef5e7caf916291b02d8277e) on 2026-11-14: a rebase onto quinn `main` at `c7b46a96` whose BBR3 commits match `01b8848d`, so nothing to port.
- **BBR3 loss samples**: a lost packet supplies its own inflight, loss, and application-limited state without borrowing or overwriting the ACK sample, including before the first ACK observation. This follows draft-06 and Google Linux BBRv3; upstream submission waits on quinn#2481 for the same callback adaptation as packet retirement.
- **BBR3 packet retirement**: per-space packet lookup and round-indexed expiry retire completed snapshots incrementally, preserving the extra ACK batch of metadata for late loss callbacks. The bounded indexes handle sparse packet numbers without walking the live flight. Upstream submission is deferred while quinn#2481 remains open: this fork also carries the packet-space and loss callback adaptations, so a port needs to target that implementation after its shape settles.
- **Classic ECN for BBR3**, ported from [moq-dev/noq#12](https://github.com/moq-dev/noq/pull/12): CE exits Startup, stops a bandwidth probe, or lowers the short-term model, once per recovery episode, instead of counting as a loss.
Expand Down