Skip to content

feat(moq-quic): port BBR3 and the controller callbacks as the default controller - #4921

Merged
kixelated merged 17 commits into
mainfrom
quest/m1/quic/fork/bbr3
Oct 7, 2026
Merged

kixelated merged 17 commits into
mainfrom
quest/m1/quic/fork/bbr3

Conversation

@kixelated

@kixelated kixelated commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

moq-quic (quinn-proto, #4879) still carried quinn's old BBR. MoQ's corrected BBR3 and the Controller callbacks it needs lived only in moq-dev/noq, which is frozen. Quest quest/m1/quic/fork/bbr3.md.

Approach

  • Base: quinn#2481, not a hand port of noq. The quest planned to move noq's bbr3/ over by hand, because Merge iroh-live's native media stack into moq-video/moq-audio (upstreaming plan) #2481 lacked the fork's fixes. Since then (2026-09-25) its author has ported moq-dev/noq's BBR fixes and regressions into Merge iroh-live's native media stack into moq-video/moq-audio (upstreaming plan) #2481. Its head (55f74c0d) is quinn-native, so I rebased its four commits onto the fork point with authorship kept. That brings in the Controller callbacks (packet number + SpaceId, sent/lost, cwnd-limited, app-limited, ACK frequency), the rate pacer that uses pacing_rate and send_quantum, and bandwidth_estimate. Diffed against noq's bbr3/mod.rs, the only fork change missing was classic ECN. Merge iroh-live's native media stack into moq-video/moq-audio (upstreaming plan) #2481 also has a few draft-06 changes noq lacks (window and Drain exit sized by BBR.bw).
  • Ported fix(proto): make BBR respond to classic ECN noq#12 (classic ECN): CE ends Startup, stops a probe, or lowers the short-term model, once per recovery episode, and is no longer counted as a loss. All 68 of the fork's BBR Sim tests are present (Merge iroh-live's native media stack into moq-video/moq-audio (upstreaming plan) #2481 adds 4 more).
  • Reworked the fork's ~580 lines of transport regressions onto quinn's Pair. They check packet identity across spaces and through a Retry, that CE is reported only when the count rises and never for invalid feedback, that starvation is reported before a resumed send but not for a backlog blocked by the window or pacer, and that BBR responds to marking at a bottleneck. Pair gained an ECN rewrite and a client-to-server Bottleneck queue.
  • Fixed a pacer spin: a wait under 1 ns truncated to zero and re-armed the pacing timer at now forever. This hung about 50 tests once BBR3 was the default. It now uses integer nanoseconds and rounds up. The regression test fails without the fix.
  • qlog reports pacing rate in bits/s. Controllers report bytes/s.
  • BBR3 is the default TransportConfig controller. Quinn's connection tests run Cubic through the harness helpers, as upstream's do. They script lockstep schedules at zero RTT that BBR3's rate pacer met only about 9 runs in 10. BBR3 tests opt in, and default_controller_is_bbr3 guards the default.
  • rs/moq-quic/README.md lists the carried changes. The quest is deleted, its references updated, and the benchmark against moq-noq moves to the switch quest.

Validation: clean-Nix just check passes after merging current main (437 moq-quic tests, format, rustdoc, dependency, quest, Markdown, and Nix checks). The qlog-feature suite also passes (439 tests). Its new regression verifies that a controller pacing rate of 125,000 bytes/s is emitted as 1,000,000 bits/s, and fails when the conversion is removed. Existing tests guard the BBR3 default, unchanged bytes/s PathStats estimate, rate pacing, packet-space callbacks, and classic ECN. The previous session ran the scripted connection tests 60 times without flakes.

Impact

moq-quic is unpublished, and nothing in the workspace consumes it yet.

  • congestion::Bbr/BbrConfig are removed. congestion::Bbr3/Bbr3Config are added, with the same names noq uses.
  • Controller: on_ack gains pn, space, on_end_acks gains space, and on_congestion_event gains largest_lost, space. New defaulted methods: on_packet_sent, on_packet_lost, on_cwnd_limited, on_app_limited, on_ack_frequency_update.
  • ControllerMetrics::send_quantum is new. pacing_rate and bandwidth_estimate are now bytes/s, and so is PathStats::bandwidth_estimate.
  • SpaceId is public (moq_quic::SpaceId, congestion::SpaceId) and #[non_exhaustive].
  • TransportConfig::default() now uses BBR3.
  • Wire: none. Pacing changes for controllers that report a rate.

Decisions

Confirmed by @kixelated in this session:

  1. ✅ Retain the updated quinn#2481 port plus classic ECN, including its draft-06 window and Drain behavior. Alternative: hand-port noq or revert those changes to match it exactly. The upstream head remains 55f74c0d.
  2. ✅ Keep BBR3 as TransportConfig::default(). Alternative: keep Cubic as the crate default because MoQ selects a controller family explicitly. The scripted zero-RTT connection harness keeps Cubic; dedicated BBR3 transport and simulation tests use BBR3.
  3. ✅ Use bytes/s for ControllerMetrics::{pacing_rate, bandwidth_estimate} and PathStats::bandwidth_estimate. Alternative: adopt quinn#2802's bits/s contract. Convert to bits/s only at the qlog boundary now and the web-transport-trait boundary during the existing switch quest.

Alternatives

Follow-ups

  • Switch quest: run the benchmark against moq-noq 1.3.x (could not run here without a relay on moq-quic), and multiply bandwidth_estimate by 8 for web-transport-trait.
  • bbr_marking_versus_dropping margins are looser than in noq's harness: in this harness, marking holds about 0.55x the queue time, against about 0.3x in noq's.

(Written by GPT-6)

kixelated and others added 14 commits October 6, 2026 00:46
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ry for some delay based algorithms such as BBRv3
… adjust the burst size and the overall pacing rate of the connection
Ports moq-dev/noq#12: CE exits Startup, stops a bandwidth probe, or lowers
the short-term model, once per recovery episode, and no longer counts as a
loss. Also adapts a cubic test quinn added after quinn#2481's base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At a high pacing rate a few bytes of shortfall take under a nanosecond to
earn. Truncating that wait to zero re-armed the pacing timer at the current
instant forever. Earn and wait in integer nanoseconds, rounding the wait up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lockstep handshake and GSO tests script exact packet schedules that the
window-derived pacer meets at the harness's zero RTT; pin them to Cubic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Controllers report their pacing rate in bytes per second, which the pacer
consumes; qlog's field is bits per second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reworks the fork's connection-level regressions onto quinn's Pair harness:
packet identity across spaces and through a Retry, CE reported only on a
count increase and never for invalid feedback, starvation reported before a
resumed send but not for a window- or pacing-blocked backlog, and BBR's
classic ECN response at a bottleneck. The harness gains an ECN rewrite and
a client-to-server bottleneck queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They script lockstep packet schedules at zero RTT, which BBR3's rate pacing
meets only as far as the randomly sized handshake left budget, so about one
run in ten failed somewhere. Run them on Cubic as upstream does, and opt the
BBR3 tests in. A new test checks BBR3 stays the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome of the unattended /quest-start run. This is a draft because these decisions are yours:

  1. Base on quinn#2481 instead of hand-porting noq's bbr3/. Merge iroh-live's native media stack into moq-video/moq-audio (upstreaming plan) #2481 now carries the fork's fixes, and noq#12 (classic ECN) is ported on top. Recommend keeping this. The alternative is to also revert Merge iroh-live's native media stack into moq-video/moq-audio (upstreaming plan) #2481's draft-06 changes that size the window and Drain exit by BBR.bw, to match moq-noq exactly. On the marking/dropping simulation the two were indistinguishable.
  2. BBR3 as TransportConfig::default(). This needs quinn's connection tests pinned to Cubic, because about 10% of runs flaked under BBR3's rate pacing at zero RTT. Recommend keeping it, as the quest asked. Alternative: keep Cubic as the crate default, since MoQ always picks a family explicitly.
  3. Units. Recommend bytes/s for ControllerMetrics::{pacing_rate, bandwidth_estimate} and PathStats::bandwidth_estimate (from Merge iroh-live's native media stack into moq-video/moq-audio (upstreaming plan) #2481), with qlog converted to bits/s. Alternative: quinn#2802's bits/s, which needs a ×8 in BBR3 and ÷8 in the pacer.

Not done: the benchmark against moq-noq 1.3.x. It needs the relay on moq-quic, so it moved to the switch quest.

(Written by Claude Opus 5.5)

kixelated and others added 2 commits October 6, 2026 10:35
Guard the bytes-per-second controller contract at the qlog boundary, which emits bits per second.

Co-Authored-By: GPT-6 <noreply@openai.com>
@kixelated
kixelated marked this pull request as ready for review October 6, 2026 17:35
@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cd4da6f2-b825-4c8e-a597-48e05ce8e0fb
📥 Commits

Reviewing files that changed from the base of the PR and between 4453428 and ef090c6.

📒 Files selected for processing (25)
  • quest/m1/bbr-ack-cleanup.md
  • quest/m1/quic/README.md
  • quest/m1/quic/fork/README.md
  • quest/m1/quic/fork/bbr3.md
  • quest/m1/quic/fork/switch.md
  • quest/m2/quic-bbr-loss-parity.md
  • rs/moq-quic/README.md
  • rs/moq-quic/src/config/transport.rs
  • rs/moq-quic/src/congestion.rs
  • rs/moq-quic/src/congestion/bbr/bw_estimation.rs
  • rs/moq-quic/src/congestion/bbr/min_max.rs
  • rs/moq-quic/src/congestion/bbr/mod.rs
  • rs/moq-quic/src/congestion/bbr3/max_filter.rs
  • rs/moq-quic/src/congestion/bbr3/mod.rs
  • rs/moq-quic/src/congestion/cubic.rs
  • rs/moq-quic/src/congestion/new_reno.rs
  • rs/moq-quic/src/connection/mod.rs
  • rs/moq-quic/src/connection/pacing.rs
  • rs/moq-quic/src/connection/packet_builder.rs
  • rs/moq-quic/src/connection/paths.rs
  • rs/moq-quic/src/connection/stats.rs
  • rs/moq-quic/src/lib.rs
  • rs/moq-quic/src/packet.rs
  • rs/moq-quic/src/tests/mod.rs
  • rs/moq-quic/src/tests/util.rs
💤 Files with no reviewable changes (5)
  • quest/m1/quic/fork/README.md
  • rs/moq-quic/src/congestion/bbr/bw_estimation.rs
  • quest/m1/quic/fork/bbr3.md
  • rs/moq-quic/src/congestion/bbr/min_max.rs
  • rs/moq-quic/src/congestion/bbr/mod.rs

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.


Walkthrough

moq-quic now uses BBR3 as its default congestion controller. The controller API and connection code pass packet numbers and packet-number spaces to congestion events and support send-quantum and application-limited callbacks. The pacer uses controller metrics, including an optional pacing rate. Tests cover the updated callbacks, pacing, ECN behavior, and BBR3 behavior. Quest plans and repository notes now describe BBR3 as present in moq-quic.

Estimated code review effort

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to ef090

This change makes BBR3 the default congestion controller for the unpublished moq-quic crate and updates pacing and controller callbacks. No concrete defects remain open, and the crate is not yet used elsewhere in the workspace, so it appears ready to merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 59.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 159 functions across 14 files. (5 skipped… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: porting BBR3 and its controller callbacks, then making BBR3 the default controller.
Description check ✅ Passed The description directly explains the problem, implementation, validation, API impact, and deferred benchmark related to the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 59.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 159 functions across 14 files. (5 skipped: 5 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent Codex review of final head 3fab9b4. No blocking finding in the transport integration, packet-space callback identities, controller pacing and send-quantum enforcement, classic ECN handling, default-controller selection, or final qlog regression.

Rates consistently use bytes/s in the controller, pacer and PathStats; the qlog boundary converts to bits/s with saturating multiplication. Pacing waits earn only the token deficit and round up rather than rearming at the current instant. The packet-space and congestion simulations exercise callback identities and ECN without relying on wall-clock sleeps. The final regression observes 125,000 bytes/s as 1,000,000 bits/s in an actual qlog event; the iteration report documents its failing negative control and 437 default/439 qlog tests passing.

This was a focused review of integration, carried behavior and regression coverage, not an exhaustive independent derivation of the imported BBR state machine. The user confirmed the upstream quinn#2481 base, BBR3 crate default and bytes/s contract. Relay/noq throughput and latency comparison remains in the existing consumer-switch quest; this PR alone does not establish production relay performance. Final-head CI remains the merge gate.

Public API: replaces Bbr/BbrConfig with Bbr3/Bbr3Config, expands Controller callbacks, exposes SpaceId and send_quantum, and makes BBR3 the crate default. No wire format change.

(Written by GPT-6)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 3fab9b4

No additional actionable regression found in this focused source review. This agrees with the existing independent review; no duplicate inline comments.

Provenance checked against quinn-rs/quinn@55f74c0 and moq-dev/noq@c85041b: max_filter.rs matches upstream; the BBR module's upstream delta carries classic ECN, while the noq comparison retains the documented draft-06 bandwidth-bound changes. All 68 noq BBR test names are present, with four additional bandwidth-bound regressions.

Direction: keeping the upstream-derived controller and isolating transport adaptations is reasonable. Checked packet-space/Retry callbacks, ACK/loss/CE ordering, app- versus cwnd-limited signals, explicit-rate pacing and GSO bounds, plus API/unit changes. The qlog conversion is correctly confined to connection/paths.rs:215–219; no dependency-manifest or wire-parser changes are introduced.

Remaining validation matters before the consumer switch: the BBR3 default at config/transport.rs:410 is not broadly exercised by the Cubic-pinned scripted harness. Retain the relay/noq throughput-and-latency comparison in quest/m1/quic/fork/switch.md and the bytes/s-to-bits/s consumer conversion. The full-flight scans at congestion/bbr3/mod.rs:1875–1882 remain a scaling concern already tracked by the ACK-cleanup quest, not a newly discovered port defect.

Limits: source/provenance and regression-coverage review, not an exhaustive BBR state-machine or security audit; I did not rerun builds, tests, or network benchmarks. GitHub's Check and Platform runs for this head report success. This comment does not establish production performance or merge readiness.

# Conflicts:
#	quest/m1/quic/fork/README.md
#	quest/m1/quic/fork/switch.md
#	rs/moq-quic/README.md
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged current main into final head ef090c6e20f1a2b68ead4a5f8ecbcbac505c1bde. Resolved only three quest/README conflicts: retain both BBR3 and the already-landed UDP carried changes, remove completed quest prerequisites, and keep the relay/noq benchmark in the existing switch quest. No BBR3 source changed during conflict resolution.

The confirmed decisions remain quinn#2481 plus classic ECN, BBR3 as the default, and bytes/s in controller and PathStats metrics with conversion at qlog. Public API impact remains the Bbr3/Bbr3Config replacement, expanded Controller callbacks, public SpaceId and send_quantum, and BBR3 default. Wire format is unchanged.

Clean-Nix just check passes, including 437 tests; just rs test -p moq-quic --features qlog passes all 439 tests. Final-head CI and review remain the gates before the normal main merge queue.

(Written by GPT-6)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Automated review: ef090c6e

Full review of the BBR3 port (no earlier Grok review). I read the transport integration (connection/mod.rs, pacing.rs, packet_builder.rs, paths.rs) and the non-test half of bbr3/mod.rs. The callback wiring is consistent: on_packet_sent and on_ack both cover only ack-eliciting packets, per-space queues keep packet numbers ordered for the binary searches, lost MTU probes still skip the congestion response, and the integer-nanosecond delay_at_rate rounds up correctly. Breaking API changes are called out in the description, and the crate is unpublished. I found no blocking issues.

Non-blocking

  1. on_end_acks re-marks app-limited when the window is full (congestion/bbr3/mod.rs:1869). on_app_limited (:1801) skips marking when in_flight >= cwnd, matching draft-06 CheckIfApplicationLimited. But on_end_acks then sets self.app_limited = max(delivered + inflight, 1) whenever the transport's app_limited flag is true, without that check. The transport sets the flag on any empty, unblocked poll (connection/mod.rs:993), including one where the app ran dry just as it filled the window, since the cwnd check only runs when there's data to send. In that case, samples from a window-limited flight come out labeled app-limited. That keeps them from lowering max_bw and stops them counting toward the Startup full-bandwidth check. Since the transport now reports starvation through on_app_limited on every empty poll, I'd drop the else if app_limited branch, or give it the same inflight < cwnd guard. A transport test with a write that exactly fills the window and then stops would pin this. It's separate from what quest/m1/quic/bbr-app-limited-edges.md covers.

  2. Per-ACK cost is O(whole flight) and now the default for every connection (:1874). on_end_acks runs retain and then a second full pass over all three queues on every ACK. quest/m1/bbr-ack-cleanup.md already tracks this (162 ms to drain 16k packets at 8 per ACK). Flipping TransportConfig::default() to BBR3 means any relay built on moq-quic pays it on every high-BDP connection. That isn't a reason to hold this PR, but it's worth landing that quest before switch.md moves real traffic. A cheap interim fix is to pop stale entries off the front of each queue only. Entries are in send order, so both the acknowledged prefix and the over-10-rounds entries sit at the front in the common case.

  3. The metrics snapshot comment overstates its guarantee (connection/mod.rs:481). on_packet_sent calls handle_restart_from_idle, which in ProbeBW resets pacing_rate to 1.0x bw (:1341), so the snapshot can go stale after the first packet of a post-idle burst. The impact is negligible, since now is fixed within the call and the pacer only drains tokens there. But the comment should say the snapshot may lag a restart from idle by one call, so nobody relies on it later.

CI: Quest, macOS, and Windows pass; Check and Test are still pending.

Verdict: MERGE once Check and Test are green. Item 1 is a small follow-up worth doing here or in the app-limited edges quest.

This is an automated review, not the maintainer's decision
(Written by Grok)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent final-head review of ef090c6 following the previously reviewed 3fab9b4.

No actionable finding in the integration delta. All rs/moq-quic/src files are byte-for-byte unchanged from the reviewed implementation. The merge preserves both BBR3 and main's UDP carried-change documentation and removes prerequisites that main already completed. The relay/noq benchmark and bytes/s consumer conversion remain in the switch quest.

Verified the final merge diff and current main's quest context. The merge agent reports clean-Nix just check (437 tests) and the qlog suite (439 tests) passing; I did not independently rerun them. Final-head CI must pass before landing. The public API and wire impact remain as documented in the PR.

(Written by GPT-6)

@kixelated
kixelated merged commit 8ecd2ca into main Oct 7, 2026
7 checks passed
@kixelated
kixelated deleted the quest/m1/quic/fork/bbr3 branch October 7, 2026 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants