From 67c42658f70c3da0b957797859126b9d347cf1cc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 31 Oct 2026 15:23:01 -0700 Subject: [PATCH 1/7] chore(quest): claim quest/m1/vendor-sync/quinn.md Quest-Claim: 01a91a51935fbafe From 9d15d25a720ec0d0f1b761fdc896b0740c8c275e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 31 Oct 2026 15:30:44 -0700 Subject: [PATCH 2/7] fix(quic): port Quinn and BBR3 correctness fixes Co-Authored-By: GPT-6.1 Sol --- Cargo.lock | 1 + quest/m1/vendor-sync/quinn.md | 59 ++++++------ rs/moq-congestion/src/bbr3/bench.rs | 24 ++++- rs/moq-congestion/src/bbr3/mod.rs | 138 +++++++++++++++++++++++++--- rs/moq-congestion/src/rate.rs | 3 +- rs/moq-quic/README.md | 13 ++- rs/moq-sock/Cargo.toml | 3 + rs/moq-sock/src/udp/apple_fast.rs | 14 +-- rs/moq-sock/src/udp/unix.rs | 6 +- rs/moq-sock/tests/udp/main.rs | 40 ++++++++ 10 files changed, 248 insertions(+), 53 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index b2b82342ed4..e3687edc418 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5917,6 +5917,7 @@ dependencies = [ "cfg_aliases", "core_affinity", "libc", + "nix", "rand 0.10.3", "socket2 0.6.5", "thiserror 2.0.21", diff --git a/quest/m1/vendor-sync/quinn.md b/quest/m1/vendor-sync/quinn.md index e56327b6052..37d6b35e2bd 100644 --- a/quest/m1/vendor-sync/quinn.md +++ b/quest/m1/vendor-sync/quinn.md @@ -17,32 +17,39 @@ checking upstream is recurring work. On 2026-10-30 the maintainer moved it into the vendor sync with moq's other vendored code. Each run also checks [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481), -the open BBR3 PR moq-congestion carries from -[`55f74c0d`](https://github.com/quinn-rs/quinn/pull/2481/commits/55f74c0dd0738b5ad7d7676c0b2869a547783f83), -for fixes to port, and records the PR commit checked in the README's BBR3 -entry. It was force-pushed to `01b8848d` on 2026-10-20. - -Last checked: the fork point -[`7616e6b2`](https://github.com/quinn-rs/quinn/commit/7616e6b2782722f3ce4a1b181ef08817d7f545e4) -(2026-10-05), plus the commits the README lists as ported. Upstream `main` was -at [`a2c55b36`](https://github.com/quinn-rs/quinn/commit/a2c55b36817ccaac1c39c6db92de8294f281819f) -(2026-10-30), 18 commits later. Record the newest upstream commit checked in -`rs/moq-quic/README.md`'s `Upstream` section. - -First run: four fixes on quinn `main` are not in the ledger yet. Check each -against the fork's code before porting, since a missing ledger entry does not -prove the bug remains: -[`f233510c`](https://github.com/quinn-rs/quinn/commit/f233510c) (honor a -disabled idle timeout when accepting), -[`6c69f9f4`](https://github.com/quinn-rs/quinn/commit/6c69f9f4) (don't -credit a reset stream's unread data twice on stop; the fork rewrote `stop` -under the README's "Stream credit on stop", so it may already hold), -[`623a0369`](https://github.com/quinn-rs/quinn/commit/623a0369) (don't make -specific IPv6 endpoints dual-stack), and -[`a2c55b36`](https://github.com/quinn-rs/quinn/commit/a2c55b36) (IPv4 source -address on Apple, in `moq_sock::udp`). Also judge the new `read_datagrams` -API ([`2697de1e`](https://github.com/quinn-rs/quinn/commit/2697de1e)), a -feature: plan it with `/quest-plan` if moq-tokio's driver would use it. +the open BBR3 PR carried from +[`55f74c0d`](https://github.com/quinn-rs/quinn/pull/2481/commits/55f74c0dd0738b5ad7d7676c0b2869a547783f83). + +Implementation checkpoint 2026-10-31 in +[moq-dev/moq#7458](https://github.com/moq-dev/moq/pull/7458): compared `main` +through [`2c575013`](https://github.com/quinn-rs/quinn/commit/2c5750139fa09aab588bf0f939c83f521d658baf), +`0.11.x` through +[`4b84837d`](https://github.com/quinn-rs/quinn/commit/4b84837df1d25424bc37b0ed990e7b376d40a9c0), +and BBR3 through +[`01b8848d`](https://github.com/quinn-rs/quinn/pull/2481/commits/01b8848dcb321eb17ad4d88dd6f84f28f45a0d6a). +All 13 published advisories predate the fork point. The README records the +ports and why the other changes already hold or do not apply. Batched +`read_datagrams` stays out until the +[planned lock profile](/quest/m2/perf/quic-lock-contention.md) shows a consumer +worth changing the API for. Dependency bumps are skipped. + +The implementation ports Apple's IPv4 source override in both UDP send paths, +and BBR3's startup ACK budget, recovery restoration, precautionary probing, +application-limited marker renewal, and saturating probe growth. Four BBR +regressions were reproduced before fixing them. The focused QUIC, socket, +and controller suite passed 808 tests; the benchmark covers recovery too. + +Remaining before completing this run: + +- Pass the full local `just ci`. Broad Cargo launches currently fail in `mbx` + with `Argument list too long`, including CI's own Clippy fixture; the + focused package commands work. Do not treat partial gate results as success. +- Run the Apple source-override regression on macOS at default features and + with `fast-apple-datapath`. This Linux host cannot reproduce or run that + platform-specific behavior. +- Obtain the required local other-vendor review and fix or refute every + finding. The prior batch exhausted Claude review quota; check the actual + result of the next attempt rather than assuming it is still unavailable. Public API: none expected; a ported fix that changes one says so. Wire: QUIC behavior follows upstream's fixes. diff --git a/rs/moq-congestion/src/bbr3/bench.rs b/rs/moq-congestion/src/bbr3/bench.rs index db58d731f40..e51a0cdee82 100644 --- a/rs/moq-congestion/src/bbr3/bench.rs +++ b/rs/moq-congestion/src/bbr3/bench.rs @@ -11,10 +11,18 @@ static ALLOCATOR: &StatsAlloc = &INSTRUMENTED_SYSTEM; const PACKET: u16 = 1200; +#[derive(Clone, Copy, Debug)] +enum Mode { + Drain, + Steady, + Recovery, +} + /// Times the controller callbacks for `flight` packets acknowledged `batch` at a time, refilled as /// they are acknowledged if `steady`. The timed region includes copying each packet's state into /// the ACK, as a transport does, and excludes the fixture's own bookkeeping. -fn measure(flight: usize, batch: usize, steady: bool) -> (Duration, usize) { +fn measure(flight: usize, batch: usize, mode: Mode) -> (Duration, usize) { + let steady = !matches!(mode, Mode::Drain); let base = Instant::now(); let region = Region::new(ALLOCATOR); let config = Bbr3Config { @@ -37,6 +45,14 @@ fn measure(flight: usize, batch: usize, steady: bool) -> (Duration, usize) { in_flight: (pn as u64 + 1) * bytes, })); } + // End a persistent-congestion episode with this flight's ACKs. The loss itself happened + // before the flight was sent, so its first ACK exercises window restoration. + if matches!(mode, Mode::Recovery) { + bbr.prior_cwnd = bbr.cwnd; + bbr.recovery_start_time = Some(base - Duration::from_millis(1)); + bbr.in_recovery = true; + bbr.cwnd = bbr.min_pipe_cwnd; + } let mut acked = Vec::with_capacity(batch); let retained = || { let stats = region.change(); @@ -94,19 +110,19 @@ fn benchmark_per_packet() { size_of::(), size_of::() ); - for steady in [false, true] { + for mode in [Mode::Drain, Mode::Steady, Mode::Recovery] { for flight in [16, 64, 1000, 4000, 16000] { for batch in [1, 8, 32] { let mut samples = [Duration::ZERO; 3]; let mut bytes = 0; for sample in &mut samples { - let (elapsed, retained) = measure(flight, batch, steady); + let (elapsed, retained) = measure(flight, batch, mode); *sample = elapsed; bytes = bytes.max(retained); } samples.sort(); println!( - "bbr_packet steady={steady} flight={flight} batch={batch} cpu_ns={} peak_bytes={bytes}", + "bbr_packet mode={mode:?} flight={flight} batch={batch} cpu_ns={} peak_bytes={bytes}", samples[1].as_nanos() ); } diff --git a/rs/moq-congestion/src/bbr3/mod.rs b/rs/moq-congestion/src/bbr3/mod.rs index 59a62cf4679..0c482bb00ae 100644 --- a/rs/moq-congestion/src/bbr3/mod.rs +++ b/rs/moq-congestion/src/bbr3/mod.rs @@ -467,6 +467,11 @@ pub struct Bbr3 { ack_phase: AckPhase, /// equivalent to BBR.is_bw_probe_sample bw_probe_samples: bool, + /// Whether the previous probe found excessive inflight, and whether this probe stopped at + /// that bound to wait for its feedback before probing higher. + prev_probe_too_high: bool, + undo_prev_probe_too_high: bool, + prev_probe_precautionary: bool, /// equivalent to BBR.loss_round_delivered: C.delivered during the first loss of the round loss_round_delivered: u64, /// equivalent to BBR.loss_in_round: flag set to true when loss occurs during the round @@ -657,6 +662,9 @@ impl Bbr3 { cycle_stamp: None, ack_phase: AckPhase::Init, bw_probe_samples: false, + prev_probe_too_high: false, + undo_prev_probe_too_high: false, + prev_probe_precautionary: false, loss_events_in_round: 0, recovery_start_time: None, recovery_start_round: 0, @@ -841,8 +849,10 @@ impl Bbr3 { if self.full_bw_reached { self.extra_acked_filter.update_max(self.round_count, extra); self.extra_acked = self.extra_acked_filter.get_max(); + } else if self.round_start { + self.extra_acked = extra; } else { - self.extra_acked = extra; // In startup, just remember 1 round + self.extra_acked = Ord::max(self.extra_acked, extra); } } @@ -934,7 +944,8 @@ impl Bbr3 { /// Fast recovery ends when a packet sent after it began is acknowledged. /// fn check_recovery_done(&mut self, sent: Instant) { - if self.recovery_start_time.is_some_and(|start| sent > start) { + if self.in_recovery && self.recovery_start_time.is_some_and(|start| sent > start) { + self.restore_cwnd(); self.in_recovery = false; } } @@ -971,7 +982,9 @@ impl Bbr3 { if !self.full_bw_reached { return; } - self.adapt_long_term_model(newly_acked, now); + if self.adapt_long_term_model(newly_acked, now) { + return; + } let state = self.state; match state { BbrState::ProbeBw(ProbeBwSubstate::Down) => { @@ -988,7 +1001,8 @@ impl Bbr3 { self.bw_probe_samples = true; self.start_probe_bw_up(); } - BbrState::ProbeBw(ProbeBwSubstate::Up) if self.maybe_go_down() => { + BbrState::ProbeBw(ProbeBwSubstate::Up) if self.maybe_go_down(newly_acked) => { + self.prev_probe_too_high = false; self.start_probe_bw_down(now); } BbrState::ProbeBw(ProbeBwSubstate::Up) => self.maybe_fill_probe_bw_up(now), @@ -1015,8 +1029,9 @@ impl Bbr3 { /// equivalent to BBRAdaptLongTermModel /// /// The probe's feedback ends once, on the first round after it stops, so later losses are not - /// taken as probe feedback and later rounds do not age the max-bw window again. - fn adapt_long_term_model(&mut self, newly_acked: u64, now: Instant) { + /// taken as probe feedback and later rounds do not age the max-bw window again. Returns + /// whether clean feedback from a precautionary probe started a new refill. + fn adapt_long_term_model(&mut self, newly_acked: u64, now: Instant) -> bool { if self.ack_phase == AckPhase::ProbeStarting && self.round_start { self.ack_phase = AckPhase::ProbeFeedback; } @@ -1029,10 +1044,17 @@ impl Bbr3 { { self.advance_max_bw_filter(now); } + if let BbrState::ProbeBw(_) = self.state + && self.prev_probe_precautionary + && !self.prev_probe_too_high + { + self.start_probe_bw_refill(); + return true; + } } if !self.is_inflight_too_high() { if self.inflight_longterm == u64::MAX { - return; + return false; } if let Some(rate_sample) = self.rs && rate_sample.tx_in_flight > self.inflight_longterm @@ -1043,6 +1065,7 @@ impl Bbr3 { self.probe_inflight_long_term_upward(newly_acked); } } + false } /// equivalent to BBRAdvanceMaxBwFilter @@ -1102,6 +1125,7 @@ impl Bbr3 { self.reset_short_term_model(); self.bw_probe_up_rounds = 0; self.bw_probe_up_acked = 0; + self.prev_probe_precautionary = false; self.ack_phase = AckPhase::Refilling; self.start_round(); self.cwnd_gain = self.default_cwnd_gain; @@ -1131,7 +1155,12 @@ impl Bbr3 { } /// equivalent to BBRIsTimeToGoDown - fn maybe_go_down(&mut self) -> bool { + fn maybe_go_down(&mut self, newly_acked: u64) -> bool { + let pre_ack_inflight = self.inflight.saturating_add(newly_acked); + if self.prev_probe_too_high && pre_ack_inflight >= self.inflight_longterm { + self.prev_probe_precautionary = true; + return true; + } if self.is_cwnd_limited && self.cwnd >= self.inflight_longterm { self.reset_full_bw(); if let Some(rate_sample) = self.rs { @@ -1153,12 +1182,14 @@ impl Bbr3 { if !self.is_cwnd_limited || self.cwnd < self.inflight_longterm { return; } - self.bw_probe_up_acked += newly_acked; + self.bw_probe_up_acked = self.bw_probe_up_acked.saturating_add(newly_acked); if self.bw_probe_up_acked >= self.probe_up_acked_per_inc && self.probe_up_acked_per_inc > 0 { let delta = self.bw_probe_up_acked / self.probe_up_acked_per_inc; self.bw_probe_up_acked -= delta * self.probe_up_acked_per_inc; - self.inflight_longterm += delta * self.smss; + self.inflight_longterm = self + .inflight_longterm + .saturating_add(delta.saturating_mul(self.smss)); } if self.round_start { self.raise_inflight_long_term_slope(); @@ -1286,13 +1317,14 @@ impl Bbr3 { /// equivalent to BBREnterProbeRTT fn enter_probe_rtt(&mut self) { self.state = BbrState::ProbeRtt; + self.prev_probe_precautionary = false; self.pacing_gain = self.default_pacing_gain; self.cwnd_gain = self.probe_rtt_cwnd_gain; } /// equivalent to BBRSaveCwnd fn save_cwnd(&mut self) { - if !self.loss_in_round && self.state != BbrState::ProbeRtt { + if !self.in_recovery && self.state != BbrState::ProbeRtt { self.prior_cwnd = self.cwnd; } else { self.prior_cwnd = Ord::max(self.prior_cwnd, self.cwnd); @@ -1652,6 +1684,7 @@ impl Bbr3 { _ => self.cwnd, }; self.undo_state = None; + self.undo_prev_probe_too_high = self.prev_probe_too_high; self.undo_bw_shortterm = self.bw_shortterm; self.undo_inflight_shortterm = self.inflight_shortterm; self.undo_inflight_longterm = self.inflight_longterm; @@ -1680,6 +1713,8 @@ impl Bbr3 { /// `rs` is the sample that showed it, if any. fn handle_inflight_too_high(&mut self, now: Instant, rs: Option) { self.bw_probe_samples = false; + self.prev_probe_too_high = true; + self.prev_probe_precautionary = false; if let Some(rate_sample) = rs && !rate_sample.is_app_limited { @@ -1868,6 +1903,8 @@ impl Controller for Bbr3 { self.bound_cwnd_for_probe_rtt(); self.loss_in_round = false; self.reset_full_bw(); + self.prev_probe_too_high = self.undo_prev_probe_too_high; + self.prev_probe_precautionary = false; self.bw_shortterm = [self.bw_shortterm, self.undo_bw_shortterm] .iter() .copied() @@ -4225,6 +4262,7 @@ mod test { // Peak max_bw seen in STARTUP; must stay ~BW, proving the burst didn't inflate the // delivery-rate estimate above the send rate. let mut peak_startup_max_bw: f64 = 0.0; + let mut last_startup_extra: Option<(u64, u64)> = None; // Captured on the STARTUP -> DRAIN edge, as in A.1. let mut transition: Option<(bool, f64)> = None; @@ -4279,6 +4317,15 @@ mod test { if bbr.state == BbrState::Startup { max_extra_acked = max_extra_acked.max(bbr.extra_acked); + if let Some((round, previous)) = last_startup_extra + && bbr.round_count == round + { + assert!( + bbr.extra_acked >= previous, + "startup lost its round's ACK aggregation budget" + ); + } + last_startup_extra = Some((bbr.round_count, bbr.extra_acked)); peak_startup_max_bw = peak_startup_max_bw.max(bbr.max_bw); // bbr.bdp is refreshed to max_bw*min_rtt on every set_cwnd; once the // aggregation estimate is positive it should lift cwnd past 2*BDP. @@ -8061,4 +8108,73 @@ mod test { assert!(marked.bbr.max_bw > 0.9 * control.bbr.max_bw); assert_eq!(marked.bbr.inflight_shortterm, u64::MAX); } + + /// Each idle ACK can pass the old application-limited marker while reporting that the + /// sender is still idle. The next keepalive must keep its application-limited label. + #[test] + fn idle_keepalives_remain_application_limited() { + let mut sim = scripted(); + sim.bbr.on_app_limited(0); + for i in 0..10 { + let now = i * 1000 * MS; + let pn = sim.pn; + sim.send(now, 1); + sim.ack(now + 10 * MS, [pn], true); + assert!( + sim.bbr.rs.unwrap().is_app_limited, + "keepalive {i} lost its label" + ); + } + } + + #[test] + fn recovery_exit_restores_the_saved_window() { + let mut sim = probing_up(); + sim.lose(15 * MS, 10); + assert_eq!(sim.bbr.prior_cwnd, 20_000); + let now = sim.at(16 * MS); + sim.bbr.on_loss(Loss { + now, + packets: &[], + in_flight: sim.inflight, + persistent: true, + }); + assert_eq!(sim.bbr.cwnd, sim.bbr.min_pipe_cwnd); + sim.bbr.inflight_longterm = 100_000; + sim.bbr.inflight_shortterm = 100_000; + sim.send(16 * MS, 1); + sim.ack(26 * MS, [sim.pn - 1], false); + assert!(!sim.bbr.in_recovery); + assert!( + sim.bbr.cwnd >= 20_000, + "recovery lost its saved window: {}", + sim.bbr.cwnd + ); + } + + #[test] + fn precautionary_probe_restarts_on_clean_feedback() { + let mut sim = probing_up(); + sim.lose(15 * MS, 10); + assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Down)); + let risky_inflight = sim.bbr.inflight_longterm; + sim.ack(20 * MS, 11..20, false); + let wait_ns = sim.bbr.bw_probe_wait.as_nanos() as u64 + 30 * MS; + sim.bbr.probe_rtt_min_stamp = Some(sim.at(wait_ns)); + sim.round(wait_ns, 5, 10 * MS); + assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Refill)); + sim.round(wait_ns + 10 * MS, 5, 10 * MS); + assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Up)); + let t_probe = wait_ns + 20 * MS; + let first = sim.pn; + sim.send(t_probe, risky_inflight.div_ceil(sim.mss)); + sim.ack(t_probe + 10 * MS, first..first + 1, false); + assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Down)); + assert_eq!(sim.bbr.inflight_longterm, risky_inflight); + sim.ack(t_probe + 12 * MS, first + 1..sim.pn, false); + sim.round(t_probe + 15 * MS, 5, 10 * MS); + assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Refill)); + sim.round(t_probe + 25 * MS, 5, 10 * MS); + assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Up)); + } } diff --git a/rs/moq-congestion/src/rate.rs b/rs/moq-congestion/src/rate.rs index 2a7285bf625..5a97b2d1b39 100644 --- a/rs/moq-congestion/src/rate.rs +++ b/rs/moq-congestion/src/rate.rs @@ -135,7 +135,8 @@ impl Sampler { if self.app_limited != 0 && self.delivered > self.app_limited { self.app_limited = 0; - } else if ack.app_limited { + } + if ack.app_limited { self.mark_app_limited(ack.in_flight); } diff --git a/rs/moq-quic/README.md b/rs/moq-quic/README.md index 677785619ac..273cc56ecc0 100644 --- a/rs/moq-quic/README.md +++ b/rs/moq-quic/README.md @@ -9,7 +9,8 @@ All credit for the code goes to the quinn developers; it stays under quinn's MIT ## Upstream Forked from quinn-proto in quinn-rs/quinn `main` at [`7616e6b2`](https://github.com/quinn-rs/quinn/commit/7616e6b2782722f3ce4a1b181ef08817d7f545e4) (2026-10-05). -That covers every quinn security advisory published through 2026-09-30. +Checked through [`2c575013`](https://github.com/quinn-rs/quinn/commit/2c5750139fa09aab588bf0f939c83f521d658baf) on `main` and [`4b84837d`](https://github.com/quinn-rs/quinn/commit/4b84837df1d25424bc37b0ed990e7b376d40a9c0) on `0.11.x` (2026-10-31). +All 13 published [security advisories](https://github.com/quinn-rs/quinn/security/advisories) were checked; the newest are still the 2026-09-30 fixes included at the fork point. Ported since the fork point: @@ -20,6 +21,15 @@ Ported since the fork point: - [`f74e6a8a`](https://github.com/quinn-rs/quinn/commit/f74e6a8a15e5aa980044b7fa03a31878c6a8c65c): check a DATAGRAM frame's encoded size against `max_datagram_frame_size`, and drop rather than refuse one whose queue metadata overflows the receive buffer, so a datagram of the advertised maximum size no longer closes the connection. - [`17596dde`](https://github.com/quinn-rs/quinn/commit/17596ddefb09dae895129520824055d0ac8da5d5) (`moq_sock::udp`): on Windows, query `IPV6_V6ONLY` only on IPv6 sockets, so Winsock refusing the option on an IPv4 socket cannot fail endpoint creation. - [`d3174f1e`](https://github.com/quinn-rs/quinn/commit/d3174f1e77e093ceadfe0edff865fdd92d51a03c): a `wasm32-unknown-unknown` build pulls in ring only through the `ring` feature. +- [`f233510c`](https://github.com/quinn-rs/quinn/commit/f233510cf37e1be9511070bf743151736d621a5e): already covered by the separate handshake timeout. Accepting an Initial never consults the connection idle timeout, disabled or otherwise. +- [`6c69f9f4`](https://github.com/quinn-rs/quinn/commit/6c69f9f4ed2362002a0e2e5312a6dfa3ceb31c1d): already covered by `Recv::unread_credit` and `stop_after_reset_credits_once`, including RESET_STREAM_AT and leased credit. +- [`623a0369`](https://github.com/quinn-rs/quinn/commit/623a0369536b9a1e731a8629c80c164eb8f2e637) (`moq_sock::bind`): already covered by `make_dual_stack`, which makes specific IPv6 addresses v6-only; `udp_ipv6_loopback_is_v6_only` checks it. +- [`a2c55b36`](https://github.com/quinn-rs/quinn/commit/a2c55b36817ccaac1c39c6db92de8294f281819f) (`moq_sock::udp`): send IPv4 source overrides with Apple's `IP_PKTINFO` in both `sendmsg` and `sendmsg_x`; `IP_RECVDSTADDR` only selects received destination metadata there. +- [`2c575013`](https://github.com/quinn-rs/quinn/commit/2c5750139fa09aab588bf0f939c83f521d658baf): already covered by the `MIN_PACKET_SPACE` check when choosing a GSO segment size, carried with bandwidth probing. +- [`c2a47f6d`](https://github.com/quinn-rs/quinn/commit/c2a47f6d06447a997451568b90d5495c7e7bc3a9) (`0.11.x`, `moq_tokio::quic::driver`): already covered by `poll_recv`, which propagates every error except `WouldBlock`. The fork has no async-io runtime. + +Dependency bumps are not ported. The `0.11.x` accept refactors are already covered by this fork's split acceptance and CID routing. +[`2697de1e`](https://github.com/quinn-rs/quinn/commit/2697de1e1314cb9f98502b7323acd5bdd836b582)'s batched datagram API is not carried: the current WebTransport reader consumes one at a time, and the [planned driver lock profile](../../quest/m2/perf/quic-lock-contention.md) measures contention before adding batching. The fork never merges upstream, and takes the workspace's formatting and names, so upstream patches no longer apply. Upstream fixes are ported by hand, together with their regression tests. @@ -40,6 +50,7 @@ Changes on top of the upstream commit, besides the renames: - **Close events**: a local close reports `connection::Error::LocallyClosed` with its code and reason as the connection's `ConnectionLost`, a handshake that idles out reports `HandshakeTimeout`, and close reasons are strings, lossily decoded. - **No early data**: TLS never offers or accepts 0-RTT, and early packets are discarded without decryption. The early keys, remembered transport parameters, stream rollback, and early-session driver APIs are removed. Session tickets still resume 1-RTT handshakes. MoQ's CONNECT and authorization already wait for the handshake. - **BBR3**, from [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481) at [`55f74c0d`](https://github.com/quinn-rs/quinn/pull/2481/commits/55f74c0dd0738b5ad7d7676c0b2869a547783f83) (open upstream), rebased onto the fork point with authorship kept. It replaces quinn's BBR and is the default controller. It carries the BBR correctness fixes moq-dev/noq shipped in `moq-noq` 1.3.1, and with them the `Controller` changes BBR needs: per-packet send and loss callbacks, `on_cwnd_limited`, `on_app_limited`, `on_ack_frequency_update`, and controller-owned pacing and burst size. `ControllerMetrics` rates are bytes per second. +- **BBR3 upstream refresh**: Checked against the force-pushed [`01b8848d`](https://github.com/quinn-rs/quinn/pull/2481/commits/01b8848dcb321eb17ad4d88dd6f84f28f45a0d6a) on 2026-10-31: carry startup's per-round ACK aggregation maximum, restore the saved window once when recovery ends, preserve it through recovery into ProbeRTT, saturate probe-bound growth, and stop a probe at the previous loss bound before accelerating again on clean feedback. Spurious-loss undo also restores that probe history, and an idle ACK renews application-limited marking even when it crosses the old marker. The two-cycle bandwidth filter and transport-owned packet snapshots already cover its other fixes. Upstream's bandwidth metric conversion to bits/s does not apply: this fork's controller metrics use bytes/s. - **BBR3 loss samples**: a lost packet supplies its own inflight, loss, and application-limited state without borrowing or overwriting the ACK sample, including before the first ACK observation. This follows draft-06 and Google Linux BBRv3; upstream submission waits on quinn#2481 for the same callback adaptation as packet retirement. - **BBR3 packet retirement**: per-space packet lookup and round-indexed expiry retire completed snapshots incrementally, preserving the extra ACK batch of metadata for late loss callbacks. The bounded indexes handle sparse packet numbers without walking the live flight. Upstream submission is deferred while quinn#2481 remains open: this fork also carries the packet-space and loss callback adaptations, so a port needs to target that implementation after its shape settles. - **Classic ECN for BBR3**, ported from [moq-dev/noq#12](https://github.com/moq-dev/noq/pull/12): CE exits Startup, stops a bandwidth probe, or lowers the short-term model, once per recovery episode, instead of counting as a loss. diff --git a/rs/moq-sock/Cargo.toml b/rs/moq-sock/Cargo.toml index 839c1fb9022..7485a1a1d02 100644 --- a/rs/moq-sock/Cargo.toml +++ b/rs/moq-sock/Cargo.toml @@ -33,5 +33,8 @@ cfg_aliases = "0.2" rand = "0.10" tracing-test = "0.2" +[target.'cfg(target_vendor = "apple")'.dev-dependencies] +nix = { workspace = true } + [lints] workspace = true diff --git a/rs/moq-sock/src/udp/apple_fast.rs b/rs/moq-sock/src/udp/apple_fast.rs index 1ec62be6e96..33af7905129 100644 --- a/rs/moq-sock/src/udp/apple_fast.rs +++ b/rs/moq-sock/src/udp/apple_fast.rs @@ -48,7 +48,6 @@ fn send_via_sendmsg_x(state: &UdpSocketState, io: SockRef<'_>, transmit: &Transm &mut hdrs[i], &mut iovs[i], &mut ctrls[i], - true, state.sendmsg_einval(), ); hdrs[i].msg_datalen = chunk.len(); @@ -69,7 +68,6 @@ fn prepare_msg_x( hdr: &mut msghdr_x, iov: &mut libc::iovec, ctrl: &mut cmsg::Aligned<[u8; cmsg::LEN]>, - #[allow(unused_variables, reason = "kept for parity with the unix `prepare_msg` signature")] encode_src_ip: bool, sendmsg_einval: bool, ) { iov.iov_base = transmit.contents.as_ptr() as *const _ as *mut _; @@ -99,12 +97,14 @@ fn prepare_msg_x( if let Some(ip) = &transmit.src_ip { match ip { IpAddr::V4(v4) => { - if encode_src_ip { - let addr = libc::in_addr { + let pktinfo = libc::in_pktinfo { + ipi_ifindex: 0, + ipi_spec_dst: libc::in_addr { s_addr: u32::from_ne_bytes(v4.octets()), - }; - encoder.push(libc::IPPROTO_IP, libc::IP_RECVDSTADDR, addr); - } + }, + ipi_addr: libc::in_addr { s_addr: 0 }, + }; + encoder.push(libc::IPPROTO_IP, libc::IP_PKTINFO, pktinfo); } IpAddr::V6(v6) => { let pktinfo = libc::in6_pktinfo { diff --git a/rs/moq-sock/src/udp/unix.rs b/rs/moq-sock/src/udp/unix.rs index 5338801d32f..874f551b1e0 100644 --- a/rs/moq-sock/src/udp/unix.rs +++ b/rs/moq-sock/src/udp/unix.rs @@ -637,7 +637,7 @@ fn prepare_msg( hdr: &mut libc::msghdr, iov: &mut libc::iovec, ctrl: &mut cmsg::Aligned<[u8; cmsg::LEN]>, - #[allow(unused_variables, reason = "only used on FreeBSD & macOS")] encode_src_ip: bool, + #[allow(unused_variables, reason = "only used on BSD & Solaris")] encode_src_ip: bool, sendmsg_einval: bool, ) { iov.iov_base = transmit.contents.as_ptr() as *const _ as *mut _; @@ -685,7 +685,7 @@ fn prepare_msg( if let Some(ip) = &transmit.src_ip { match ip { IpAddr::V4(v4) => { - #[cfg(any(target_os = "linux", target_os = "android"))] + #[cfg(any(target_os = "linux", target_os = "android", apple))] { let pktinfo = libc::in_pktinfo { ipi_ifindex: 0, @@ -696,7 +696,7 @@ fn prepare_msg( }; encoder.push(libc::IPPROTO_IP, libc::IP_PKTINFO, pktinfo); } - #[cfg(any(bsd, apple, solarish))] + #[cfg(any(bsd, solarish))] { if encode_src_ip { let addr = libc::in_addr { diff --git a/rs/moq-sock/tests/udp/main.rs b/rs/moq-sock/tests/udp/main.rs index dd4081ab099..f1b81e8da05 100644 --- a/rs/moq-sock/tests/udp/main.rs +++ b/rs/moq-sock/tests/udp/main.rs @@ -74,6 +74,46 @@ fn basic_src_ip() { ); } +/// A wildcard sender must use the requested source even when the route prefers loopback. +#[test] +#[cfg(apple)] +fn ipv4_source_address_override() { + use nix::{ifaddrs::getifaddrs, net::if_::InterfaceFlags}; + + let src_ip = getifaddrs() + .unwrap() + .filter(|interface| interface.flags.contains(InterfaceFlags::IFF_UP)) + .filter_map(|interface| Some(interface.address?.as_sockaddr_in()?.ip())) + .find(|ip| !ip.is_loopback() && !ip.is_unspecified() && !ip.is_multicast() && !ip.is_broadcast()) + .expect("source override test needs an assigned non-loopback IPv4 address"); + let send = UdpSocket::bind((Ipv4Addr::UNSPECIFIED, 0)).unwrap(); + let recv = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).unwrap(); + recv.set_read_timeout(Some(Duration::from_secs(3))).unwrap(); + let state = UdpSocketState::new((&send).into()).unwrap(); + #[cfg(apple_fast)] + // SAFETY: the macOS test host supports sendmsg_x. + unsafe { + state.set_apple_fast_path() + }; + socket2::SockRef::from(&send).set_nonblocking(false).unwrap(); + state + .try_send( + (&send).into(), + &Transmit { + destination: recv.local_addr().unwrap(), + ecn: None, + contents: b"hello", + segment_size: None, + src_ip: Some(src_ip.into()), + }, + ) + .unwrap(); + let mut buf = [0; 32]; + let (received, source) = recv.recv_from(&mut buf).unwrap(); + assert_eq!(&buf[..received], b"hello"); + assert_eq!(source.ip(), src_ip); +} + // The WASI backend sets no ECN codepoint and always reports `None`. #[test] #[cfg(not(target_os = "wasi"))] From aa2a609178c6808fe486588e5b9fe4c30e33b92e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 31 Oct 2026 15:35:57 -0700 Subject: [PATCH 3/7] docs(quest): record vendor sync review blocker Co-Authored-By: GPT-6.1 Sol --- quest/m1/vendor-sync/quinn.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/quest/m1/vendor-sync/quinn.md b/quest/m1/vendor-sync/quinn.md index 37d6b35e2bd..58b66ee6479 100644 --- a/quest/m1/vendor-sync/quinn.md +++ b/quest/m1/vendor-sync/quinn.md @@ -43,13 +43,16 @@ Remaining before completing this run: - Pass the full local `just ci`. Broad Cargo launches currently fail in `mbx` with `Argument list too long`, including CI's own Clippy fixture; the - focused package commands work. Do not treat partial gate results as success. + focused package commands work. The captured Bash environment reload overrides + PATH-only fixes. A gate attempt using `BASH_ENV=/dev/null` and the Nix Cargo + path is testing the full selection. Do not treat partial results as success. - Run the Apple source-override regression on macOS at default features and with `fast-apple-datapath`. This Linux host cannot reproduce or run that platform-specific behavior. - Obtain the required local other-vendor review and fix or refute every - finding. The prior batch exhausted Claude review quota; check the actual - result of the next attempt rather than assuming it is still unavailable. + finding. `quest review` on `9d15d25a720ec0d0f1b761fdc896b0740c8c275e` + failed before review: Claude reported its weekly limit, resetting + 2026-11-02 at 02:00 America/Los_Angeles. No completed review is claimed. Public API: none expected; a ported fix that changes one says so. Wire: QUIC behavior follows upstream's fixes. From 55b600dc564e0ecba77df19f38ae59a416b650f6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 31 Oct 2026 15:51:11 -0700 Subject: [PATCH 4/7] docs(quest): record completed local validation Co-Authored-By: GPT-6.1 Sol --- quest/m1/vendor-sync/quinn.md | 15 +++++++++------ rs/moq-quic/README.md | 2 +- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/quest/m1/vendor-sync/quinn.md b/quest/m1/vendor-sync/quinn.md index 58b66ee6479..12733ace850 100644 --- a/quest/m1/vendor-sync/quinn.md +++ b/quest/m1/vendor-sync/quinn.md @@ -39,18 +39,21 @@ application-limited marker renewal, and saturating probe growth. Four BBR regressions were reproduced before fixing them. The focused QUIC, socket, and controller suite passed 808 tests; the benchmark covers recovery too. +The full local gate ran with the captured Bash environment reload disabled +(`BASH_ENV=/dev/null`) and the Nix Cargo path selected explicitly. All Rust +lint, workspace tests, isolated feature shapes, and CI self-tests passed. +That invocation exited 1 only for README formatting; formatting was fixed +and the Markdown check passed separately. The normal captured environment +still makes broad Cargo launches report `mbx: Argument list too long`. + Remaining before completing this run: -- Pass the full local `just ci`. Broad Cargo launches currently fail in `mbx` - with `Argument list too long`, including CI's own Clippy fixture; the - focused package commands work. The captured Bash environment reload overrides - PATH-only fixes. A gate attempt using `BASH_ENV=/dev/null` and the Nix Cargo - path is testing the full selection. Do not treat partial results as success. - Run the Apple source-override regression on macOS at default features and with `fast-apple-datapath`. This Linux host cannot reproduce or run that platform-specific behavior. - Obtain the required local other-vendor review and fix or refute every - finding. `quest review` on `9d15d25a720ec0d0f1b761fdc896b0740c8c275e` + finding. `quest review` on + [`9d15d25a`](https://github.com/moq-dev/moq/commit/9d15d25a720ec0d0f1b761fdc896b0740c8c275e) failed before review: Claude reported its weekly limit, resetting 2026-11-02 at 02:00 America/Los_Angeles. No completed review is claimed. diff --git a/rs/moq-quic/README.md b/rs/moq-quic/README.md index 273cc56ecc0..ad797b9bd07 100644 --- a/rs/moq-quic/README.md +++ b/rs/moq-quic/README.md @@ -22,7 +22,7 @@ Ported since the fork point: - [`17596dde`](https://github.com/quinn-rs/quinn/commit/17596ddefb09dae895129520824055d0ac8da5d5) (`moq_sock::udp`): on Windows, query `IPV6_V6ONLY` only on IPv6 sockets, so Winsock refusing the option on an IPv4 socket cannot fail endpoint creation. - [`d3174f1e`](https://github.com/quinn-rs/quinn/commit/d3174f1e77e093ceadfe0edff865fdd92d51a03c): a `wasm32-unknown-unknown` build pulls in ring only through the `ring` feature. - [`f233510c`](https://github.com/quinn-rs/quinn/commit/f233510cf37e1be9511070bf743151736d621a5e): already covered by the separate handshake timeout. Accepting an Initial never consults the connection idle timeout, disabled or otherwise. -- [`6c69f9f4`](https://github.com/quinn-rs/quinn/commit/6c69f9f4ed2362002a0e2e5312a6dfa3ceb31c1d): already covered by `Recv::unread_credit` and `stop_after_reset_credits_once`, including RESET_STREAM_AT and leased credit. +- [`6c69f9f4`](https://github.com/quinn-rs/quinn/commit/6c69f9f4ed2362002a0e2e5312a6dfa3ceb31c1d): already covered by `Recv::unread_credit` and `stop_after_reset_credits_once`, including RESET\_STREAM\_AT and leased credit. - [`623a0369`](https://github.com/quinn-rs/quinn/commit/623a0369536b9a1e731a8629c80c164eb8f2e637) (`moq_sock::bind`): already covered by `make_dual_stack`, which makes specific IPv6 addresses v6-only; `udp_ipv6_loopback_is_v6_only` checks it. - [`a2c55b36`](https://github.com/quinn-rs/quinn/commit/a2c55b36817ccaac1c39c6db92de8294f281819f) (`moq_sock::udp`): send IPv4 source overrides with Apple's `IP_PKTINFO` in both `sendmsg` and `sendmsg_x`; `IP_RECVDSTADDR` only selects received destination metadata there. - [`2c575013`](https://github.com/quinn-rs/quinn/commit/2c5750139fa09aab588bf0f939c83f521d658baf): already covered by the `MIN_PACKET_SPACE` check when choosing a GSO segment size, carried with bandwidth probing. From d7a215434d5ea08e0fb6ae957b6b75d695e4e943 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 31 Oct 2026 22:49:49 -0700 Subject: [PATCH 5/7] chore(quest): finish the quinn vendor sync run The Apple source-address fix landed on main in moq-dev/moq#7418, so the merge keeps main's moq-sock and the fork ledger credits it. With the BBR3 refresh ported and the ledger checked through 2c575013, this run is done. Co-Authored-By: Claude Opus 5.5 --- quest/m1/vendor-sync/README.md | 1 - quest/m1/vendor-sync/quinn.md | 56 ---------------------------------- 2 files changed, 57 deletions(-) delete mode 100644 quest/m1/vendor-sync/quinn.md diff --git a/quest/m1/vendor-sync/README.md b/quest/m1/vendor-sync/README.md index 263aba80890..342653185e3 100644 --- a/quest/m1/vendor-sync/README.md +++ b/quest/m1/vendor-sync/README.md @@ -58,7 +58,6 @@ behavior follows quinn's fixes. ## Required -- [moq-quic has every quinn fix it needs](/quest/m1/vendor-sync/quinn.md) - quinn's fixes since the last check are ported with their tests - [moq-v4l has every libv4l-rs fix it needs](/quest/m1/vendor-sync/v4l.md) - libv4l-rs fixes since v4l 0.14 are ported or the API they fix is gone - [UniFFI generator forks are current](/quest/m1/vendor-sync/uniffi.md) - the generator forks carry upstream's fixes until rust2 replaces them - [moq-vaapi has every cros-codecs fix it needs](/quest/m1/vendor-sync/vaapi.md) - cros-libva and cros-codecs fixes reach moq-dev/vaapi diff --git a/quest/m1/vendor-sync/quinn.md b/quest/m1/vendor-sync/quinn.md deleted file mode 100644 index 1d8a7ef890d..00000000000 --- a/quest/m1/vendor-sync/quinn.md +++ /dev/null @@ -1,56 +0,0 @@ -# [S] moq-quic has every quinn fix it needs - -## Goal - -moq-quic, moq-sock's `udp`, moq-tokio's `quic::driver`, and moq-congestion's -BBR3 carry every -security and correctness fix quinn has landed since the last sync, each -ported with its regression test or recorded as not applying. The fork -ledger in `rs/moq-quic/README.md` says how far upstream was checked. - -## Plan - -Recurring child of the [vendor sync](/quest/m1/vendor-sync/README.md), which -says how each run works and recurs. Decided 2026-10-27 (maintainer), from a -second audit of main against release: the hard fork never merges upstream, so -checking upstream is recurring work. On 2026-10-30 the maintainer moved it into -the vendor sync with moq's other vendored code. - -Each run also checks [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481), -the open BBR3 PR moq-congestion carries from -[`55f74c0d`](https://github.com/quinn-rs/quinn/pull/2481/commits/55f74c0dd0738b5ad7d7676c0b2869a547783f83), -for fixes to port, and records the PR commit checked in the README's BBR3 -entry. It was force-pushed to `01b8848d` on 2026-10-20. - -Last checked: quinn `main` through -[`2697de1e`](https://github.com/quinn-rs/quinn/commit/2697de1e1314cb9f98502b7323acd5bdd836b582) -(2026-10-28) and `0.11.x` through -[`4b84837d`](https://github.com/quinn-rs/quinn/commit/4b84837df1d25424bc37b0ed990e7b376d40a9c0), -recorded in `rs/moq-quic/README.md`'s `Upstream` section -([moq-dev/moq#7287](https://github.com/moq-dev/moq/pull/7287) checked -`f233510c`, `6c69f9f4`, and `623a0369`). Record the newest upstream commit -checked there. - -Remaining this run: -[`a2c55b36`](https://github.com/quinn-rs/quinn/commit/a2c55b36817ccaac1c39c6db92de8294f281819f) -(honor IPv4 source addresses on Apple: send `IP_PKTINFO` rather than -`IP_RECVDSTADDR`, in `moq_sock::udp`), the quinn#2481 check above, and judging -the new `read_datagrams` API -([`2697de1e`](https://github.com/quinn-rs/quinn/commit/2697de1e)), a feature: -plan it with `/quest-plan` if moq-tokio's driver would use it. - -Learned so far: a fix missing from the ledger is often already held, since the -fork reworked the same code (`stop` under `RESET_STREAM_AT`, the stale-accept -check under `handshake_timeout`, and socket setup in `moq_sock::bind`), so -check the fork's code before porting. A fix to quinn's `Endpoint::client` or -`server` lands in `moq_sock::bind`, which the driver's test helpers bind -through. Most of `0.11.x` is backports of `main`; compare its commits by -subject. - -Public API: none expected; a ported fix that changes one says so. Wire: -QUIC behavior follows upstream's fixes. - -## Related - -- [moq-quic's API improves on quinn's](/quest/m1/quic-api/README.md) - renames that make each port a hand translation -- [Dependency audit](/quest/m2/audit-deps.md) - covers crates cargo-deny can see From 26263090db40b78e6617236af83dabdb7209e3f5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 1 Nov 2026 21:24:03 -0800 Subject: [PATCH 6/7] fix(congestion): keep the precautionary refill across a spurious loss Snapshot the precautionary-stop flag with the other loss-undo state and restore it while its feedback round is still running, so a spurious loss in that round no longer delays the next probe by a full bw_probe_wait. Clear the flag once the feedback round ends, so it never outlives the probe it marks. Drop the bench's Recovery mode: only its first ACK differed from Steady. Move the deleted quinn quest's sync hints into the moq-quic README, so the next run finds them. Co-Authored-By: Claude Opus 5.5 --- rs/moq-congestion/src/bbr3/bench.rs | 24 +++------------ rs/moq-congestion/src/bbr3/mod.rs | 47 +++++++++++++++++++++++++++-- rs/moq-quic/README.md | 4 +++ 3 files changed, 53 insertions(+), 22 deletions(-) diff --git a/rs/moq-congestion/src/bbr3/bench.rs b/rs/moq-congestion/src/bbr3/bench.rs index 60eec0b5562..6cc91f6b01c 100644 --- a/rs/moq-congestion/src/bbr3/bench.rs +++ b/rs/moq-congestion/src/bbr3/bench.rs @@ -11,18 +11,10 @@ static ALLOCATOR: &StatsAlloc = &INSTRUMENTED_SYSTEM; const PACKET: u16 = 1200; -#[derive(Clone, Copy, Debug)] -enum Mode { - Drain, - Steady, - Recovery, -} - /// Times the controller callbacks for `flight` packets acknowledged `batch` at a time, refilled as /// they are acknowledged if `steady`. The timed region includes copying each packet's state into /// the ACK, as a transport does, and excludes the fixture's own bookkeeping. -fn measure(flight: usize, batch: usize, mode: Mode) -> (Duration, usize) { - let steady = !matches!(mode, Mode::Drain); +fn measure(flight: usize, batch: usize, steady: bool) -> (Duration, usize) { let base = Instant::now(); let region = Region::new(ALLOCATOR); let config = Config { @@ -45,14 +37,6 @@ fn measure(flight: usize, batch: usize, mode: Mode) -> (Duration, usize) { in_flight: (pn as u64 + 1) * bytes, })); } - // End a persistent-congestion episode with this flight's ACKs. The loss itself happened - // before the flight was sent, so its first ACK exercises window restoration. - if matches!(mode, Mode::Recovery) { - bbr.prior_cwnd = bbr.cwnd; - bbr.recovery_start_time = Some(base - Duration::from_millis(1)); - bbr.in_recovery = true; - bbr.cwnd = bbr.min_pipe_cwnd; - } let mut acked = Vec::with_capacity(batch); let retained = || { let stats = region.change(); @@ -111,19 +95,19 @@ fn benchmark_per_packet() { size_of::(), size_of::() ); - for mode in [Mode::Drain, Mode::Steady, Mode::Recovery] { + for steady in [false, true] { for flight in [16, 64, 1000, 4000, 16000] { for batch in [1, 8, 32] { let mut samples = [Duration::ZERO; 3]; let mut bytes = 0; for sample in &mut samples { - let (elapsed, retained) = measure(flight, batch, mode); + let (elapsed, retained) = measure(flight, batch, steady); *sample = elapsed; bytes = bytes.max(retained); } samples.sort(); println!( - "bbr_packet mode={mode:?} flight={flight} batch={batch} cpu_ns={} peak_bytes={bytes}", + "bbr_packet steady={steady} flight={flight} batch={batch} cpu_ns={} peak_bytes={bytes}", samples[1].as_nanos() ); } diff --git a/rs/moq-congestion/src/bbr3/mod.rs b/rs/moq-congestion/src/bbr3/mod.rs index 2e8458b88fb..d6e9a36ce3c 100644 --- a/rs/moq-congestion/src/bbr3/mod.rs +++ b/rs/moq-congestion/src/bbr3/mod.rs @@ -473,6 +473,7 @@ pub struct Bbr3 { prev_probe_too_high: bool, undo_prev_probe_too_high: bool, prev_probe_precautionary: bool, + undo_prev_probe_precautionary: bool, /// equivalent to BBR.loss_round_delivered: C.delivered during the first loss of the round loss_round_delivered: u64, /// equivalent to BBR.loss_in_round: flag set to true when loss occurs during the round @@ -655,6 +656,7 @@ impl Bbr3 { prev_probe_too_high: false, undo_prev_probe_too_high: false, prev_probe_precautionary: false, + undo_prev_probe_precautionary: false, loss_events_in_round: 0, recovery_start_time: None, recovery_start_round: 0, @@ -1036,8 +1038,10 @@ impl Bbr3 { { self.advance_max_bw_filter(now); } + // A precautionary stop waits for this one round of feedback, and no later. + let precautionary = std::mem::take(&mut self.prev_probe_precautionary); if let BbrState::ProbeBw(_) = self.state - && self.prev_probe_precautionary + && precautionary && !self.prev_probe_too_high { self.start_probe_bw_refill(); @@ -1677,6 +1681,7 @@ impl Bbr3 { }; self.undo_state = None; self.undo_prev_probe_too_high = self.prev_probe_too_high; + self.undo_prev_probe_precautionary = self.prev_probe_precautionary; self.undo_bw_shortterm = self.bw_shortterm; self.undo_inflight_shortterm = self.inflight_shortterm; self.undo_inflight_longterm = self.inflight_longterm; @@ -1917,7 +1922,11 @@ impl Controller for Bbr3 { self.loss_in_round = false; self.reset_full_bw(); self.prev_probe_too_high = self.undo_prev_probe_too_high; - self.prev_probe_precautionary = false; + // A precautionary stop still waiting for its feedback round waits again; once that + // round has passed, the spurious loss cost the early refill for good. + self.prev_probe_precautionary = self.undo_prev_probe_precautionary + && self.state == BbrState::ProbeBw(ProbeBwSubstate::Down) + && self.ack_phase == AckPhase::ProbeStopping; self.bw_shortterm = [self.bw_shortterm, self.undo_bw_shortterm] .iter() .copied() @@ -8249,4 +8258,38 @@ mod test { sim.round(t_probe + 25 * MS, 5, 10 * MS); assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Up)); } + + /// A loss in a precautionary stop's feedback round, later declared spurious, still lets the + /// clean feedback start the next refill early. + #[test] + fn spurious_loss_keeps_the_precautionary_refill() { + let mut sim = probing_up(); + sim.lose(15 * MS, 10); + let risky_inflight = sim.bbr.inflight_longterm; + sim.ack(20 * MS, 11..20, false); + let wait_ns = sim.bbr.bw_probe_wait.as_nanos() as u64 + 30 * MS; + sim.bbr.probe_rtt_min_stamp = Some(sim.at(wait_ns)); + sim.round(wait_ns, 5, 10 * MS); + sim.round(wait_ns + 10 * MS, 5, 10 * MS); + assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Up)); + let t_probe = wait_ns + 20 * MS; + let first = sim.pn; + sim.send(t_probe, risky_inflight.div_ceil(sim.mss)); + sim.ack(t_probe + 10 * MS, first..first + 1, false); + assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Down)); + assert!(sim.bbr.prev_probe_precautionary); + + sim.lose(t_probe + 11 * MS, first + 1); + assert!( + !sim.bbr.prev_probe_precautionary, + "the loss reads as too high" + ); + sim.bbr.on_spurious_loss(); + assert!(sim.bbr.prev_probe_precautionary); + assert!(!sim.bbr.prev_probe_too_high); + + sim.ack(t_probe + 12 * MS, first + 2..sim.pn, false); + sim.round(t_probe + 15 * MS, 5, 10 * MS); + assert_eq!(sim.bbr.state, BbrState::ProbeBw(ProbeBwSubstate::Refill)); + } } diff --git a/rs/moq-quic/README.md b/rs/moq-quic/README.md index 24ad4cb1c61..44a96d36842 100644 --- a/rs/moq-quic/README.md +++ b/rs/moq-quic/README.md @@ -34,6 +34,10 @@ Checked through quinn `main` at [`2c575013`](https://github.com/quinn-rs/quinn/c The fork never merges upstream, and takes the workspace's formatting and names, so upstream patches no longer apply. Upstream fixes are ported by hand, together with their regression tests. +A fix missing from this list is often already held, because the fork reworked the same code (`stop` under `RESET_STREAM_AT`, the stale-accept check under `handshake_timeout`, socket setup in `moq_sock::bind`), so check the fork's code before porting. +A fix to quinn's `Endpoint::client` or `server` lands in `moq_sock::bind`, which the driver's test helpers bind through. +Most of `0.11.x` backports `main`, so compare its commits by subject. +Each sync also checks [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481), the BBR3 PR moq-congestion carries, for fixes to port, and records the PR commit checked in the BBR3 entries below. quinn-udp, from the same commit, is the `udp` module of [moq-sock](../moq-sock/README.md), with its tests in `tests/udp`. Its benchmark is not carried (moq-uring's `udp_tokio` covers it), and the module logs through `tracing` only, so quinn-udp's `log` feature and no-op logger are dropped. From af9218d6e1449338b0634b0c5b849ce8ffe6f348 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 1 Nov 2026 22:03:10 -0800 Subject: [PATCH 7/7] test(congestion): an idle ACK's renewed marker labels the next round Since an idle ACK renews the application-limited marker, the round sent right after it is application-limited, as draft-06 labels every packet sent while C.app_limited is set. Startup's probe fill therefore ends one round later, on the sender's next round of its own. Co-Authored-By: Claude Opus 5.5 --- rs/moq-congestion/src/bbr3/mod.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rs/moq-congestion/src/bbr3/mod.rs b/rs/moq-congestion/src/bbr3/mod.rs index d6e9a36ce3c..337eb210698 100644 --- a/rs/moq-congestion/src/bbr3/mod.rs +++ b/rs/moq-congestion/src/bbr3/mod.rs @@ -7310,7 +7310,11 @@ mod test { } limited(&mut sim, MAX_FULL_BW_COUNT); assert!(sim.bbr.probe_inflight().is_some()); + // The idle ACK renewed the application-limited marker, so the round sent right after it + // still counts as application-limited; the sender's next round is its own. sim.round((MAX_FULL_BW_COUNT + 1) * 20 * MS, 100, 10 * MS); + assert!(sim.bbr.probe_inflight().is_some(), "sent under the marker"); + sim.round((MAX_FULL_BW_COUNT + 2) * 20 * MS, 100, 10 * MS); assert_eq!(sim.bbr.state, BbrState::Startup); assert_eq!( sim.bbr.probe_inflight(),