Repository navigation
fix(qmux): return connection credit for stopped receive streams (0.5.4) - #543
Conversation
Port of #520 to the 0.5 line. Stopping or dropping a RecvStream with unread data returns that data's MAX_DATA credit, as do late STREAM data after STOP_SENDING and the RESET_STREAM final-size gap. A stopped QMux stream keeps its backend entry and MAX_STREAMS slot until FIN or RESET_STREAM; the legacy webtransport binding still retires on STOP_SENDING. Bump qmux to 0.5.4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UnboundedSender::send counts a message before pushing it, so a send that began before close() can land after try_recv reports the queue empty. The send succeeds, the reader does not credit it, and the drain missed it. Drain until Disconnected, which tokio reports only once no send is in flight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
Caution
RecvStream::discard() loops until try_recv returns Disconnected. Tokio only returns that for a closed receiver starting in 1.49.0 (also backported to 1.47.3 and 1.43.4). qmux still declares tokio = "1.27". A consumer that runs cargo update -p qmux and keeps an older locked tokio would spin forever on every RecvStream stop or drop that happens before FIN.
Reviewed changes
Initial review of the 0.5.4 backport of #520's receive connection-credit fix, plus the follow-up that drains discard until Disconnected.
RecvStream::discard: onstop()and drop, closesinbound_data, credits buffered and queued bytes to the connection window, and drains until no send is in flight.- Reader-side credit: when
inbound_data.sendfails on a stopped stream, the reader credits those bytes through the newSessionState::credit_conn. The RESET gap uses the same helper. - Stopped-stream retention: under QMux, the writer no longer retires a receive entry on STOP_SENDING. The entry lives until FIN or RESET_STREAM. The legacy binding still retires it on STOP_SENDING.
StreamSlot: the backend entry and the frontend share one MAX_STREAMS slot, which is returned only after both drop.- Own-stream late frames: these are now ignored before any connection credit is charged.
- Tests: five tests ported from #520, plus a multi-threaded discard-race test that uses a new
#[cfg(test)]accessor,Credit::window().
claude-opus-5-5 | 𝕏
Draining until TryRecvError::Disconnected relies on tokio >= 1.49 (and 1.47.3/1.43.4); older tokio returns Empty forever for a closed receiver with a live sender, so discard spun forever, hanging a current_thread runtime. Track the bytes queued for a RecvStream in an atomic with a stopped bit instead: the reader counts bytes before queueing them unless the frontend stopped, in which case it credits them itself, and discard credits whatever is still counted. Also restore recv_open's doc comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backport from #510. CI runners now ship /usr/lib/llvm-22, whose libclang cannot load its dependencies inside the dev shell, so boring-sys failed the build job on this branch too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Design change: Each stream now shares an (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
✅ No new issues found. The new
Unreadcounter fixes the tokio-version hang from the prior review without raising thetokiofloor.
Reviewed changes
Re-reviewed the two commits pushed since the prior Pullfrog review at d09a6fa.
- Replaced the
discarddrain with a per-streamUnreadcounter: the reader counts bytes inRecvState::deliverbefore it queues them.discardcredits the buffered bytes plusUnread::stop(), so it no longer waits ontry_recvreturningDisconnected. Every byte is still credited exactly once: by the frontend on read or discard, or by the reader onceaddfails after the stop.cargo test -p qmux --lib --all-featurespasses all 145 tests, includingdata_racing_drop_is_credited. - Routed both reader delivery sites through
deliver: the fast path and the new-stream path now credit the connection window whenever delivery is refused. Dropping thedata_len > 0guard means a FIN-only frame after a stop callscredit_conn(0), which is harmless. - Restored
recv_open's doc comment abovefn recv_open. - Added
LIBCLANG_PATHto the Nix dev shell so bindgen loads Nix's libclang and not a host LLVM.
ℹ️ PR description still describes the superseded drain design
The description's "One fix goes beyond #520" paragraph, the release audit's correctness bullet ("waiting out any send already in flight"), the Review block (Head: d09a6fa), and the qmux-discard-race follow-up all describe the drain-until-Disconnected approach that 2317efd replaced. The follow-up quest would port that superseded drain to main, where it would hit the same tokio problem.
Technical details
# Stale PR description and follow-up quest after 2317efd
## Affected sites
- PR #543 body: "One fix goes beyond #520" paragraph and the "Discard race" validation bullet describe `try_recv` until `Disconnected` plus `yield_now`
- PR #543 body: "Release audit → Correctness" says `discard` waits out in-flight sends
- `quest/a0/qmux-discard-race.md` (added in #541): plans to port that drain to `main`
## Required outcome
- The PR body describes the shipped `Unread` counter design and its tokio-version independence.
- The `main` follow-up ports the `Unread` design, not the `Disconnected` drain, or states the tokio >= 1.49 floor it would need.claude-opus-5-5 | 𝕏

Ports moq-dev/web-transport#520's receive-credit fix to the
qmux-v0.5.xline and bumpsqmuxto 0.5.4, for quest/a0/qmux-05-recv-credit.md. Its claim PR is moq-dev/web-transport#541, which deletes the quest onmainonce 0.5.4 is published (@moq/qmux 0.3.5 already landed, so it can't carry the deletion).Summary
Stopping or dropping a
RecvStreamwith unread data now returns that data's connection credit (MAX_DATA). Late STREAM data after STOP_SENDING and the RESET_STREAM final-size gap on a stopped stream are credited back the same way. Before this, consumers pinned to 0.5 (such as moq-relay 0.17.x over WebSocket) wedged a connection once about one window (1 MiB by default) of abandoned data had piled up.The port keeps #520's design:
RecvStream::discardcloses the inbound channel, then credits everything buffered or queued. It runs onstop()and on drop. Data the reader delivers afterwards fails to send, and the reader credits it instead.webtransportbinding still retires the entry on STOP_SENDING, since it has no flow control and no promise of a RESET_STREAM.StreamSlot) is shared by the backend entry and the frontend. It is returned only when both are gone, so a peer that ignores STOP_SENDING stays bounded by its stream limit.One fix goes beyond #520, from this PR's review. #520's
discardclosed the channel and drained it untilTryRecvError::Empty. Tokio'sUnboundedSender::sendcounts a message before pushing it, so a reader send that began beforeclose()could land after the drain saw an empty queue. The send succeeded, so the reader did not credit it, and the drain had already missed it. On a multi-threaded runtime that leaked connection credit a little at a time. Each stream now shares anUnreadcounter (anAtomicU64with a stopped bit) between the reader and theRecvStream. The reader counts a chunk's bytes before queueing it, unless the frontend has stopped, in which case it credits them itself. The frontend uncounts a chunk when it takes it, anddiscardsets the stopped bit and credits whatever is still counted. A first fix drained untilDisconnectedinstead, but tokio before 1.49 never returns that for a closed receiver with a live sender, so it hung; see theDesign change:comment.mainhas the same race; see Follow-ups.This PR also backports moq-dev/web-transport#510's
LIBCLANG_PATHexport toflake.nix. CI runners now ship/usr/lib/llvm-22, whose libclang can't loadlibstdc++.so.6inside the dev shell, soboring-sysfailed thebuildjob on this branch whatever the change.Differences from #520, all mechanical: on 0.5,
Credit::consumereturnsOption<u64>instead of sending the frame itself, so the MAX_DATA and MAX_STREAMS sends are inlined. A smallSessionState::credit_connhelper covers the three reader-side sites, including the existing RESET-gap one. 0.5'sWriter::sendcontract and public API are unchanged.Release audit (full diff since
qmux-v0.5.3)qmux-v0.5.3is theqmux-v0.5.xhead, so the release diff is exactly this PR:rs/qmux/src/session.rs,rs/qmux/src/credit.rs(a#[cfg(test)]accessor only),flake.nix(dev shell only, not published),rs/qmux/Cargo.toml(0.5.3 to 0.5.4), andrs/qmux/CHANGELOG.md.RecvState,RecvStream's private fields, and the new privateStreamSlot).cargo publish --dry-runpackages and builds 0.5.4.MAX_DATAfor bytes it discards, and it sendsMAX_STREAMSfor a stopped peer stream at FIN or RESET_STREAM instead of when theRecvStreamis dropped.RecvStream::stop(), buffered but unread data is discarded, so a later read returns the stop error instead of the leftover bytes. On 0.5.3,read_chunkreturned buffered data before checking the stop.mainhas behaved this way since fix(qmux): return connection credit for stopped receive streams #520.closed()call, or drop. A sender that holds an idleSendStreamit never writes to again keeps the slot until then. This is RFC 9000's model and fix(qmux): return connection credit for stopped receive streams #520's accepted choice.@moq/qmuxonmainreplies to STOP_SENDING with RESET_STREAM immediately.discardcredits the buffer plus every byteUnreadstill counts, which includes any send already in flight, since the reader counts before it sends. After the stopped bit is set, the reader credits its own bytes instead of queueing them. After that, the reader credits on send failure. RESET_STREAM credits only the gap beyondrecv_offset, and the entry is removed in the same lookup, so a duplicate RESET or late STREAM data finds no entry and is ignored (therecv_openclosed check for peer streams, or the new own-stream early return for ours). No blocking findings.Validation
qmux 0.5.3from crates.io: N=0 stalls after exactly 64 streams (1,048,576 bytes). N=1024 stalls after 69 (the quest's Plan predicted 68; I did not chase the one-stream difference, since both stall at about one window).dropped_streams_return_connection_credit,stopped_stream_credits_late_data,stopped_stream_credits_reset_gap,stopped_local_stream_credits_late_data, andstopped_stream_holds_its_stream_slot_until_reset. All five fail on the 0.5.3 session code (with the new test module grafted on) and pass here. They check exactMAX_DATAvalues, so double-counting would fail them too.data_racing_drop_is_crediteddrops each of 100k streams while a 30-byte frame races the drop, then asserts the connection window returns exactly to its initial 1,000 bytes. With fix(qmux): return connection credit for stopped receive streams #520'sEmpty-terminated drain it failed 2 of 6 runs (2 of 3 at 300k iterations), losing 60 to 240 bytes. With theUnreadcounter it passed 13 of 13. It adds about 0.9 s to the qmux lib suite (0.32 s to 1.1 s); a race a few instructions wide needs that many iterations to show up.Disconnected-loop head (d09a6fa) hung (30 s timeout, once after only 168 streams). This head completes 400 of 400 in both modes on 1.48.0 and on 1.54.1.cargo test -p qmux --all-features: 145 unit tests plus every integration suite pass.cargo clippy -p qmux --all-targets --all-features -D warnings,cargo fmt --all --check, andcargo publish -p qmux --dry-runare clean.nix develop --command just checkfails on this host atboring-sys's bindgen (Unable to find libclang). It fails identically on theqmux-v0.5.3base, and CI hit the same panic on ubuntu-latest (runner llvm-22). The flake backport fixes CI's variant; this host's remaining one is the known Nix libclang issue the orchestrator is fixing separately, so CI is the check for the rest of the workspace.Follow-ups
quest/): port the discard-race fix and its test tomain, where fix(qmux): return connection credit for stopped receive streams #520 has the same drain. No open decisions.Release steps after merge
Per the quest's Plan, merging this PR approves the release. An agent then runs
cargo publish -p qmuxfrom the merged head, runsgit tag qmux-v0.5.4 <merged-head>(cargo publishmakes no tag), and pushes it withgit push origin refs/tags/qmux-v0.5.4. After that, moq-dev/web-transport#541 deletes the quest onmain.Choices made without asking
tokioto 1.49 so theDisconnecteddrain would be safe. TheUnreadcounter keepstokio >= 1.27, so no 0.5 consumer is forced to upgrade in a patch release, and the drain no longer depends on tokio's in-flight accounting at all.LIBCLANG_PATHline. Its quiche half (quiche 0.30.1 test expectations) is left out: CI'sjust checkandjust testpass without it.quest startcut the claim branch frommain. Retargeting it toqmux-v0.5.xwould need a force-push, which the rules forbid. So the release ships here fromrelease/qmux-0.5.4(as 0.5.3 did in moq-dev/web-transport#461), and the claim PR Release qmux 0.5.4 with the receive-credit fix #541 carries the main-side quest deletion that the Plan already put in a separate main PR.Public API and wire impact
No public API change, no new frames or encodings; timing of
MAX_DATAandMAX_STREAMSchanges as described in the audit. Semver patch.Caused by
Models
Size: M
Implementer: Claude, claude-opus-5-5, unknown effort (arm A)
🤖 Generated with Claude Code
(Written by Claude Opus 5.5)
Review
Rewritten by
quest reviewafter each round.Head: 00a2038
Base: origin/qmux-v0.5.x at ea2320a
Risk: medium,
the base has no RISK.md to name what is always high or customer-facing, so every change is mediumClassified: medium by default
Reviewer: Codex, gpt-6.1-sol, high effort
Session:
01a1de1c-30ba-7123-97bc-f82328449bcf, round 1Score: 9/10,
Static review found no actionable regressions and the credit accounting and stream-retirement changes appear sound, though builds and tests were not run as instructed.Rounds: 3; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0
Findings: none
Reviewer: Codex, gpt-6.1-sol, medium effort, shadow
Session:
01a1de1c-30ba-7222-b902-d11ff38d493f, round 1Score: 9/10,
The credit accounting and stream-retirement changes appear sound with focused regression coverage and no actionable regressions found by inspection, though builds and tests were not run.Rounds: 2; first round 0; findings 0: fixed 0, refuted 0, open 0
Findings: none