Skip to content

fix(qmux): return connection credit for stopped receive streams (0.5.4) - #543

Merged
kixelated merged 4 commits into
qmux-v0.5.xfrom
release/qmux-0.5.4
Nov 15, 2026
Merged

kixelated merged 4 commits into
qmux-v0.5.xfrom
release/qmux-0.5.4

Conversation

@kixelated

@kixelated kixelated commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Ports moq-dev/web-transport#520's receive-credit fix to the qmux-v0.5.x line and bumps qmux to 0.5.4, for quest/a0/qmux-05-recv-credit.md. Its claim PR is moq-dev/web-transport#541, which deletes the quest on main once 0.5.4 is published (@moq/qmux 0.3.5 already landed, so it can't carry the deletion).

Summary

Stopping or dropping a RecvStream with unread data now returns that data's connection credit (MAX_DATA). Late STREAM data after STOP_SENDING and the RESET_STREAM final-size gap on a stopped stream are credited back the same way. Before this, consumers pinned to 0.5 (such as moq-relay 0.17.x over WebSocket) wedged a connection once about one window (1 MiB by default) of abandoned data had piled up.

The port keeps #520's design:

  • RecvStream::discard closes the inbound channel, then credits everything buffered or queued. It runs on stop() and on drop. Data the reader delivers afterwards fails to send, and the reader credits it instead.
  • Under QMux, a stopped stream's backend entry stays until FIN or RESET_STREAM, so late data and the reset gap are counted exactly once. The legacy webtransport binding still retires the entry on STOP_SENDING, since it has no flow control and no promise of a RESET_STREAM.
  • A peer-initiated stream's MAX_STREAMS slot (StreamSlot) is shared by the backend entry and the frontend. It is returned only when both are gone, so a peer that ignores STOP_SENDING stays bounded by its stream limit.
  • A late frame on one of our own retired streams is ignored before it is charged connection credit that nothing would return.

One fix goes beyond #520, from this PR's review. #520's discard closed the channel and drained it until TryRecvError::Empty. Tokio's UnboundedSender::send counts a message before pushing it, so a reader send that began before close() could land after the drain saw an empty queue. The send succeeded, so the reader did not credit it, and the drain had already missed it. On a multi-threaded runtime that leaked connection credit a little at a time. Each stream now shares an Unread counter (an AtomicU64 with a stopped bit) between the reader and the RecvStream. The reader counts a chunk's bytes before queueing it, unless the frontend has stopped, in which case it credits them itself. The frontend uncounts a chunk when it takes it, and discard sets the stopped bit and credits whatever is still counted. A first fix drained until Disconnected instead, but tokio before 1.49 never returns that for a closed receiver with a live sender, so it hung; see the Design change: comment. main has the same race; see Follow-ups.

This PR also backports moq-dev/web-transport#510's LIBCLANG_PATH export to flake.nix. CI runners now ship /usr/lib/llvm-22, whose libclang can't load libstdc++.so.6 inside the dev shell, so boring-sys failed the build job on this branch whatever the change.

Differences from #520, all mechanical: on 0.5, Credit::consume returns Option<u64> instead of sending the frame itself, so the MAX_DATA and MAX_STREAMS sends are inlined. A small SessionState::credit_conn helper covers the three reader-side sites, including the existing RESET-gap one. 0.5's Writer::send contract and public API are unchanged.

Release audit (full diff since qmux-v0.5.3)

qmux-v0.5.3 is the qmux-v0.5.x head, so the release diff is exactly this PR: rs/qmux/src/session.rs, rs/qmux/src/credit.rs (a #[cfg(test)] accessor only), flake.nix (dev shell only, not published), rs/qmux/Cargo.toml (0.5.3 to 0.5.4), and rs/qmux/CHANGELOG.md.

  • Public API: none. Only private fields and types change (RecvState, RecvStream's private fields, and the new private StreamSlot). cargo publish --dry-run packages and builds 0.5.4.
  • Wire: no new frames or encodings. The session now sends MAX_DATA for bytes it discards, and it sends MAX_STREAMS for a stopped peer stream at FIN or RESET_STREAM instead of when the RecvStream is dropped.
  • Behavior changes a caller could notice:
    • After RecvStream::stop(), buffered but unread data is discarded, so a later read returns the stop error instead of the leftover bytes. On 0.5.3, read_chunk returned buffered data before checking the stop. main has behaved this way since fix(qmux): return connection credit for stopped receive streams #520.
    • A peer-initiated stream's MAX_STREAMS slot is now freed only when the stream also ends on the wire (FIN or RESET_STREAM), not as soon as the application drops it. A 0.5 sender replies to STOP_SENDING with RESET_STREAM only once it notices the stop: on its next write, a closed() call, or drop. A sender that holds an idle SendStream it never writes to again keeps the slot until then. This is RFC 9000's model and fix(qmux): return connection credit for stopped receive streams #520's accepted choice. @moq/qmux on main replies to STOP_SENDING with RESET_STREAM immediately.
  • Correctness: each byte is credited exactly once. Before the frontend stops, bytes are credited on read. When it stops, discard credits the buffer plus every byte Unread still counts, which includes any send already in flight, since the reader counts before it sends. After the stopped bit is set, the reader credits its own bytes instead of queueing them. After that, the reader credits on send failure. RESET_STREAM credits only the gap beyond recv_offset, and the entry is removed in the same lookup, so a duplicate RESET or late STREAM data finds no entry and is ignored (the recv_open closed check for peer streams, or the new own-stream early return for ours). No blocking findings.

Validation

  • Real crate, before and after. A scratch binary runs over TCP on localhost with the default config (1 MiB window). The sender opens 16 KiB uni streams; the receiver accepts each, reads N bytes, and drops it. A stream that makes no progress within 2 s counts as a stall.
    • qmux 0.5.3 from crates.io: N=0 stalls after exactly 64 streams (1,048,576 bytes). N=1024 stalls after 69 (the quest's Plan predicted 68; I did not chase the one-stream difference, since both stall at about one window).
    • This branch: 400 of 400 for both N=0 and N=1024.
  • Regression tests. Ported fix(qmux): return connection credit for stopped receive streams #520's five tests: dropped_streams_return_connection_credit, stopped_stream_credits_late_data, stopped_stream_credits_reset_gap, stopped_local_stream_credits_late_data, and stopped_stream_holds_its_stream_slot_until_reset. All five fail on the 0.5.3 session code (with the new test module grafted on) and pass here. They check exact MAX_DATA values, so double-counting would fail them too.
  • Discard race. The new multi-threaded test data_racing_drop_is_credited drops each of 100k streams while a 30-byte frame races the drop, then asserts the connection window returns exactly to its initial 1,000 bytes. With fix(qmux): return connection credit for stopped receive streams #520's Empty-terminated drain it failed 2 of 6 runs (2 of 3 at 300k iterations), losing 60 to 240 bytes. With the Unread counter it passed 13 of 13. It adds about 0.9 s to the qmux lib suite (0.32 s to 1.1 s); a race a few instructions wide needs that many iterations to show up.
  • Old tokio. With tokio pinned to 1.48.0 in the stall harness, the intermediate Disconnected-loop head (d09a6fa) hung (30 s timeout, once after only 168 streams). This head completes 400 of 400 in both modes on 1.48.0 and on 1.54.1.
  • cargo test -p qmux --all-features: 145 unit tests plus every integration suite pass. cargo clippy -p qmux --all-targets --all-features -D warnings, cargo fmt --all --check, and cargo publish -p qmux --dry-run are clean.
  • nix develop --command just check fails on this host at boring-sys's bindgen (Unable to find libclang). It fails identically on the qmux-v0.5.3 base, and CI hit the same panic on ubuntu-latest (runner llvm-22). The flake backport fixes CI's variant; this host's remaining one is the known Nix libclang issue the orchestrator is fixing separately, so CI is the check for the rest of the workspace.

Follow-ups

Release steps after merge

Per the quest's Plan, merging this PR approves the release. An agent then runs cargo publish -p qmux from the merged head, runs git tag qmux-v0.5.4 <merged-head> (cargo publish makes no tag), and pushes it with git push origin refs/tags/qmux-v0.5.4. After that, moq-dev/web-transport#541 deletes the quest on main.

Choices made without asking

Public API and wire impact

No public API change, no new frames or encodings; timing of MAX_DATA and MAX_STREAMS changes as described in the audit. Semver patch.

Caused by

Models

Size: M
Implementer: Claude, claude-opus-5-5, unknown effort (arm A)

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 00a2038

  • Base: origin/qmux-v0.5.x at ea2320a

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1de1c-30ba-7123-97bc-f82328449bcf, round 1

  • Score: 9/10, Static review found no actionable regressions and the credit accounting and stream-retirement changes appear sound, though builds and tests were not run as instructed.

  • Rounds: 3; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, medium effort, shadow

  • Session: 01a1de1c-30ba-7222-b902-d11ff38d493f, round 1

  • Score: 9/10, The credit accounting and stream-retirement changes appear sound with focused regression coverage and no actionable regressions found by inspection, though builds and tests were not run.

  • Rounds: 2; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

Port of #520 to the 0.5 line. Stopping or dropping a RecvStream with
unread data returns that data's MAX_DATA credit, as do late STREAM data
after STOP_SENDING and the RESET_STREAM final-size gap. A stopped QMux
stream keeps its backend entry and MAX_STREAMS slot until FIN or
RESET_STREAM; the legacy webtransport binding still retires on
STOP_SENDING. Bump qmux to 0.5.4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UnboundedSender::send counts a message before pushing it, so a send that
began before close() can land after try_recv reports the queue empty. The
send succeeds, the reader does not credit it, and the drain missed it.
Drain until Disconnected, which tokio reports only once no send is in
flight.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 15, 2026 07:17
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T07:30:13.102576Z 00a2038 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

RecvStream::discard() loops until try_recv returns Disconnected. Tokio only returns that for a closed receiver starting in 1.49.0 (also backported to 1.47.3 and 1.43.4). qmux still declares tokio = "1.27". A consumer that runs cargo update -p qmux and keeps an older locked tokio would spin forever on every RecvStream stop or drop that happens before FIN.

Reviewed changes

Initial review of the 0.5.4 backport of #520's receive connection-credit fix, plus the follow-up that drains discard until Disconnected.

  • RecvStream::discard: on stop() and drop, closes inbound_data, credits buffered and queued bytes to the connection window, and drains until no send is in flight.
  • Reader-side credit: when inbound_data.send fails on a stopped stream, the reader credits those bytes through the new SessionState::credit_conn. The RESET gap uses the same helper.
  • Stopped-stream retention: under QMux, the writer no longer retires a receive entry on STOP_SENDING. The entry lives until FIN or RESET_STREAM. The legacy binding still retires it on STOP_SENDING.
  • StreamSlot: the backend entry and the frontend share one MAX_STREAMS slot, which is returned only after both drop.
  • Own-stream late frames: these are now ignored before any connection credit is charged.
  • Tests: five tests ported from #520, plus a multi-threaded discard-race test that uses a new #[cfg(test)] accessor, Credit::window().

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread rs/qmux/src/session.rs Resolved
Comment thread rs/qmux/src/session.rs Resolved
kixelated and others added 2 commits November 14, 2026 23:28
Draining until TryRecvError::Disconnected relies on tokio >= 1.49 (and
1.47.3/1.43.4); older tokio returns Empty forever for a closed receiver
with a live sender, so discard spun forever, hanging a current_thread
runtime. Track the bytes queued for a RecvStream in an atomic with a
stopped bit instead: the reader counts bytes before queueing them unless
the frontend stopped, in which case it credits them itself, and discard
credits whatever is still counted. Also restore recv_open's doc comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backport from #510. CI runners now ship /usr/lib/llvm-22, whose libclang
cannot load its dependencies inside the dev shell, so boring-sys failed
the build job on this branch too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Design change: RecvStream::discard no longer drains its channel until TryRecvError::Disconnected. Pullfrog pointed out that tokio returns Disconnected for a closed, empty receiver with a live sender only since 1.49 (and 1.47.3/1.43.4). qmux allows tokio >= 1.27, and on older tokio that loop spun forever. I reproduced it: with tokio pinned to 1.48.0, the stall harness hung on the previous head (d09a6fa) in both modes, once after only 168 streams.

Each stream now shares an Unread counter (an AtomicU64 with a stopped bit) between the reader and the RecvStream. The reader counts a chunk's bytes before queueing it, unless the frontend has stopped, in which case the reader credits them itself. The frontend uncounts a chunk when it takes it off the channel, and discard sets the stopped bit and credits whatever is still counted. Every byte is credited exactly once, whatever tokio version resolves, with no spinning. I considered raising the tokio floor to 1.49 instead; I rejected it because the drain would still depend on tokio's in-flight accounting, and the floor raise would force an upgrade on 0.5 consumers in a patch release.

(Written by Claude Opus 5.5)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The new Unread counter fixes the tokio-version hang from the prior review without raising the tokio floor.

Reviewed changes

Re-reviewed the two commits pushed since the prior Pullfrog review at d09a6fa.

  • Replaced the discard drain with a per-stream Unread counter: the reader counts bytes in RecvState::deliver before it queues them. discard credits the buffered bytes plus Unread::stop(), so it no longer waits on try_recv returning Disconnected. Every byte is still credited exactly once: by the frontend on read or discard, or by the reader once add fails after the stop. cargo test -p qmux --lib --all-features passes all 145 tests, including data_racing_drop_is_credited.
  • Routed both reader delivery sites through deliver: the fast path and the new-stream path now credit the connection window whenever delivery is refused. Dropping the data_len > 0 guard means a FIN-only frame after a stop calls credit_conn(0), which is harmless.
  • Restored recv_open's doc comment above fn recv_open.
  • Added LIBCLANG_PATH to the Nix dev shell so bindgen loads Nix's libclang and not a host LLVM.

ℹ️ PR description still describes the superseded drain design

The description's "One fix goes beyond #520" paragraph, the release audit's correctness bullet ("waiting out any send already in flight"), the Review block (Head: d09a6fa), and the qmux-discard-race follow-up all describe the drain-until-Disconnected approach that 2317efd replaced. The follow-up quest would port that superseded drain to main, where it would hit the same tokio problem.

Technical details
# Stale PR description and follow-up quest after 2317efd

## Affected sites
- PR #543 body: "One fix goes beyond #520" paragraph and the "Discard race" validation bullet describe `try_recv` until `Disconnected` plus `yield_now`
- PR #543 body: "Release audit → Correctness" says `discard` waits out in-flight sends
- `quest/a0/qmux-discard-race.md` (added in #541): plans to port that drain to `main`

## Required outcome
- The PR body describes the shipped `Unread` counter design and its tokio-version independence.
- The `main` follow-up ports the `Unread` design, not the `Disconnected` drain, or states the tokio >= 1.49 floor it would need.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated merged commit 3625c96 into qmux-v0.5.x Nov 15, 2026
3 checks passed
@kixelated
kixelated deleted the release/qmux-0.5.4 branch November 15, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant