Skip to content

fix(relay): a connection's task frees its startup state once the session starts - #8943

Merged
kixelated merged 3 commits into
mainfrom
quest/m2/relay-memory
Nov 16, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m2/relay-memory

Conversation

@kixelated

@kixelated kixelated commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m2/relay-memory.md.

Problem

The 2026-11-05 release pairing recorded relay RSS but never reported it: in chat (2,001 connections) the relay used about 17% more memory on main than on release. The cut waits on a cause, and on the comparison reporting memory so the next one cannot miss it.

Approach

Reproduced on today's main (6488ce6b87) against origin/release (ca3b801a6c): chat RSS +16.1% (+35 KiB per connection), every pair larger. A jemalloc heap profile of both sides, attributed by owner in KiB per connection (main minus release):

Owner before after this PR
relay (moq-relay's per-connection tasks) +65.3 -13.1
origin (moq_net::model) +12.4 +9.7
session (moq_net protocol, kio) -1.3 +1.9
transport (QUIC, TLS, Tokio driver) -38.3 -38.6
total +38.8 -39.2
  • Cause, fixed. Connection::run passed its startup future by value through Lifecycle::start, Lifecycle::run and Lifecycle::isolate. An async fn keeps its arguments for its whole life, so the connection's task held 44.8 KiB (startup's largest state, twice) plus a 20.4 KiB box, until the connection closed. run now boxes the startup future before building the task, so it is freed when the session starts, and start builds the supervision with join/map instead of an async block that kept a second copy of the session's state. Per-connection futures go from 44.8 + 20.4 + 12.3 KiB to 1.4 + 6.2 KiB (-Zprint-type-sizes).
  • Accepted. Origin's +10 KiB per connection is per-reader bookkeeping release lacks (report record moq-dev/moq#8440, one-delivery record moq-dev/moq#5446, demand preferences moq-dev/moq#5670), recorded in .github/release-perf.md. The in-tree QUIC stack saves 38 KiB per connection over noq.
  • Reporting. bench/compare.py pairs rss_bytes like CPU; bench/README.md and the performance audit's Release baseline ask for it under the same accept-or-fix rule.

Impact

  • moq_relay::Connection::run is fn run(self) -> impl Future<Output = ()> instead of async fn run(self): same output and auto traits, so callers that .await or spawn it are unchanged.
  • The negotiated log line is emitted when start returns rather than on the supervision's first poll, in the same span.
  • No wire or config change.

Alternatives

  • Shrinking supervise itself, which keeps its 2.3 KiB of arguments twice. It is public (moq --listen uses it) and turning it into a non-async fn would move its setup out of the first poll; 2 KiB per connection did not earn that.
  • Putting Lifecycle (1 KiB, of which Cluster is 792 bytes by value) behind an Arc: the task now holds one copy, so it would save under 1 KiB per connection; left as is.

Caused by

  • moq-dev/moq#5761 - moved the startup box behind by-value async fn arguments, which put a second copy of startup's state in the connection's task (found by the pre-cut audit of the release pairing). Holding startup's state in one box for the session's life is as old as the task: release's task holds about 27 KiB.

Decisions

Taken while unattended, each undoable in review:

  • Fixed the cause here, as the quest's Plan directs for accidental cost.
  • Accepted origin's ~10 KiB per connection as the price of its features, as recorded.
  • The run directories stayed in this checkout's .tmp/bench (the session's brief), not a durable directory outside it as the quest suggested; every number is copied into .github/release-perf.md and here.
  • Removed the cut's "waits on relay memory" decision and Required link along with the quest.

Follow-ups

  • Relay CPU (new, m2): the same pairing found chat and fanout relay CPU per delivered frame +43% and +35% on main, every pair slower, the opposite of the 2026-11-05 result. Lead: moq_quic's sent-packet Slots frees and reallocates an 8 KiB chunk per packet on a connection that has everything acknowledged (half of chat's allocation churn). Open decision: whether the cut waits on it (recommended: yes, so it stays in m2) or it moves to m3 for the performance audit.

Validation

  • just bench-relay origin/release quic, five alternating rounds each, on fedora (32 CPUs, 1-minute load 13 to 26 at snapshots). Median relay RSS, release / main:
    • before: chat 427.0 / 495.6 MiB (+16.1%, range +14.1 to +29.4), fanout 37.2 / 41.1 (+10.3%), video 30.5 / 29.1 (-3.5%)
    • after: chat 427.1 / 392.1 MiB (-7.2%, range -13.4 to -6.8), fanout 36.9 / 37.4 (-2.7%), video 29.9 / 27.4 (-6.0%)
    • Delivered frames matched in every pair.
  • Second, independent measurement: just bench-profile heap chat 8 tokio-workers origin/release, jemalloc retained heap at steady end: release 466 / 471 MiB, main 542 MiB before and 395 MiB after (the table above). Steady-window growth was 7.7 MiB of 568, and teardown returns near the idle heap on both sides: no leak.
  • RUSTC_BOOTSTRAP=1 cargo rustc --release -p moq-relay --lib -- -Zprint-type-sizes before and after for the future sizes quoted.
  • just rs test -p moq-relay --all-features (492 passed), just check, python3 bench/compare.test.py.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, high effort (arm B)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 3185c96

  • Base: origin/main at 13e593c

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1e4ff-2f35-7ff3-8a8f-60186751e297, round 2

  • Score: 10/10, The incremental change fixes unavailable RSS handling, and code inspection found no new issues.

  • Rounds: 2; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0

  • Findings: none

  • The last round's findings, at 3fb409e:

    • Fixed: medium, bench/compare.py:24: Allow unavailable RSS when host sampling is absent; bench/compare.py includes rss_bytes in HOST_METRICS and reports it unavailable when all paired samples omit it; bench/compare.test.py covers missing CPU and RSS on both sides.

kixelated and others added 2 commits November 16, 2026 06:25
Quest-Claim: e2c3babe2126dc5c
…ion starts

The chat workload's relay held 35 KiB more per connection on main than on
release. A jemalloc profile put it on moq-relay's per-connection task: the
startup future passed by value through Lifecycle's async fns, which keep
their arguments for their whole life, so the task held 44.8 KiB and a box
beside it 20.4 KiB until the connection closed. Box the startup future
before the task is built and build the supervision with combinators; the
task is now 1.4 KiB plus a 6.2 KiB supervision, and chat's relay RSS is 7%
under release's.

The release comparison now reports relay RSS beside CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title The relay's memory growth since release is explained before the cut fix(relay): a connection's task frees its startup state once the session starts Nov 16, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 16, 2026 15:40
@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a14cfc74-7afd-47ad-88f8-af1bfa2cb6e5
Commits

Reviewing files that changed from the base of the PR and between 6488ce6 and 3185c96.

Files selected for processing: 11

  • .github/release-perf.md
  • bench/README.md
  • bench/compare.py
  • bench/compare.test.py
  • quest/m2/README.md
  • quest/m2/relay-cpu.md
  • quest/m2/relay-memory.md
  • quest/m3/audit-perf.md
  • quest/m3/release/cut.md
  • rs/moq-relay/src/connection.rs
  • rs/moq-relay/src/incoming.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

I reviewed the whole PR at 3185c96: the restructured relay connection task, RSS pairing in the bench comparison, and the quest/report bookkeeping.

  • Startup future freed once the session starts. Connection::run now boxes Self::start(...) before building the task, so the Box drops when isolate completes and the startup state no longer lives as long as the connection. I traced both paths. Without a shared runtime, start and supervision are each still isolated under the connection span, and a panic still closes the connection with connection panicked. With a shared runtime, start still runs on the worker and supervision is still spawned on the shared runtime with the same close logging.
  • Supervision built with combinators. start returns join(supervise(..), R::drive(driver)).map(..).boxed() directly, and the negotiated log moves into start, still under the connection span through isolate. isolate now takes Pin<Box<F>> with F: ?Sized, so passing an existing BoxFuture does not box it a second time. The new 'static bound on Sealed::drive holds for every impl: each is an async fn over a concrete 'static Driver.
  • RSS in bench/compare.py. rss_bytes is paired like the host CPU metrics, with the same "unavailable" handling when both sides have no samples. summary.py already emits it, and the tests cover both the delta and the case where samples are missing.
  • Quest and report updates. .github/release-perf.md records the measurements, and quest/m2/relay-cpu.md is the new follow-up. Removing quest/m2/relay-memory.md leaves no dangling links, and the new #relay-memory-2026-11-16 and #method anchors both resolve.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed 3185c96bfd391e062203cdacc0b728494810c366. No blocker for the memory fix. The boxed startup future is gone before the session runs, and both phases retain panic isolation, the connection span, and the existing driver/supervision join. All 19 tests in bench/compare.test.py passed locally; I did not rerun the Rust tests or relay pairing.

I’d narrow the release note: chat RSS consistently improved in this pairing; fanout’s paired range still crosses zero. Keep the CPU-per-frame regression separate from the memory result and retire the earlier blanket claim of lower CPU until the follow-up pairing explains the reversal.

@kixelated
kixelated added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit fc38289 Nov 16, 2026
7 checks passed
@kixelated
kixelated deleted the quest/m2/relay-memory branch November 16, 2026 21:41
@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (2026-11-16, /quest-run): merge, and keep follow-up quest/m2/relay-cpu.md in m2 with its decision answered: ✅ the cut waits on it (rejected: moving it to m3; dropping it).

(Written by Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant