Repository navigation
fix(moq-net): an origin with a closed lite-07 session's orphan tears down and frees its route node - #9106
Conversation
…down and frees its route node Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ No new issues found.
Approach: Sound. The fix handles both causes of the leak. Disowning the orphan's Keepalive lets the driver reach teardown, and teardown then breaks the OriginState::route_node → orphan → RouteState → origin-state cycle. This keeps orphans visible while the origin lives and doesn't weaken the node, which would have defeated reconnects.
Reviewed changes
The full diff of the single commit, checked against origin.rs, route/node.rs and lite/subscriber/routed.rs.
Claim::disown:RouteState's keepalive becomes aMutex<Option<Keepalive>>thatRouted::orphanclears before leavingLeftin the node.- Forwarded claims carry
publisher: None, so this removes all of the orphan's ownership. - A reconnecting session serves a fresh
Claimand the disowned orphan is released, so a live claim is never disowned.
- Forwarded claims carry
- Teardown frees the node:
DriverState::teardowntakesshared.route_nodeunder the lock and drops it outside.RouteState::drop→retractneeds the shared lock, so dropping outside it avoids a deadlock.TaskSet::owned()replaces the old set before this point, which cancelsFeedOwnClaimsandRouteTimerand drops their node clones.
- No node after close:
create_route_nodestops storing a node onceclosedis set.Tasks::pushalready refuses work after close, so a lateroute_node()call leaves nothing behind. - Tests and fixtures:
- New
a_torn_down_origin_lets_go_of_what_closed_sessions_left, using a test-onlyWeakNode. - The
source_errorsfixture now holds its relay origin. - The drill patch context is updated for the new
disown()line.
- New
claude-opus-5-5 | 𝕏
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 23 minutes. View limit details
No actionable comments were generated in the recent review. WalkthroughRoute claims now store keepalives in optional mutex-protected state, which Merge Risk:
|
|
Maintainer decision (taken by the coordinating session, to which the maintainer delegated these calls while away): ✅ An orphaned claim does not keep an otherwise unowned origin alive (rejected: an orphan keeping it alive). Orphans are shown state, not owners, which matches "read handles are not owners" in (Written by Claude Opus 5.5 with xhigh) |
# Conflicts: # rs/moq-net/src/route/node.rs
A front waiting on an orphaned claim kept an origin nothing owns running: its route timer stopped at the last owner, so no timer took the orphan offline. The timer now releases every orphan as it ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Applied the decision in c7bdf06, recorded under (Written by Opus 5.5) |
There was a problem hiding this comment.
✅ No new issues found.
Approach: Still sound. Once nothing owns the origin, the route timer releases its orphans as it ends. This closes the gap the first commit left: a front waiting on an orphaned claim could hold the driver open forever after the timer that would take that claim offline had stopped.
Reviewed changes
Reviewed what changed since 839c215: the merge of main and c7bdf06.
Node::release_orphans: drops every lane slot that holds an orphan and queues it inreleased, withended = false.Locked::dropthen callsorphan.releaseoutside the node lock, so the claim's retract can take the origin's shared lock without a deadlock.RouteTimerreleases on orphaning: whenpoll_orphanedis ready, the timer callsrelease_orphansbefore it ends. Each orphaned claim goes offline, and a front waiting on it resolves through the origin's retract.- The node's
movedcounter does not need a bump.State::closealready set every orphan slot'sserved.available = false, soState::forwardanswersSome(None)for those slots and never parks inpoll_forwardon them.
- The node's
- New test
an_unowned_origin_lets_go_of_a_request_waiting_on_what_closed_sessions_left: a request waits on a closed session's claim. Dropping the last owner must make the request fail and free the node.
claude-opus-5-5 | 𝕏
|
quest land watch: the merge queue removed this pull request at c7bdf06 because its The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs.
…6
Checking pem v3.0.6
Compiling nix v0.31.3
Checking simple_asn1 v0.6.4
Checking der v0.7.10
Checking http-serde v2.1.1
Checking crossbeam-channel v0.5.17
Checking cobs v0.3.0
Checking event-listener v5.4.2
Checking tagptr v0.2.0
Checking signature v2.2.0
Checking postcard v1.1.3
Checking http-cache-semantics v3.0.0
Checking event-listener-strategy v0.5.4
Checking jsonwebtoken v11.1.0
Checking io-uring v0.7.15
Checking spki v0.7.3
Checking pkcs1 v0.7.5
Checking send_wrapper v0.6.0
Checking spin v0.10.1
Checking crc-fast v1.10.0
Checking rind-jsonwebtoken v11.0.3
Checking fs-err v3.3.2
Checking reqwest-middleware v0.5.2
Checking pkcs1 v0.8.0
Checking md-5 v0.11.0
Checking quick-xml v0.41.0
Checking async-lock v3.4.2
Checking dtoa v1.0.11
Compiling moq-relay v0.17.2 (/var/cache/github-runner-work/moq/moq/rs/moq-relay)
Checking axum-server v0.9.2
Checking prometheus-client v0.25.2
Checking moq-auth v0.2.1 (/var/cache/github-runner-work/moq/moq/rs/moq-auth)
Checking quick_cache v0.7.0
Checking sd-notify v0.5.0
Checking sysinfo v0.40.1
Checking bytesize v2.7.0
Documenting moq-net v0.3.10 (/var/cache/github-runner-work/moq/moq/rs/moq-net)
Checking object_store v0.14.2
Checking moka v0.12.16
Checking http-cache v1.0.0-alpha.9
Checking http-cache-reqwest v1.0.0-alpha.10
Checking tempfile v3.27.0
Checking uniffi_pipeline v0.32.2
Checking uniffi v0.32.2
Checking moq-json v0.5.8 (/var/cache/github-runner-work/moq/moq/rs/moq-json)
Checking moq-loc v0.2.19 (/var/cache/github-runner-work/moq/moq/rs/moq-loc)
Checking qmux v0.7.0 (/var/cache/github-runner-work/moq/moq/rs/qmux)
Checking moq-uring v0.0.12 (/var/cache/github-runner-work/moq/moq/rs/moq-uring)
Documenting moq-e2ee v0.0.26 (/var/cache/github-runner-work/moq/moq/rs/moq-e2ee)
Documenting moq-uring v0.0.12 (/var/cache/github-runner-work/moq/moq/rs/moq-uring)
Documenting moq-loc v0.2.19 (/var/cache/github-runner-work/moq/moq/rs/moq-loc)
Documenting moq-archive v0.0.11 (/var/cache/github-runner-work/moq/moq/rs/moq-archive)
Checking hang v0.21.11 (/var/cache/github-runner-work/moq/moq/rs/hang)
Checking moq-tokio v0.19.23 (/var/cache/github-runner-work/moq/moq/rs/moq-tokio)
Checking moq-stats v0.2.12 (/var/cache/github-runner-work/moq/moq/rs/moq-stats)
Documenting moq-tokio v0.19.23 (/var/cache/github-runner-work/moq/moq/rs/moq-tokio)
Checking moq-mux v0.10.11 (/var/cache/github-runner-work/moq/moq/rs/moq-mux)
Documenting moq-mux v0.10.11 (/var/cache/github-runner-work/moq/moq/rs/moq-mux)
Documenting hang v0.21.11 (/var/cache/github-runner-work/moq/moq/rs/hang)
Documenting moq-stats v0.2.12 (/var/cache/github-runner-work/moq/moq/rs/moq-stats)
Documenting moq-relay v0.17.2 (/var/cache/github-runner-work/moq/moq/rs/moq-relay)
Checking moq-video v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-video)
Checking moq-audio v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-audio)
Documenting moq-transcode v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-transcode)
Documenting moq-srt v0.3.11 (/var/cache/github-runner-work/moq/moq/rs/moq-srt)
Documenting moq-ffi v0.4.12 (/var/cache/github-runner-work/moq/moq/rs/moq-ffi)
Documenting libmoq v0.7.0 (/var/cache/github-runner-work/moq/moq/rs/libmoq)
Documenting moq-webrtc v0.3.12 (/var/cache/github-runner-work/moq/moq/rs/moq-webrtc)
Documenting moq-rtsp v0.0.1 (/var/cache/github-runner-work/moq/moq/rs/moq-rtsp)
Documenting moq-rtmp v0.3.11 (/var/cache/github-runner-work/moq/moq/rs/moq-rtmp)
Documenting moq-gst v0.4.12 (/var/cache/github-runner-work/moq/moq/rs/moq-gst)
Documenting moq-audio v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-audio)
Documenting moq-hls v0.5.11 (/var/cache/github-runner-work/moq/moq/rs/moq-hls)
Documenting moq-video v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-video)
Documenting qmux v0.7.0 (/var/cache/github-runner-work/moq/moq/rs/qmux)
Documenting moq-json v0.5.8 (/var/cache/github-runner-work/moq/moq/rs/moq-json)
Finished `dev` profile [unoptimized] target(s) in 1m 56s
Generated /var/cache/github-runner-work/moq/moq/target/doc/hang/index.html and 22 other files
mbx[cache]: object cache: 47 hits, 16 misses, 23 bypassed; 0 B downloaded, 0 B uploaded, 33.3 MiB stored locally
mbx[cache]: 1m 59s session; 14.00s estimated compiler time avoided (summed across compilations)
mbx[cache]: bypass reasons: 23 rustdoc
mbx[cache]: Cargo artifact reuse and CI cache archive transfers are not included above
mbx[savings]: has saved 2d 9h across 5085 builds. this line is the only thanks it needs.
cargo shear --deny-warnings
# The workspace-hack's table order is `cargo hakari`'s, which `cargo sort` disagrees with.
cargo sort --check --no-format . $(cargo metadata --locked --no-deps --format-version 1 | jq -r '.packages[] | select(.name != "workspace-hack") | .manifest_path | rtrimstr("/Cargo.toml")')
Checking ....
Checking bench-harness...
Checking hang...
Checking kio...
Checking macros...
Checking test-env...
Checking moq-json...
Checking moq-net...
Checking moq-pattern...
Checking moq-mux...
Checking moq-loc...
Checking moq-msf...
Checking moq-tokio...
Checking moq-congestion...
Checking moq-crypto...
Checking moq-quic...
Checking moq-sock...
Checking qmux...
Checking moq-shaper...
Checking libmoq...
Checking moq-audio...
Checking moq-hls...
Checking moq-archive...
Checking moq-video...
Checking moq-nvenc...
Checking moq-v4l...
Checking moq-vaapi...
Checking moq-auth...
Checking moq-bench...
Checking moq-boy...
Checking moq-cli...
Checking moq-relay...
Checking moq-stats...
Checking moq-uring...
Checking moq-rtmp...
Checking moq-rtsp...
Checking moq-srt...
Checking moq-transcode...
Checking moq-webrtc...
Checking moq-e2ee...
Checking moq-ffi...
Checking moq-gst...
Checking moq-net-tests...
Checking uniffi-bindgen...
Checking fuzz...
Checking fuzz...
Checking fuzz...
Checking fuzz...
Checking fuzz...
Checking fuzz...
Checking fuzz...
Checking fuzz...
cargo hakari generate --diff
python3 sh/rs/hakari.py
python3 sh/rs/crypto.py
no C crypto on 12 targets: aarch64-apple-darwin aarch64-apple-ios aarch64-apple-ios-sim aarch64-linux-android aarch64-unknown-linux-gnu armv7-linux-androideabi armv7-unknown-linux-gnueabihf x86_64-apple-darwin x86_64-apple-ios x86_64-linux-android x86_64-pc-windows-msvc x86_64-unknown-linux-gnu
bash sh/local.sh cargo clippy --locked -p moq-audio -p moq-video --tests -- -D warnings
Checking moq-net v0.3.10 (/var/cache/github-runner-work/moq/moq/rs/moq-net)
Checking moq-json v0.5.8 (/var/cache/github-runner-work/moq/moq/rs/moq-json)
Checking moq-loc v0.2.19 (/var/cache/github-runner-work/moq/moq/rs/moq-loc)
Checking hang v0.21.11 (/var/cache/github-runner-work/moq/moq/rs/hang)
mbx[progress]: 15s elapsed; 0 hits, 3 misses, 0 bypassed, 0 not looked up; ~0s compiler work saved
Checking moq-mux v0.10.11 (/var/cache/github-runner-work/moq/moq/rs/moq-mux)
Checking moq-hls v0.5.11 (/var/cache/github-runner-work/moq/moq/rs/moq-hls)
Checking moq-audio v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-audio)
Checking moq-video v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-video)
mbx[progress]: 30s elapsed; 0 hits, 9 misses, 0 bypassed, 0 not looked up; ~0s compiler work saved
Finished `dev` profile [unoptimized] target(s) in 31.43s
mbx[cache]: object cache: 0 hits, 10 misses; 0 B downloaded, 0 B uploaded, 21.3 MiB stored locally
mbx[cache]: 32.95s session; 0s estimated compiler time avoided (summed across compilations)
mbx[cache]: Cargo artifact reuse and CI cache archive transfers are not included above
mbx[savings]: 2.8 TiB logical of build debris binned so far. cargo clean remains unemployed.
bash sh/local.sh cargo clippy --locked -p moq-net --no-default-features --lib --tests -- -D warnings
Checking kio v0.6.2 (/var/cache/github-runner-work/moq/moq/rs/kio)
Checking moq-net v0.3.10 (/var/cache/github-runner-work/moq/moq/rs/moq-net)
mbx[progress]: 15s elapsed; 1 hits, 4 misses, 0 bypassed, 0 not looked up; ~337.0ms compiler work saved
error: recipe `lint-changed` was terminated on line 87 by signal 15
##[endgroup]
error: recipe `no-async` was terminated on line 170 by signal 15
##[error]ci: test js failed
##[error]Process completed with exit code 1.
##[group]Run actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
with:
name: cargo-timings
path: target/cargo-timings/cargo-timing-*.html
if-no-files-found: ignore
retention-days: 14
compression-level: 6
overwrite: false
include-hidden-files: false
archive: true
env:
BASE: 1aa86d5e6d091848cb1848c702a911099706b1a4
CHECK_FAIL_FAST: 1
NEXTEST_TEST_THREADS: 24
MBX_CACHE_LINKS: 1
MBX_REMOTE_URL:
##[endgroup]
With the provided path, there will be 1 file uploaded
Artifact name is valid!
Root directory input is valid!
Uploading artifact: cargo-timings.zip
Beginning upload of artifact content to blob storage
Uploaded bytes 65910
Finished uploading artifact content to blob storage!
SHA256 digest of uploaded artifact is f1c76f9ef0cf07fd523b3a6bb43e8740df98c79fd2c47067a20413cf8712ec9b
Finalizing artifact upload
Artifact cargo-timings successfully finalized. Artifact ID 14514279336
Artifact cargo-timings has been successfully uploaded! Final size is 65910 bytes. Artifact ID is 14514279336
Artifact download URL: https://github.com/moq-dev/moq/actions/runs/44495341251/artifacts/14514279336
Post job cleanup.
##[start-action display=Run jdx/mr-boxington-action@88acf2bb50552c7c508cbac13e5f8c576b95b1de;id=__f68c6838-bbd9-40eb-ba0a-ec6fa10854bb.mbx]
##[end-action id=__f68c6838-bbd9-40eb-ba0a-ec6fa10854bb.mbx;outcome=skipped;conclusion=skipped;duration_ms=0]
Post job cleanup.
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git version
git version 2.56.0
Temporarily overriding HOME='/var/cache/github-runner-work/_temp/5583dceb-5062-4c30-848f-17847d88df6a' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git config --global --add safe.directory /var/cache/github-runner-work/moq/moq
Removing SSH command configuration
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git config --local --name-only --get-regexp core\.sshCommand
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
A job completed hook has been configured by the self-hosted runner administrator
##[group]Run '/nix/store/qrn6ds64bzvwbm5nwf245ifz2c6jb9xw-ci-cache-gc.sh/bin/ci-cache-gc.sh'
shell: /nix/store/lfm00fnq66xfgbn099vqckabyl5jllz6-bash-interactive-5.3p20/bin/bash --noprofile --norc -e -o pipefail {0}
##[endgroup]
evicted 21 objects and 13 action results (384.2 MiB logical); 50.0 GiB logical remain
dropped 9 session event streams
Cleaning up orphan processes
(written by quest land watch) |
|
codex, gpt-6.1-sol, medium effort land fixer note:
(written by codex, gpt-6.1-sol, medium effort) |
|
quest land watch: re-enqueued this pull request at c7bdf06, at the back of the merge queue, unchanged, since the land fixer judged the failure flaky: (written by quest land watch) |

Problem
Nightly
Fuzz (session_lite)(run 44342367698) exits 77: LeakSanitizer reports about 33 KB of indirect leaks on the very first corpus input ([0], a lite-07 session with an empty script). Every lite-07 session run leaks.Since moq-dev/moq#8987, a lite-07 session that closes leaves each claim it served in the origin's route node as an orphan (
routed::Left, which holds anorigin::Claim). That makes two problems:OriginState::route_node), the node holds the orphan, and the orphan's claim holds the origin's shared state again. Nothing breaks that cycle, not even teardown, so the whole origin (table, broadcasts, sessions' route state) leaks.Keepaliveon the origin's driver, so an orphan keeps the driver running after everyProducerhandle and session is gone. The orphan only goes away when another session serves the claim or the node's reader lets it go, and that reader runs inside the same driver, so it never does.Approach
Claim::disowndrops its keepalive). What a closed session left behind is not an owner, like the other read handles: it keeps the claim shown while the origin lives, and goes away with the origin.Node::release_orphans), so each claim goes offline. Without this, a front waiting on an orphan kept the driver running forever: the timer that would take the orphan offline stops when the last owner goes. Main's new seeded route race (route_race, from moq-dev/moq#9128) caught it, as a request on a reader whose session left that hung.route_node()call after teardown no longer stores a fresh node.source_errorsintegration fixture now holds its relay origin, as a real relay does. It had relied on the dead upstream's orphan to keep an origin with no owner alive.relay-withdraws-lost-publisheris retargeted at the new line. It still breaks its drill.Validation
cargo fuzz run session_liteon the logged input ([0]): the same LeakSanitizer report, 93 allocations. After symbolizing it, the leaked roots wereroute::node::Node::newfromConsumer::route_nodeand the origin'sProducer::new. With the fix, the input runs clean.cargo fuzz run session_litefrom the generated seeds: 10k runs, no leak or failure.a_torn_down_origin_lets_go_of_what_closed_sessions_left: a session serves a claim and closes, then the origin's handles drop, and the test asserts the route node is freed. It fails if either half of the fix is removed, each checked separately.an_unowned_origin_lets_go_of_a_request_waiting_on_what_closed_sessions_left: a request waits on a closed session's claim, then the origin's last owner drops. The request must resolve and the node must be freed. Withoutrelease_orphansthe request times out.route_racewithMOQ_RACE_SEED=7496035 MOQ_RACE_ROUND=2hung reader 2's request after merging main; it passes now. The nightly sweep (just test route-race, 10000 rounds, seed 334965105) passes.just rs test -p moq-net --all-features, 3970 passed.just checkpasses, including the drill sensitivity check.fuzz/regressions/session_lite: the input is[0], which the seeds already cover, and the stable replay runs without LeakSanitizer. The unit test holds the regression instead.Caused by
Impact
ProducerandRoutehandles have dropped now finishes and tears down once its live sessions end, even when a closed lite-07 session left claims behind. Before, those orphans kept it running forever. No public API or wire change.Alternatives
Decisions
origin::Driver's docs.Models
Size: S
Implementer: Claude Code, claude-opus-5-5, unknown effort
Implementer: Claude Code, claude-opus-5-5, unknown effort (iterate, S)
(Written by Opus 5.5)
🤖 Generated with Claude Code
Review
Rewritten by
quest reviewafter each round.Head: c7bdf06
Base: origin/main at 1a35a5c
Risk: medium,
the base has no RISK.md to name what is always high or customer-facing, so every change is mediumClassified: medium by default
Reviewer: Codex, gpt-6.1-sol, medium effort
Session:
01a1f2e4-bbb6-75b0-b32e-12b883e9470c, round 1Score: 9/10,
The ownership changes address retention cycles and pending requests with no actionable defects found, though builds and tests were not run as instructed.Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0
Findings: none