Skip to content

test(ts): judge the export on a congested path, not random loss - #5918

Merged
kixelated merged 7 commits into
mainfrom
quest/m1/bufferbloat
Oct 19, 2026
Merged

kixelated merged 7 commits into
mainfrom
quest/m1/bufferbloat

Conversation

@kixelated

@kixelated kixelated commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Quest: quest/m1/bufferbloat.md (deleted here).

Summary

  • just test ts gains --rate <bits/s> and --queue <duration> (default 2 s), passed to moq-shaper's token bucket on the relay -> subscriber leg (25 ms each way, 1500 B burst). --loss stays for correctness runs, and the two can combine.
  • Under --rate the oracle is the one the quest set:
    • T-STD conformant and continuous. compliance.py, pcr-timing.py and PCR accuracy gate as on a clean path.
    • Sheds whole groups, never stalls. holes.py --shed matches the capture's frames to the source's by PTS. Of each source group the capture keeps a decode-order prefix and never resumes mid-group.
    • Latency stays bounded. The source's last group must arrive whole: a stall, or latency grown past the export's 1 s delay, costs it.
    • No loss-percentage or hole-count thresholds. decode.py is skipped and --open-gop is refused, since both owe every unit. Passthrough runs passthrough.py --loss-ok.
  • Activation check. The rate limit must have throttled, and the export must have shed something (media skipped upstream or missed its deadline in its log), so a rate above the media rate fails loud.
  • Nightly ts.yml: --hrd --loss 1 becomes --hrd --rate 7000000 (below its 9 Mb/s video), and --passthrough --loss 1 becomes --passthrough --rate 7000000 (below its 10 Mb/s multiplex).
  • test/ts/README.md: the lossy-path section is now "Impaired path", with congested (--rate) and lossy (--loss, correctness only) subsections and the measurements below. The old random-loss ceiling table is gone; the random-loss audit re-judges what it drove.
  • test/drill/README.md and the impairment() doc in drills.rs say the drills prove correctness, so they keep random loss.

Why holes.py --shed matches by PTS: hang carries no DTS, so the export derives each picture's DTS. When a group is cut after a P picture, that picture's DTS moves (source 352800, export 363600). A DTS-step check misreads that as a mid-group resume.

Public API: none. Wire: none.

Measurements (2026-10-18, shared host)

Run Queue Duration Result
--hrd --rate 7000000 2 s 120 s passes; 37.6 s of video shed in whole groups, 0 overflowed
--hrd --rate 7000000 150 ms to 1 s 40-120 s 0 overflowed
--hrd --rate 7000000 100 ms 40-120 s, 4 runs 0, 0, 3, 28 overflowed
--hrd --rate 7000000 50 ms 120 s graders pass; 269 overflowed
--hrd --rate 7000000 --live 2 s 60 s passes; 99.8 % of PCRs on time, drift -28 ms
--passthrough --rate 7000000 2 s 120 s passes; gaps leave 463 CC errors (report only)
--hrd --rate 7000000, CUBIC (MOQ_QUIC_CONGESTION_CONTROL=loss) 2 s 60 s fails: 918 overflowed, latency outgrew the delay, export refused on clock drift

The CUBIC row is the negative control: a sender that fills the queue fails the run. --loss 1 and a clean default run still pass.

Decisions

1. Queue depth vs "the queue must overflow". The quest asks for a deep queue that overflows. Under the default BBRv3 those can't both hold: it keeps about one RTT queued (inflight about 2x BDP), so only a queue about one RTT deep (50 ms here) overflows reliably. 100 ms overflows intermittently, and 150 ms or deeper never does. The backlog lands at the relay instead, which evicts stale groups (Old).

  • ✅ A (implemented, recommended): a 2 s queue (twice the export's delay). The activation check is "the export shed media", not "the queue overflowed". This keeps the latency oracle meaningful: CUBIC fails the run, and BBR passes because it does not bloat the queue.
  • B: --queue 50ms in nightly, with "the shaper overflowed" as the activation check. Congestion drops then happen every run, but a 50 ms queue can't show latency growing with it.
  • C: both, as two nightly steps (about 2 more minutes per nightly run).

Settled as A by the orchestrating session under the user's overnight delegation, open to challenge (comment). The per-drill calls below stand on the same basis.

Calls I made (please review)

  • Relay drills, per drill:
    • cancel_under_backpressure_releases_the_reader: correctness (resources released, a rejoin is served). Keeps the 5% loss.
    • relay_killed_mid_group_aborts_then_resumes: correctness (abort surfaces, the same handles resume). Keeps.
    • interrupted_publisher_republishes_new_content: correctness (withdrawal, new content served). Keeps.
    • bursts_cross_a_cluster / _flapping_peer / _transport_peer: correctness. Every group is recovered live or by FETCH, the route returns, and IETF FETCH is served. The 10 s SETTLE is a stall bound (a third of Cross-node routing on cdn.moq.pro loses groups, stalls tracks and keeps closed broadcasts announced under bursty control traffic (A/B/C repro inside) #4349's 30 s), not a latency target. They already run a 100 kbit/s, 50 ms bottleneck that must overflow. Their random loss drops the MAX_STREAMS/MAX_DATA frames the credit waits exercise. Keep.
    • no_publisher_never_delivers: negative control. Keeps.
  • moq-quic sim and BBR3 unit tests: correctness. The sim asserts delivery, no deadlock and a clean drain, with a 600 s stall deadline, and checks priority only on a clean path. The BBR3 drop tests assert state transitions. Unchanged.
  • Passthrough under --rate: compliance stays report-only (--loss-ok), since verbatim packets keep their continuity counters across a gap. 59 of 60 gaps resumed on a video RAP. One near the source's end resumed on a non-PUSI packet.
  • No nightly --loss step remains. --loss is manual and correctness-only.

Checks

Follow-ups

  • Passthrough under shedding: decide whether export ts --passthrough should signal a discontinuity across a gap, or accept CC errors. Also investigate the end-of-stream gap that resumed mid-PES.
  • The TS export derives a different DTS for a picture whose group was cut short. It is T-STD valid, but worth a look in the TS recovery work.
  • Optional: a --rate arm of --live in nightly would grade end-to-end latency directly; it passed locally.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 2 commits October 18, 2026 22:12
`just test ts` gains --rate and --queue: moq-shaper's token bucket on the subscriber's leg behind a 2 s queue. Under --rate the export must stay T-STD compliant and continuous, shed only whole groups (holes.py --shed, matched by PTS), and still deliver the last group. The nightly --hrd and --passthrough lossy steps move to --rate 7000000; --loss stays for correctness. The relay drills, moq-quic's sim and BBR3's unit tests prove correctness and keep their random loss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: the harness work is done. The PR stays a draft on one open decision, queue depth vs "the queue must overflow" (see the body). BBRv3 keeps about one RTT queued, so only a queue about 50 ms deep overflows reliably. I recommend keeping the 2 s queue with "the export shed media" as the activation check (option A). Option B is --queue 50ms with an overflow check. The per-drill calls are in the body: all drills prove correctness and keep their random loss.

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Orchestrator decision, made under the user's overnight delegation to settle iterate decisions with the recommended option. It is open to challenge.

Queue depth: Option A. Nightly keeps the deep 2 s queue (twice the export's delay), and the activation check is "the export shed media", not "the queue overflowed".

Why: AGENTS.md asks for a rate limit below the media bitrate behind a deep but finite queue. Under BBRv3 a deep queue does not overflow, since it keeps about a round trip queued, so requiring overflow would force a queue too shallow (about 50 ms) to show latency growing with it. With 2 s the latency oracle stays meaningful: CUBIC fills the queue and fails the run, BBR does not and passes.

The per-drill calls in the description also stand: every relay drill proves correctness, so the impaired lane keeps its 5% random loss. That changes only if review shows a drill judges behavior.

(Written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 19, 2026 06:19
@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 18 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2b928757-003d-414d-bf75-2a0e9e498352
📥 Commits

Reviewing files that changed from the base of the PR and between ab6e69a and 6050cdd.

📒 Files selected for processing: 13
  • .github/workflows/ts.yml
  • quest/m1/README.md
  • quest/m1/bufferbloat.md
  • quest/m1/random-loss-audit.md
  • quest/m1/ts-reliability/grader-determinism.md
  • rs/moq-relay/tests/drills.rs
  • test/drill/README.md
  • test/ts/README.md
  • test/ts/holes.py
  • test/ts/holes.test.py
  • test/ts/passthrough.py
  • test/ts/passthrough.test.py
  • test/ts/run.sh
✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T17:49:28.032292Z 6050cdd Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 11f4bc70da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/ts/holes.py Resolved
kixelated and others added 2 commits October 19, 2026 09:34
# Conflicts:
#	.github/workflows/ts.yml
holes.py --shed graded the captured PTS as a set, so a capture that
reordered or repeated frames passed. Match the capture as an ordered
subsequence of the source's decode-order PTS instead, and fail any
frame past the last one that continues the source's order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9834620458

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/ts/run.sh Resolved
BBRv3 never fills a queue deeper than about 1.5 round trips (100 ms
overflowed in 3 of 6 runs, 150 ms and deeper never), so the 2 s congested
lane covers relay shedding but no tail drop. The new lane runs the relay's
loss-based CUBIC behind a 250 ms queue (five round trips), which overflowed
423-430 datagrams in each of four 120 s runs while every grader passed, and
fails a loss-based run whose queue never overflowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 037b250ff1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/ts/holes.py Resolved
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 037b250ff1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/ts/run.sh Resolved
kixelated and others added 2 commits October 19, 2026 10:23
… tail under --loss-ok

holes.py --shed now counts frames a capture skipped before its first kept one
against that group, so a capture opening mid-GOP fails as resuming mid-group.
passthrough.py --loss-ok still reports internal gaps but fails a capture that
ends before the source's last packet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 6050cddf48

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@kixelated
kixelated added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit d4b0a98 Oct 19, 2026
9 checks passed
@kixelated
kixelated deleted the quest/m1/bufferbloat branch October 19, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant