Skip to content

ci(bench): the GStreamer catalog bench runs in the supplemental Nightly group - #9199

Open
kixelated wants to merge 1 commit into
mainfrom
fix/nightly-gst-bench-extra
Open

kixelated wants to merge 1 commit into
mainfrom
fix/nightly-gst-bench-extra

Conversation

@kixelated

@kixelated kixelated commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Nightly's Rust bench targets job has failed at its 55-minute Run workload timeout every night since at least 2026-11-16, and moq-gst's catalog bench, which runs last in that group, never ran. Because it never finished, no night kept its executable as the next night's base, so each night has to build the base from source.

  • 2026-11-16 to 11-18: the base side hung for 600 s each in moq-net/auth, moq-net/publisher, and moq-net/session. Those hangs were already fixed on main by moq-dev/moq#9039 and the commits around it, and they did not recur in run 44562804083 (job 135696388596).
  • 2026-11-19 had no hang and still timed out: rust-bench finished moq-net, its CPU clock, kio, and their confirmation rounds at minute 43.5. The gst head build took 7m21s on a cold cache, and the step timed out while the base was building from source (No kept base executable ... building the base).

The suite outgrew the budget. gst costs about 16 minutes with no kept base and about 9 with one, and either amount pushes rust-bench (43.5 min without it) past 55 minutes.

Approach

Move criterion gst -p moq-gst --bench catalog from rust-bench to rust-extra ("Rust supplemental benchmarks"), still under its own label, next to msf's same-named catalog bench. Same dev shell, same paired comparison.

Budget from the 2026-11-19 run (55-minute workload step each):

Group Workload before After (worst case, base built from source)
rust-bench timed out at 55 (43.5 without gst) ~43.5
rust-extra 28.5 ~44.5

Once a night keeps gst's executable, rust-extra drops to about 38 minutes.

Validation

  • python3 sh/gh/nightly.test.py: 22 tests pass, including test_every_cargo_bench_target_runs_nightly (moq-gst is still covered).
  • cargo bench -p moq-gst --bench catalog -- --test in the dev shell: all 9 cases pass. The bench hadn't run in Nightly for days, so I checked it still works.
  • Timings come from the logs of jobs 135696388596 (rust-bench) and 135696388603 (rust-extra) in run 44562804083.
  • just check passes.

Caused by

No single PR: gst has run last in rust-bench since moq-dev/moq#5759, and the group grew past its budget over many bench additions plus the paired base run.

Impact

None: CI layout only. No public API or wire change.

Alternatives

  • Raise the step timeout: the job's own timeout-minutes: 60 caps it, and the cold-cache build cost would stay in the critical path.
  • Move gst to rust-relay (about 16 minutes of work): more headroom, but that job is the relay session suite.
  • Split moq-net across two jobs: a bigger change than one 9 to 16 minute target needs.

Follow-ups

  • Any 600 s hang watchdog trip costs a third of a group's budget, and a hang on the base side repeats the next night. Suggested in the triage report; no quest yet.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, unknown effort

(Written by Opus 5.5)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 742ed46

  • Base: origin/main at 1471225

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1f4e4-32ba-7ea0-9975-655f9d43529a, round 1

  • Score: 10/10, The benchmark moves to another existing Nightly group without losing coverage, and the updated assertions reflect that move; source inspection found no actionable defects, with tests intentionally not run.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

…ly group

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 19, 2026 17:39
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2b4e3efe-95b2-499e-94c8-69aa1e05d751
Commits

Reviewing files that changed from the base of the PR and between 1471225 and 742ed46.

Files selected for processing: 2

  • sh/gh/nightly.sh
  • sh/gh/nightly.test.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound — moves the GStreamer catalog bench into the existing rust-extra group, so it keeps its label and paired comparison and rust-bench returns inside its 55-minute budget.

Reviewed changes

Reviewed the full diff of both files.

  • sh/gh/nightly.sh — criterion gst -p moq-gst --bench catalog moves from rust-bench to rust-extra, after msf's catalog bench, and the comment about the shared target name moves with it.
  • sh/gh/nightly.test.py — rust-bench now expects 3 release invocations ending in kio. The failed-build test's last call is updated to match. test_every_cargo_bench_target_runs_nightly still covers moq-gst.

I found no other references to the old placement in sh/gh or .github.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant