You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci(benchmark): run the engine benchmark on GitHub Actions and post the report to Discussions #2125
Is your feature request related to a problem? Please describe.
The engine benchmark (examples/benchmark/run.sh) only runs by hand on a local machine. So the docs' Engine Support speed table goes stale between releases. #2109 had to re-measure it once 3.15 shipped.
Describe the solution you'd like
A benchmark.yml workflow:
Triggers: a weekly schedule, at an off-hour slot clear of the Saturday crons and of Monday's CI Timing and Python Versions, plus workflow_dispatch with inputs for the Python versions and the rounds.
Steps: run examples/benchmark/run.sh on ubuntu-latest, which is native x86_64 with Docker, then upload out/ as an artifact.
Report: a discussion job, modelled on ci-timing.yml's, the only job with discussions: write. It posts the ratio table (table.rst), with its absolute-time footnote, through util/discussion_report.py to a benchmark thread in CI reports, naming the runner image.
Docs: stay hand-updated from the report; the workflow never pushes.
Describe alternatives you've considered
Running per push or PR: the runner pool is already saturated. Auto-committing the docs tables: that bypasses review, and the absolute ms come from shared, unknown hardware. The within-pass ratios cancel most runner noise; the absolute numbers do not.
Additional context
Runtime of a full matrix on a hosted runner is unmeasured. The first dispatch measures it, and that sets timeout-minutes and the cadence.
Activity
ci(benchmark): run the engine benchmark on Actions and post the repor…
VERIFY ONLY: compare benchmark runners (#2125)
ci(benchmark): run the engine benchmark on Actions and post the repor…
ci(benchmark): run the engine benchmark on Actions and post the repor…
ci(benchmark): run the engine benchmark on Actions and post the repor…