Repository navigation
ci(benchmark): run the engine benchmark on Actions and post the report to Discussions (#2125) - #2128
Conversation
|
NEEDS CHANGES at
A revision is in progress; I'll re-run the comparison on it. |
5663e92 to
abb234d
Compare
|
GOOD TO GO at
Runner comparison: run 40047570595 on |
|
Runner comparison (40047570595). Every engine except pyshark, on 3.10–3.15, one leg per runner:
|
1a4bffc to
8e38aa6
Compare
|
GOOD TO GO at
|
…t to Discussions (#2125) - benchmark.yml: weekly (Wed 03:17 UTC) and on dispatch, with the runner, Python versions, engines, rounds and repeats as inputs. One job, on ubuntu-26.04-arm by default, installs the engine tiers from python-versions.toml and runs native.sh; `discussion`, the only job with `discussions: write`, posts the ratio table to `benchmark: weekly engine benchmark report` in CI reports. Nothing pushes; the docs stay hand-updated. - examples/benchmark/native.sh: run.sh without Docker, on the interpreters already installed, with run.sh's virtualenvs, version checks, gap notes and exit statuses. run.sh and the images stay the reproducible local path; the README says what a native run does not pin, and why arm is the default. - update_python_versions.py: a job whose runner is a dispatch input adds no tier label, so the arm default leaves the tiers' runners as they were. - workflows.rst: the table row, graph node and CI Reports thread. - The apt, AppArmor and Python-version tests allow benchmark.yml's new steps; test_benchmark_workflow_2125.py pins the workflow and runs its steps.
8e38aa6 to
63cb5cf
Compare
|
Nits applied at |
|
Coverage: 91.55% (unit tier, Python 3.15,
Per-file detail: the |
|
Ready to merge at
This closes #2125. Once it merges, a dispatch with |
make pylint,make mypy,make isort)make testpasses, and a test case covers the changeWhat is the purpose of your pull request?
ci— workflows or build toolingDescription of your pull request and other information
Closes #2125.
benchmark.ymlhas a weekly schedule (Wed 03:17 UTC, clear of every other cron) andworkflow_dispatchwith inputs forrunner,pythons,engines,roundsandrepeats.examples/benchmark/native.shdoes whatrun.shdoes, on the runner's own interpreters. Those are the engine tiers of.github/python-versions.toml, installed asengine-testsinstalls them. One job measures every version, so the per-version table stays one machine.run.shand theDockerfileare unchanged for local, pinned runs.discussionjob hasdiscussions: write, and the top-level scope iscontents: read. It poststable.rstto thebenchmark: weekly engine benchmark reportthread inCI reports, naming the runner image. Nothing pushes or edits the docs.ubuntu-latest, the only runner where every engine runs on every interpreter at the newest patch. Windows has no unattended Npcap, and macOS has older patches for 3.10–3.12. I'll measure the four candidates with a build-only comparison run and post the timings here.workflows.rstgets the At a Glance row, the graph node and the CI Reports sentence.examples/benchmark/README.mdgets an "On GitHub Actions" section.test_benchmark_workflow_2125.pygives 23 passed. Each pin fails when its part of the workflow is mutated. The 47 tests that read workflows orworkflows.rstgive 1134 passed. actionlint 1.7.12 is clean.timeout-minutes: 300is a placeholder; the Windows and macOS tshark installs; and interpreter discovery on Windows.