Skip to content

ci(benchmark): run the engine benchmark on Actions and post the report to Discussions (#2125) #2128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JarryShaw merged 1 commit into from
Oct 24, 2026
Merged

ci(benchmark): run the engine benchmark on Actions and post the report to Discussions (#2125) #2128

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
288 changes: 288 additions & 0 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,288 @@
name: Engine Benchmark

# #2125: times every extraction engine on every interpreter the engines are
# tested on, so the docs' Engine Support speed table no longer goes stale
# between hand runs (#2109). Natively, through examples/benchmark/native.sh
# rather than run.sh: a hosted runner is already a fresh VM per job, so
# containers would add a build per version and no isolation, and a native run
# makes macOS and Windows runners candidates too. One job measures every
# version, since the per-version table's absolute times compare only within
# one machine: by default `ubuntu-26.04-arm`, the fastest runner that
# measures every environment and engine. What a native run does not pin --
# the interpreter's patch release, tshark and libpcap -- is recorded in the
# tables; examples/benchmark/README.md's "On GitHub Actions" says more, and
# gives the runner figures.
#
# The `discussion` job posts the ratio table to this job's thread in the
# Discussions category REPORT_CATEGORY names (#2030); it alone holds a write
# scope, and a report it cannot post is a warning. Nothing here pushes, and the
# docs stay hand-updated from the report.
on:
schedule:
# Wednesdays 03:17 UTC: off the hour, where GitHub delays scheduled runs
# most, and midweek, clear of the Saturday crons (#1967), Monday's Python
# Versions and CI Timing, and Project Status at 00:37.
- cron: '17 3 * * 3'
workflow_dispatch:
inputs:
runner:
description: Runner to measure on; dispatch once per runner to compare them.
type: choice
default: ubuntu-26.04-arm
options:
- ubuntu-26.04-arm
- ubuntu-latest
- macos-latest
- windows-latest
pythons:
description: Comma-separated Python versions to measure (empty = every version the engines are tested on).
type: string
default: ''
engines:
description: Comma-separated engines, including default (empty = all).
type: string
default: ''
rounds:
description: Timed extractions per engine per pass (empty = benchmark.py's default).
type: string
default: ''
repeats:
description: Passes over the whole engine set (empty = benchmark.py's default).
type: string
default: ''

permissions:
contents: read

# One run per runner at a time; dispatches on different runners still run side
# by side, which is how two runners are compared.
concurrency:
group: benchmark-${{ inputs.runner || 'ubuntu-26.04-arm' }}
cancel-in-progress: false

# #2030: one Discussions category holds every report with no pull request to
# comment on, one thread per job.
env:
REPORT_CATEGORY: CI reports

jobs:
# #1660: the interpreters are the engine tiers of .github/python-versions.toml,
# as unit-tests.yml's `engine-tests` installs them.
python-versions:
name: Read the Python versions
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
engine-supported: ${{ steps.read.outputs.engine-supported }}
engine-ktlo: ${{ steps.read.outputs.engine-ktlo }}

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
sparse-checkout: |
.github/python-versions.toml
util/_actions.py
util/python_versions.py
sparse-checkout-cone-mode: false
persist-credentials: false

- name: Read .github/python-versions.toml
id: read
run: python3 util/python_versions.py --github-output

benchmark:
name: Benchmark on ${{ matrix.os }}
needs: [ python-versions ]
# One leg, on the dispatch's runner, or `ubuntu-26.04-arm` on the schedule,
# which has no inputs; spelt as the other matrix jobs spell their runner.
runs-on: ${{ matrix.os || 'ubuntu-26.04-arm' }}
# Unmeasured on a hosted runner; the first dispatch measures it. Locally a
# full matrix took about two hours (the README's "How long it takes"),
# nearly all of it a tshark per pyshark extraction, so this leaves room for
# a slower runner under the six-hour job limit.
timeout-minutes: 300
strategy:
matrix:
os: [ "${{ inputs.runner || 'ubuntu-26.04-arm' }}" ]
defaults:
run:
shell: bash
outputs:
status: ${{ steps.measure.outputs.status }}
env:
# Read by util/apt_packages.py install: libpcap's headers for pypcap's
# build, and tshark for pyshark.
APT_PACKAGES: libpcap-dev tshark
APT_SET: benchmark

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

# As in `engine-tests`: each interpreter is then reachable, and the KTLO
# one gets a step of its own so that losing it cannot fail the job.
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
if: ${{ needs.python-versions.outputs.engine-ktlo != '' }}
continue-on-error: true
with:
python-version: ${{ needs.python-versions.outputs.engine-ktlo }}

- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ needs.python-versions.outputs.engine-supported }}

# Without its system package an engine is a row with its reason, so a
# failed install warns rather than failing the job.
- name: Install libpcap headers and tshark
if: runner.os == 'Linux'
timeout-minutes: 10
run: |
/usr/bin/python3 util/apt_packages.py install \
|| echo '::warning title=System packages not installed::pypcap and pyshark may be reported unmeasured.'

# #2019: tshark's AppArmor profile on Ubuntu 25.04 and later; see
# unit-tests.yml's `engine-tests` step of the same name, whose body this is.
- name: Let tshark read the checkout past its AppArmor profile
if: runner.os == 'Linux'
run: |
set -eu
if [ -f /etc/apparmor.d/tshark ]; then
sudo mkdir -p /etc/apparmor.d/local
echo "file r ${GITHUB_WORKSPACE}/**," | sudo tee -a /etc/apparmor.d/local/tshark >/dev/null
sudo apparmor_parser -r /etc/apparmor.d/tshark
fi

# macOS ships libpcap; Homebrew's `wireshark` formula is the tshark CLI.
- name: Install tshark with Homebrew
if: runner.os == 'macOS'
timeout-minutes: 20
env:
HOMEBREW_NO_AUTO_UPDATE: '1'
HOMEBREW_NO_INSTALL_CLEANUP: '1'
run: |
brew install wireshark \
|| echo '::warning title=tshark not installed::pyshark will be reported unmeasured.'

# Reading a file needs no capture driver. Npcap's free edition cannot
# install unattended, so the pcap engines report the missing libpcap.
- name: Install tshark with Chocolatey
if: runner.os == 'Windows'
timeout-minutes: 15
run: |
if choco install wireshark --yes --no-progress; then
echo 'C:\Program Files\Wireshark' >> "$GITHUB_PATH"
else
echo '::warning title=tshark not installed::pyshark will be reported unmeasured.'
fi

# native.sh's exit status 3 is a table with gaps, each explained in it,
# so it warns; anything else but 0 is no table and fails.
- name: Run examples/benchmark/native.sh
id: measure
env:
INSTALLED: ${{ needs.python-versions.outputs.engine-ktlo }} ${{ needs.python-versions.outputs.engine-supported }}
PYTHONS: ${{ inputs.pythons }}
ENGINES: ${{ inputs.engines }}
ROUNDS: ${{ inputs.rounds }}
REPEATS: ${{ inputs.repeats }}
RUNNER_LABEL: ${{ matrix.os }}
run: |
set -uo pipefail
# `engine-supported` is a multi-line output, one version a line, and an
# input may say `3.11, 3.12`, so both are split on commas and whitespace.
read -ra installed <<< "$(echo "$INSTALLED" | tr -s ',[:space:]' ' ')"
read -ra wanted <<< "$(echo "${PYTHONS:-${installed[*]}}" | tr -s ',[:space:]' ' ')"
for version in "${wanted[@]}"; do
case " ${installed[*]} " in
*" $version "*) ;;
*) echo "::error title=Unknown Python version::'$version' is not among ${installed[*]}."; exit 2 ;;
esac
done
args=(--pythons "$(IFS=,; echo "${wanted[*]}")" --out out
--image "GitHub-hosted $RUNNER_LABEL, image ${ImageOS:-unknown} ${ImageVersion:-unknown}")
if [ -n "$ENGINES" ]; then args+=(--engines "$ENGINES"); fi
if [ -n "$ROUNDS" ]; then args+=(--rounds "$ROUNDS"); fi
if [ -n "$REPEATS" ]; then args+=(--repeats "$REPEATS"); fi
code=0
examples/benchmark/native.sh "${args[@]}" || code=$?
case "$code" in
0) echo 'status=success' >> "$GITHUB_OUTPUT" ;;
3) echo 'status=warning' >> "$GITHUB_OUTPUT"
echo '::warning title=Not every Python version measured in full::The tables explain each gap.' ;;
*) echo 'status=failure' >> "$GITHUB_OUTPUT"
exit "$code" ;;
esac

# The report names the runner, as the tables' Image row does, since a
# native run has no image digest to name.
- name: Write the report
if: ${{ always() && hashFiles('out/table.rst') != '' }}
env:
RUNNER_LABEL: ${{ matrix.os }}
run: |
set -eu
{
echo "## Engine benchmark on $RUNNER_LABEL"
echo
echo "Measured natively on $RUNNER_OS $RUNNER_ARCH, image ${ImageOS:-unknown} ${ImageVersion:-unknown},"
echo "so the absolute times compare only within this run. The docs' tables are updated by hand."
echo
echo '```rst'
cat out/table.rst
echo '```'
} > out/report.md
{
cat out/report.md
echo
echo '```rst'
cat out/table-versions.rst
echo '```'
} >> "$GITHUB_STEP_SUMMARY"

- name: Upload the results
if: ${{ always() }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: benchmark-out
path: out/
if-no-files-found: ignore
retention-days: 90

# #2030: posts the ratio table, with its absolute-time footnote, on
# `always()` like ci-timing.yml's, so a run with gaps is posted too.
# util/discussion_report.py writes a warning rather than failing, including
# for a run that wrote no report; `continue-on-error` covers the rest.
discussion:
name: Post the engine benchmark report
needs: [ benchmark ]
if: ${{ always() && needs.benchmark.result != 'skipped' }}
runs-on: ubuntu-latest
timeout-minutes: 10
continue-on-error: true
permissions:
contents: read
discussions: write

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Download the report
continue-on-error: true
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: benchmark-out
path: report

- name: Post the report to Discussions
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
STATUS: ${{ needs.benchmark.outputs.status || needs.benchmark.result }}
# `scheduled` on dispatch too: a manual run is a deliberate report.
run: |
python3 util/discussion_report.py --category "$REPORT_CATEGORY" \
--title 'benchmark: weekly engine benchmark report' --report report/report.md \
--mode scheduled --status "$STATUS"
29 changes: 23 additions & 6 deletions docs/source/contributing/workflows.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,13 @@ At a Glance
- Mon 05:41
- --
- ``workflow_dispatch``
* - Engine Benchmark
- :file:`benchmark.yml`
- --
- --
- Wed 03:17
- --
- ``workflow_dispatch``

.. note::

Expand Down Expand Up @@ -157,6 +164,7 @@ than starting a new one.
PR["pull_request: main"]
SAT["schedule: Saturday"]
MON["schedule: Monday"]
WED["schedule: Wednesday"]
DISPATCH["workflow_dispatch"]
DAILY["schedule: daily"]
ISSUES["issues"]
Expand All @@ -173,6 +181,7 @@ than starting a new one.
VU["Vendor Update<br/>cron-vendor.yml"]
PV["Python Versions<br/>cron-python-versions.yml"]
CT["CI Timing<br/>ci-timing.yml"]
BM["Engine Benchmark<br/>benchmark.yml"]
PS["Project Status<br/>project-status.yml"]
CR["Create Release<br/>create-release.yml"]

Expand Down Expand Up @@ -210,6 +219,9 @@ than starting a new one.
MON -->|07:23| CT
DISPATCH --> CT

WED -->|03:17| BM
DISPATCH --> BM

DISPATCH --> PS
DAILY -->|00:37| PS
ISSUES --> PS
Expand All @@ -218,7 +230,7 @@ than starting a new one.
TAG --> CR

classDef trig fill:none,stroke-dasharray:2 2
class PUSH,PR,SAT,MON,DISPATCH,DAILY,ISSUES,PRT,TAG trig
class PUSH,PR,SAT,MON,WED,DISPATCH,DAILY,ISSUES,PRT,TAG trig

.. mermaid::

Expand Down Expand Up @@ -1301,8 +1313,8 @@ to a script. Each script's docstring says what it does; each has tests under
- ``Post the report to Discussions`` in the ``discussion`` job of
``ci-timing.yml``, ``cron-vendor.yml``, ``cron-conda.yml``,
``cron-python-versions.yml``, ``lint.yml``, ``deploy-pages.yml``,
``codeql-analysis.yml``, ``coverage-comment.yml`` and ``ci-reports.yml``
(see `CI Reports`_).
``codeql-analysis.yml``, ``coverage-comment.yml``, ``ci-reports.yml`` and
``benchmark.yml`` (see `CI Reports`_).
- No.
* - :file:`util/check_suite_report.py`
- ``Write the run report`` in the ``discussion`` job of
Expand All @@ -1312,7 +1324,8 @@ to a script. Each script's docstring says what it does; each has tests under
* - :file:`util/apt_packages.py`
- The apt steps of ``unit-tests.yml``'s ``engine-tests`` and ``pypcap-parity``
(and so its KTLO twin): the plan, install and collect steps, as
``apt_packages.py plan``, ``install`` and ``collect``. The
``apt_packages.py plan``, ``install`` and ``collect``, and of
``benchmark.yml``'s ``benchmark``, as ``install`` alone. The
``actions/cache`` steps, the seed steps (``Seed apt's archive with this
engine's cached .deb files`` and ``Seed apt's archive with the cached .deb
files``) and the AppArmor step (`#2019
Expand Down Expand Up @@ -1469,8 +1482,12 @@ marker. How often depends on the script's ``--mode``:
counts as green.

``ci-timing.yml`` posts as ``scheduled`` to ``ci-timing: weekly CI timing
report``. ``cron-vendor.yml`` posts to ``cron-vendor: vendor constant update
report``, as ``push`` on its ``workflow_run`` and ``scheduled`` otherwise, and
report``, and ``benchmark.yml`` to ``benchmark: weekly engine benchmark report``:
the ratio table :file:`examples/benchmark/native.sh` writes, with its
absolute-time footnote and the runner it ran on, reporting ``warning`` where a
Python version was not measured in full (:issue:`2125`). ``cron-vendor.yml``
posts to ``cron-vendor: vendor constant update report``, as ``push`` on its
``workflow_run`` and ``scheduled`` otherwise, and
reports ``warning`` where ``vendor-update`` passed over a failed target or test
gate. A failed or cancelled ``head`` skips ``vendor-update``, so its
``discussion`` job runs on ``always()`` and then posts a line of its own with
Expand Down
Loading
Loading