Skip to content

ci(benchmark): run the engine benchmark on Actions and post the report to Discussions (#2125) - #2128

Merged
JarryShaw merged 1 commit into
mainfrom
feat/2125-benchmark-workflow
Oct 24, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
feat/2125-benchmark-workflow

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • ci — workflows or build tooling

Description of your pull request and other information

Closes #2125.

  • Workflow: benchmark.yml has a weekly schedule (Wed 03:17 UTC, clear of every other cron) and workflow_dispatch with inputs for runner, pythons, engines, rounds and repeats.
  • Native, no Docker: following the follow-up on ci(benchmark): run the engine benchmark on GitHub Actions and post the report to Discussions #2125, a new examples/benchmark/native.sh does what run.sh does, on the runner's own interpreters. Those are the engine tiers of .github/python-versions.toml, installed as engine-tests installs them. One job measures every version, so the per-version table stays one machine. run.sh and the Dockerfile are unchanged for local, pinned runs.
  • Report: only the discussion job has discussions: write, and the top-level scope is contents: read. It posts table.rst to the benchmark: weekly engine benchmark report thread in CI reports, naming the runner image. Nothing pushes or edits the docs.
  • Runner: the default is ubuntu-latest, the only runner where every engine runs on every interpreter at the newest patch. Windows has no unattended Npcap, and macOS has older patches for 3.10–3.12. I'll measure the four candidates with a build-only comparison run and post the timings here.
  • Docs: workflows.rst gets the At a Glance row, the graph node and the CI Reports sentence. examples/benchmark/README.md gets an "On GitHub Actions" section.
  • Tests: the new test_benchmark_workflow_2125.py gives 23 passed. Each pin fails when its part of the workflow is mutated. The 47 tests that read workflows or workflows.rst give 1134 passed. actionlint 1.7.12 is clean.
  • Unverified until a hosted run: the runtime, so timeout-minutes: 300 is a placeholder; the Windows and macOS tshark installs; and interpreter discovery on Windows.

@JarryShaw JarryShaw added enhancement Issues requesting a new capability (set by the feature request template) ci Pull requests that change CI or workflow configuration (ci: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

NEEDS CHANGES at 5663e9267 (Sonnet review).

  • Blocker: util/python_versions.py writes engine-supported as a multiline output. The Run examples/benchmark/native.sh step joins it with spaces, so its version check rejects 3.11–3.15 with exit 2 on every scheduled run. The test fed a space-separated list, so it missed this. I confirmed the multiline output locally and cancelled the runner-comparison run, which would have hit it.
  • Smaller: native.sh should refuse a version missing from python-images.txt and drop duplicates, as run.sh does. Re-wrap the spliced workflows.rst CI Reports line, and use :issue: for any reference this PR adds in .rst.
  • Passes: permissions and inputs through env:, the discussion status/mode mapping, and the native.sh exit codes and parity with run.sh.

A revision is in progress; I'll re-run the comparison on it.

@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 24, 2026
@JarryShaw
JarryShaw force-pushed the feat/2125-benchmark-workflow branch from 5663e92 to abb234d Compare October 24, 2026 00:58
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at abb234d11 (Sonnet delta review).

  • The blocker: the Run examples/benchmark/native.sh step was run on the real util/python_versions.py --github-output, rendered as GitHub renders it. A scheduled run now passes --pythons 3.10,3.11,3.12,3.13,3.14,3.15. A dispatch of ' 3.12, 3.11 ,3.13' becomes 3.12,3.11,3.13, and 3.1 / 3.9 exit 2. Four tests fail on the old step.
  • native.sh: it refuses a version missing from python-images.txt before writing out/, and drops repeats.
  • workflows.rst: the paragraph is re-wrapped, and the added reference is :issue:`2125` .
  • Pins-row guard: a new engine-tier version now needs a python-images.txt row, so the Python Versions bot's PR goes red until someone adds it. That is intended: otherwise the next weekly run would exit 2. The bot cannot write the row itself, because a row needs a resolved image digest.
  • Tests: the four files give 101 passed.

Runner comparison: run 40047570595 on verify/bench-2125. I'll post its timings here.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet review: good-to-go Cross-review at the current head says ready; CI state is separate labels Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Runner comparison (40047570595). Every engine except pyshark, on 3.10–3.15, one leg per runner:

Runner pcapkit ms/packet Measured Measurement / job
ubuntu-latest 0.757 all 8 envs, all engines 188 s / 7m16
ubuntu-26.04-arm 0.580 all 8 envs, all engines 153 s / 7m19
macos-latest 0.339 6 of 8: no pcap_ct (dlopen(libc.so.6)), no pypcap; 3.10–3.12 on older patches; spread up to 121% 67 s / 4m07
windows-latest 0.779 6 of 8: no pcap_ct (no Npcap), no pypcap; 3.10–3.12 on older patches 145 s / 7m46
  • Recommendation: switch the default to ubuntu-26.04-arm. It is about 24% faster than x64 and, with ubuntu-latest, the only runner that fills every cell. macOS is faster still, but it cannot measure the pcap engines and is the noisiest.
  • Ratios agree within an OS family: the two Ubuntu runners match within range. Windows' pypcapfile ratio is about 44% above Ubuntu's, so the README's "machine-independent" holds per OS rather than across operating systems.
  • A bug found along the way: on macOS and Windows, preflight("pypcap") in benchmark.py re-raises the OSError from the unusable pcap module, killing the whole 3.10 and 3.11 pcap_ct environments instead of reporting the engine as unmeasured. I'll file it separately.

@JarryShaw
JarryShaw force-pushed the feat/2125-benchmark-workflow branch from 1a4bffc to 8e38aa6 Compare October 24, 2026 02:22
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at 8e38aa653 (Sonnet delta review), with documentation nits being applied.

  • The CI failure is fixed. test_benchmark_workflow_2125.py carries __tier__ = 'LATEST', and its sys.version_info branch is now try: import tomllib. The tier guard passes, and the five relevant test files give 236 passed.
  • The default runner is ubuntu-26.04-arm in all five places: dispatch default, options, concurrency group, runs-on fallback and matrix default. python-versions and discussion stay on ubuntu-latest.
  • util/update_python_versions.py workflow_labels now skips a job whose os axis reads inputs.. Only benchmark.yml matches. The tiers' default, supported and latest labels come out identical, and default is back to ['ubuntu-latest']. A fixed arm axis would still count. The rule is a heuristic, so the docstring will state its constraint: a gating job must not take its runner from an input.
  • Correction to my comparison above: 0.580 vs 0.757 ms/packet is about 23% less time, not 24%. And ubuntu-26.04-arm is the fastest runner that measures every engine: macOS was faster but incomplete. Both wordings are being fixed in the README and the workflow comment.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 24, 2026
…t to Discussions (#2125)

- benchmark.yml: weekly (Wed 03:17 UTC) and on dispatch, with the runner,
  Python versions, engines, rounds and repeats as inputs. One job, on
  ubuntu-26.04-arm by default, installs the engine tiers from
  python-versions.toml and runs native.sh; `discussion`, the only job with
  `discussions: write`, posts the ratio table to `benchmark: weekly engine
  benchmark report` in CI reports. Nothing pushes; the docs stay hand-updated.
- examples/benchmark/native.sh: run.sh without Docker, on the interpreters
  already installed, with run.sh's virtualenvs, version checks, gap notes and
  exit statuses. run.sh and the images stay the reproducible local path; the
  README says what a native run does not pin, and why arm is the default.
- update_python_versions.py: a job whose runner is a dispatch input adds no
  tier label, so the arm default leaves the tiers' runners as they were.
- workflows.rst: the table row, graph node and CI Reports thread.
- The apt, AppArmor and Python-version tests allow benchmark.yml's new steps;
  test_benchmark_workflow_2125.py pins the workflow and runs its steps.
@JarryShaw
JarryShaw force-pushed the feat/2125-benchmark-workflow branch from 8e38aa6 to 63cb5cf Compare October 24, 2026 02:29
@JarryShaw

Copy link
Copy Markdown
Owner Author

Nits applied at 63cb5cfc2, prose only: the README now reads "about 23% less time per packet" and "the fastest runner that measures every environment and engine". The benchmark.yml header comment and a test comment match it, and the workflow_labels docstring states that a gating job must not take its runner from an input. The range-diff against 8e38aa653 shows nothing else; the GOOD TO GO verdict carries over.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage: 91.55% (unit tier, Python 3.15, 63cb5cfc2, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19599 880 2618 866 92.12%
pcapkit/corekit 2589 75 816 37 96.48%
pcapkit/dumpkit 282 3 102 3 98.44%
pcapkit/foundation 2969 23 1036 25 98.80%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17418 219 4574 185 98.14%
pcapkit/toolkit 869 36 298 10 95.20%
pcapkit/utilities 498 7 144 5 98.13%
pcapkit/vendor 4432 1988 1018 167 51.87%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Ready to merge at 63cb5cfc2.

  • Review: GOOD TO GO, with the Sonnet round-3 delta review's prose nits applied.
  • CI: every blocking check passed (Required checks passed SUCCESS), and the 3.8–3.10 KTLO legs carry no warnings.
  • Ordering: the tests/project plain-unittest ordering leg passed on a scratch merge into main 8fdc2fb33, at 2225 tests with 0 failures.
  • For you to weigh: util/update_python_versions.py workflow_labels now skips a job whose runner comes from a dispatch input, which only benchmark.yml has. The tiers' labels are unchanged, and the docstring states the constraint.

This closes #2125. Once it merges, a dispatch with runner=ubuntu-26.04-arm gives the first real report and measures the full runtime, pyshark included.

@JarryShaw
JarryShaw merged commit c264485 into main Oct 24, 2026
60 checks passed
@JarryShaw
JarryShaw deleted the feat/2125-benchmark-workflow branch October 24, 2026 04:30
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 24, 2026
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 24, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Pull requests that change CI or workflow configuration (ci: subject prefix) enhancement Issues requesting a new capability (set by the feature request template)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

ci(benchmark): run the engine benchmark on GitHub Actions and post the report to Discussions

1 participant