Skip to content

ci(benchmark): emit reST footnotes and the engines page in the report, and size the timeout from the first run - #2142

Merged
JarryShaw merged 1 commit into
mainfrom
ci/benchmark-report-footnotes
Oct 24, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
ci/benchmark-report-footnotes

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • ci — workflows or build tooling

Description of your pull request and other information

Follow-up to #2128 and the new docs/source/engines.rst page (f9de043d6, d7aa41535), so a benchmark report pastes onto the page without hand edits.

  • examples/benchmark/report.py footnotes: it emits reST footnotes instead of literal † / ‡ / markers and hand-written legends. Each noted row or column gets [*]_, overlapping rows share a labelled [#overlap]_, and the bodies follow each table in reference order. A gap with no recorded reason now gets an explicit "no reason recorded" note instead of a dangling marker.
  • report.py attribution: the "Measured with" line names the script that actually ran. The new --runner / --workflow options are fed by native.sh and, through entrypoint.sh, by run.sh. Every index.rst reference becomes engines.rst, in the script and in examples/benchmark/README.md.
  • benchmark.yml: timeout-minutes drops from 300 to 90. The first full run, 40062653380 on ubuntu-26.04-arm with every engine, took about 38 minutes.
  • Tests: TestFootnotes in examples/benchmark/test_harness.py parses the output with docutils, so each reference must reach its own row's note. Against main, 20 tests fail. A new test pins the timeout.
  • Check: run 40062653380's output, spliced into engines.rst, changes no words. Only two tables' first-column padding differs.

…, and size the timeout from the first run

- report.py: rows and columns that need a note carry an auto-symbol
  footnote reference, [*]_, with the reason as the body directly after
  its table; overlapping rows share one labelled [#overlap]_ note. The
  literal markers and hand-written legends are gone from the RST, and
  the comment that justified them is corrected: docutils numbers
  footnotes page-wide, so nothing on the page collides. A gap no
  environment explained still gets a body.
- report.py: the per-version lead names the script that ran (--runner)
  and, on Actions, the workflow (--workflow); native.sh passes both and
  run.sh passes its name through BENCH_RUNNER to entrypoint.sh. Every
  docs path now says docs/source/engines.rst.
- README.md: points at engines.rst and describes the footnote output.
- benchmark.yml: timeout-minutes 300 -> 90, from run 40062653380's
  38-minute job on ubuntu-26.04-arm.

Run 40062653380's documents now render the page's Test Results sections
word for word. Harness and workflow tests pass.
@JarryShaw JarryShaw added docs Pull requests that change documentation only (docs: subject prefix) ci Pull requests that change CI or workflow configuration (ci: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 24, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at 208047c7f (Sonnet review).

  • Footnotes: a docutils render of run 40062653380's output, spliced into engines.rst, has no footnote, table or indentation errors. Each [*]_ reaches the body after its own table, alongside the page's existing footnotes. The shared [#overlap]_ resolves several references to one body, which the tests pin.
  • Gaps and attribution: a gap with no recorded reason gets an honest "no reason recorded" note, not a dangling reference. The --runner / --workflow plumbing (native.sh; run.sh → entrypoint.sh via BENCH_RUNNER) is safe under set -euo pipefail and on bash 3.2.
  • Timeout: 90 min is about 2.3× the measured 38 min on arm, and about 50 min is estimated for ubuntu-latest (1.31× slower per packet in run 40047570595). Pyshark has only been measured on Linux, so a Windows or macOS dispatch with pyshark is the one case that could run long.
  • Tests: 171 passed. With main's examples/benchmark/, 20 tests fail.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage: 91.55% (unit tier, Python 3.15, 208047c7f, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19599 880 2618 866 92.12%
pcapkit/corekit 2589 75 816 37 96.48%
pcapkit/dumpkit 282 3 102 3 98.44%
pcapkit/foundation 2971 23 1036 25 98.80%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17418 219 4574 185 98.14%
pcapkit/toolkit 869 36 298 10 95.20%
pcapkit/utilities 498 7 144 5 98.13%
pcapkit/vendor 4432 1988 1018 167 51.87%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Ready to merge at 208047c7f.

  • Review: the Sonnet review is GOOD TO GO.
  • CI: every blocking check passed (Required checks passed SUCCESS). The 3.8–3.10 KTLO legs carry no warnings.
  • Ordering: the tests/project plain-unittest ordering leg passed on a scratch merge into main d7aa41535: 2233 tests, 0 failures.
  • After merge: the next Engine Benchmark report pastes into docs/source/engines.rst without hand edits, apart from that page's own intro paragraph and Run row. A Windows or macOS dispatch with pyshark has never been timed against the new 90-minute limit.

@JarryShaw
JarryShaw merged commit da7f03a into main Oct 24, 2026
66 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 24, 2026
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 24, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Pull requests that change CI or workflow configuration (ci: subject prefix) docs Pull requests that change documentation only (docs: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant