Skip to content

ci(benchmark): run the engine benchmark on GitHub Actions and post the report to Discussions #2125

Closed

Description

@JarryShaw

Is your feature request related to a problem? Please describe.
The engine benchmark (examples/benchmark/run.sh) only runs by hand on a local machine. So the docs' Engine Support speed table goes stale between releases. #2109 had to re-measure it once 3.15 shipped.

Describe the solution you'd like
A benchmark.yml workflow:

  • Triggers: a weekly schedule, at an off-hour slot clear of the Saturday crons and of Monday's CI Timing and Python Versions, plus workflow_dispatch with inputs for the Python versions and the rounds.
  • Steps: run examples/benchmark/run.sh on ubuntu-latest, which is native x86_64 with Docker, then upload out/ as an artifact.
  • Report: a discussion job, modelled on ci-timing.yml's, the only job with discussions: write. It posts the ratio table (table.rst), with its absolute-time footnote, through util/discussion_report.py to a benchmark thread in CI reports, naming the runner image.
  • Docs: stay hand-updated from the report; the workflow never pushes.

Describe alternatives you've considered
Running per push or PR: the runner pool is already saturated. Auto-committing the docs tables: that bypasses review, and the absolute ms come from shared, unknown hardware. The within-pass ratios cancel most runner noise; the absolute numbers do not.

Additional context
Runtime of a full matrix on a hosted runner is unmeasured. The first dispatch measures it, and that sets timeout-minutes and the cadence.

Activity

  1. added
    enhancementIssues requesting a new capability (set by the feature request template)
    ciPull requests that change CI or workflow configuration (ci: subject prefix)
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Oct 24, 2026
  2. removed
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Oct 24, 2026
  3. added this to the PyPCAPKit v1.5 milestone on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciPull requests that change CI or workflow configuration (ci: subject prefix)enhancementIssues requesting a new capability (set by the feature request template)

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions