Skip to content

docs: regenerate the engine benchmark with Python 3.15 measured #2109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JarryShaw merged 1 commit into from
Oct 23, 2026
Merged

docs: regenerate the engine benchmark with Python 3.15 measured #2109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 136 additions & 47 deletions docs/source/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,8 +122,8 @@ Engine Comparison
-----------------

Being a **comprehensive** packet processor costs speed: :mod:`pcapkit`'s own
engine takes about *0.2* milliseconds per packet, slower than most of the
third-party engines below.
engine is slower than every third-party engine below except `PyShark`_, as
`Test Results`_ measures.

:mod:`pcapkit` also supports alternative extraction engines for speed:
`Scapy`_, `DPKT`_, `PyShark`_, `PyPCAP`_, `pcap-ct`_ and `PyPCAPFile`_,
Expand Down Expand Up @@ -173,23 +173,19 @@ Engine Support by Python Version
--------------------------------

Which engines can run at all, by interpreter. Verified by installing each engine
and extracting a capture on 3.10 through 3.14; 3.15 remains inferred.

================== ======== ======== ======== =========== ======== ===========
Engine 3.10 3.11 3.12 3.13 3.14 3.15 [*]_
================== ======== ======== ======== =========== ======== ===========
``pcapkit`` yes yes yes yes yes yes
``dpkt`` yes yes yes yes yes yes
``scapy`` yes yes yes yes yes yes
``pcap_ct`` yes yes yes yes yes yes
``pypcap`` yes yes no no no no
``pypcapfile`` yes yes no no no no
``pyshark`` [*]_ yes yes yes yes no no
================== ======== ======== ======== =========== ======== ===========

.. [*] Inferred rather than measured: no 3.15 interpreter was available.
.. [*] |pyshark|_ also needs Wireshark's :program:`tshark`, which was absent here,
so only the interpreter half of each verdict in this row was verified.
and extracting a capture on 3.10 through 3.15.

================== ======== ======== ======== ======== ======== ========
Engine 3.10 3.11 3.12 3.13 3.14 3.15
================== ======== ======== ======== ======== ======== ========
``pcapkit`` yes yes yes yes yes yes
``dpkt`` yes yes yes yes yes yes
``scapy`` yes yes yes yes yes yes
``pcap_ct`` yes yes yes yes yes yes
``pypcap`` yes yes no no no no
``pypcapfile`` yes yes no no no no
``pyshark`` yes yes yes yes no no
================== ======== ======== ======== ======== ======== ========

``pypcap`` and ``pypcapfile`` stop at 3.11, and ``pyshark`` at 3.13, for the
reasons under `Engine prerequisites`_. **Python 3.11 is the last version on which
Expand All @@ -202,42 +198,135 @@ Test Environment

.. list-table::

* - Operating System
- macOS 26.6.2
* - Chip
- Apple M2 Pro
* - Memory
- 16 GB
* - Image (3.10)
- ``pcapkit-benchmark:py3.10 (sha256:6308e38915a2613714069a05800c1a14e4486ee260eddb5619727e82b971a004)``
* - Image (3.11)
- ``pcapkit-benchmark:py3.11 (sha256:1af87ef232d03c867d0c504d75586132994a41bc4f6c3da36db0d6dd37d22fa7)``
* - Image (3.12)
- ``pcapkit-benchmark:py3.12 (sha256:26a6605738070c5ff5670a12a41c88086745865096363e680577523e5957546a)``
* - Image (3.13)
- ``pcapkit-benchmark:py3.13 (sha256:584e631d430bd7abdd696492229dd4b0e13a0d92687dba8968c0910ef4727b67)``
* - Image (3.14)
- ``pcapkit-benchmark:py3.14 (sha256:6c09093cf744db366ad7f1e5fa00ccf8f6ba9e167c298acc52e87acf393d2fa3)``
* - Image (3.15)
- ``pcapkit-benchmark:py3.15 (sha256:78d25409346c3c97f1df809a69cd021418c6146ba2c096ff00e7d97530addd53)``
* - ``pcapkit`` revision
- ``cfabfb2d3``
* - Python
- CPython 3.10.21, 3.11.16, 3.12.14, 3.13.15, 3.14.7, 3.15.1
* - Architecture
- ``x86_64``
* - Capture
- ``in.pcap`` -- 605 bytes, SHA-256 ``7aabadca49b329c6...``
* - Packets per extraction
- 6
* - Iterations
- 1000 timed extractions per engine per pass, the first discarded as a warm-up
* - Passes
- 3 over the whole engine set, in 8 environments: ``3.10-pcap_ct``, ``3.10-pypcap``, ``3.11-pcap_ct``, ``3.11-pypcap``, ``3.12-pcap_ct``, ``3.13-pcap_ct``, ``3.14-pcap_ct``, ``3.15-pcap_ct``
* - tshark
- ``TShark (Wireshark) 4.0.17 (Git v4.0.17 packaged as 4.0.17-0+deb12u3).``
* - libpcap
- ``libpcap version 1.10.3 (with TPACKET_V3)``

Resolved package versions:

============== ============ =========== ============ =========== ============ ============ ============ ============
Package 3.10-pcap_ct 3.10-pypcap 3.11-pcap_ct 3.11-pypcap 3.12-pcap_ct 3.13-pcap_ct 3.14-pcap_ct 3.15-pcap_ct
============== ============ =========== ============ =========== ============ ============ ============ ============
``dpkt`` 1.9.8 1.9.8 1.9.8 1.9.8 1.9.8 1.9.8 1.9.8 1.9.8
``libpcap`` 1.11.0b29 *absent* 1.11.0b29 *absent* 1.11.0b29 1.11.0b29 1.11.0b29 1.11.0b29
``lxml`` 6.1.3 6.1.3 6.1.3 6.1.3 6.1.3 6.1.3 6.1.3 6.1.3
``pcap-ct`` 1.3.0b3 *absent* 1.3.0b3 *absent* 1.3.0b3 1.3.0b3 1.3.0b3 1.3.0b3
``pypcap`` *absent* 1.3.0 *absent* 1.3.0 *absent* *absent* *absent* *absent*
``pypcapfile`` 0.12.0 0.12.0 0.12.0 0.12.0 0.12.0 0.12.0 0.12.0 0.12.0
``pypcapkit`` 1.5.0b14 1.5.0b14 1.5.0b14 1.5.0b14 1.5.0b14 1.5.0b14 1.5.0b14 1.5.0b14
``pyshark`` 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6
``scapy`` 2.8.0 2.8.0 2.8.0 2.8.0 2.8.0 2.8.0 2.8.0 2.8.0
============== ============ =========== ============ =========== ============ ============ ============ ============

Test Results
------------

Measured with ``examples/legacy_smoke/test_time.py``: 1,000 timed
extractions of ``examples/captures/in.pcap`` per engine and Python version. The
first extraction is discarded as a warm-up. Values are milliseconds per packet.

=================== ======= ======= ======= ======= =======
Engine 3.10 3.11 3.12 3.13 3.14
=================== ======= ======= ======= ======= =======
``pypcapfile`` [1]_ 0.0133 0.0093 -- -- --
``dpkt`` 0.0163 0.0113 0.0118 0.0117 0.0125
``pypcap`` [2]_ 0.0289 0.0212 -- -- --
``pcap_ct`` 0.0364 0.0320 0.0366 0.0322 0.0396
``scapy`` 0.1082 0.0822 0.0877 0.0832 0.0967
``pcapkit`` 0.2342 0.1906 0.1985 0.1920 0.2392
``pyshark`` [3]_ 18.8158 18.7852 19.2694 20.3751 --
=================== ======= ======= ======= ======= =======
Measured with ``examples/benchmark/run.sh``: 1000 timed
extractions of ``examples/captures/in.pcap`` per engine and Python version.
The first extraction is discarded as a warm-up. Values are milliseconds per
packet, the median of 3 passes.

All columns come from one run on one machine, differing in the interpreter
and nothing else, so they may be compared with each other. They may not be
compared with figures from another machine: absolute times are a property of
the host as much as of the engine.

================ ======= ======= ======= ======= ======= =======
Engine 3.10 3.11 3.12 3.13 3.14 3.15
================ ======= ======= ======= ======= ======= =======
``pypcapfile`` † 0.0320 0.0259 -- -- -- --
``dpkt`` 0.0583 0.0465 0.0503 0.0533 0.0530 0.0523
``pypcap`` † 0.1018 0.0890 -- -- -- --
``pcap_ct`` 0.1134 0.0988 0.1149 0.1167 0.1178 0.2702
``scapy`` 0.1896 0.1607 0.1882 0.1813 0.1816 0.1776
``pcapkit`` 0.5204 0.4301 0.4869 0.4949 0.4887 0.4814
``pyshark`` † 30.9862 31.0405 30.7182 31.4615 -- --
================ ======= ======= ======= ======= ======= =======

The unavailable cells were attempted. They are not zeroes and must not be
compared with a measured row.

.. [1] `PyPCAPFile`_ 0.12.0 imports :mod:`imp`, which Python removed in 3.12.

.. [2] `PyPCAP`_ 1.3.0 does not build on Python 3.12+: its pre-generated
:file:`pcap.c` does not compile against that C API.

.. [3] `PyShark`_ 0.6 cannot create the implicit event loop it expects on Python
3.14; it ran on 3.10--3.13.
``†`` not measured, or not measured in full, for the
reason recorded at the time:

* ``pypcapfile`` on 3.12 -- pypcapfile does not support Python 3.12; its linklayer module imports \`imp\`, removed in Python 3.12
* ``pypcapfile`` on 3.13 -- pypcapfile does not support Python 3.13; its linklayer module imports \`imp\`, removed in Python 3.12
* ``pypcapfile`` on 3.14 -- pypcapfile does not support Python 3.14; its linklayer module imports \`imp\`, removed in Python 3.12
* ``pypcapfile`` on 3.15 -- pypcapfile does not support Python 3.15; its linklayer module imports \`imp\`, removed in Python 3.12
* ``pypcap`` on 3.12, 3.13, 3.14, 3.15 -- pypcap is not installable on this interpreter, so it was not built into this image: pypcap 1.3.0 ships a pcap.c pre-generated by Cython 0.29.32, which does not compile against the Python 3.12+ C API. The engine is unsupported here rather than broken here.
* ``pyshark`` on 3.14 -- pyshark does not support Python 3.14; it builds its event loop with \`asyncio.get\_event\_loop\_policy().get\_event\_loop()\`, which raises RuntimeError from Python 3.14 when no current event loop exists
* ``pyshark`` on 3.15 -- pyshark does not support Python 3.15; it builds its event loop with \`asyncio.get\_event\_loop\_policy().get\_event\_loop()\`, which raises RuntimeError from Python 3.14 when no current event loop exists

Test Results (Relative)
-----------------------

Times relative to ``pcapkit``, which is 1 by definition; lower is
faster. Each ratio is taken against the baseline measured in the same
environment on the same pass, so the machine cancels out. "Observed range" is
the smallest and largest ratio actually seen across the passes.

Ratios are pooled across every environment in the run, which means across
every Python version measured as well as both ``pcap`` providers. The
observed range therefore includes any disagreement between interpreters
about an engine; the per-version table is where that is broken out.

============== ============= ============== =======
Engine Relative time Observed range Samples
============== ============= ============== =======
``pypcapfile`` 0.0613 0.0597-0.0623 12
``dpkt`` 0.108 0.101-0.139 24
``pypcap`` 0.203 0.198-0.209 6
``pcap_ct`` ‡ 0.236 0.215-0.566 18
``scapy`` ‡ 0.372 0.355-0.388 24
``pcapkit`` 1 *baseline* 24
``pyshark`` ¶ 63.6 58.7-72.9 18
============== ============= ============== =======

The single absolute figure, for converting the ratios back: ``pcapkit``
itself ran at **0.4879 ms per packet** on this run
(0.425-0.522 across 24 passes). Absolute times are a property of the
machine as much as of the engine, which is why every other figure here is a
ratio.

Run-to-run spread: the median row varied by 8.9% of its ratio across
the passes and the widest by 149%. A gap between two rows that is
smaller than that is noise, not a result.

``‡`` this run does not separate these rows: their observed
ranges overlap, so the order their medians suggest is not established.

``¶`` measured, but on less evidence than the rows beside it.
What was collected is real; what was lost is named here rather than left to
be inferred from the pass count:

* ``pyshark`` -- 2 individual extraction(s) failed and were discarded

Installation
============
Expand Down
3 changes: 1 addition & 2 deletions docs/source/pcapkit/foundation/engines/3rdparty.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,7 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
3.10 and 3.11 return a loop silently, 3.12 returns one with a
:exc:`DeprecationWarning`, and **3.14 raises**
``RuntimeError: There is no current event loop in thread 'MainThread'``. Python
3.13 is unchecked and expected to work, being on the deprecated-but-functional
side of that change.
3.13 works, being on the deprecated-but-functional side of that change.

**Wireshark's** :program:`tshark` **binary.** |pyshark|_ shells out to it and
parses nothing itself. It need not be on :envvar:`PATH`: |pyshark|_ consults
Expand Down
34 changes: 10 additions & 24 deletions examples/benchmark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,7 @@ pair:
| Python | environments | why |
| --- | --- | --- |
| 3.10, 3.11 | `-pypcap` and `-pcap_ct` | both `pcap` providers install, and they are mutually exclusive |
| 3.12, 3.13, 3.14 | `-pcap_ct` only | `pypcap` cannot be installed at all, so a second virtualenv would compile for minutes to produce nothing |
| 3.15 | opt-in, `-pcap_ct` only | no released image; see below |
| 3.12 to 3.15 | `-pcap_ct` only | `pypcap` cannot be installed at all, so a second virtualenv would compile for minutes to produce nothing |

A version named with `--pythons` that is not in the file is a typo rather than a
column.
Expand Down Expand Up @@ -169,7 +168,7 @@ them silently narrows the table:
mechanism `/opt/install-failures/` uses for `pypcap`, one level up.
- the reporting pass turns each of those files into a column of `--` **with the
reason attached** -- in the per-version table's notes, and in the `Test
Environment` block as a `Python 3.15 (not measured)` row, so neither snippet
Environment` block as a `Python <version> (not measured)` row, so neither snippet
reads as a complete matrix when it is not.
- only a run in which *every* version failed exits without a table, since then there
is genuinely nothing to report.
Expand Down Expand Up @@ -358,7 +357,7 @@ would undo that, and would leak machine identity into a public README.
- **measure: 1.5-2 hours.** 1,000 rounds x 3 passes in every environment, and **almost
all of it is** `pyshark`, which spawns a `tshark` process per extraction and is
two to three orders of magnitude slower than anything else in the table. It runs in
every environment but 3.14's, where it cannot run at all.
every environment but 3.14's and 3.15's, where it cannot run at all.

Ways to cut it, in descending order of how much they save:

Expand All @@ -384,7 +383,7 @@ so.
--repeats N passes over the whole engine set (default 3)
--engines LIST comma-separated; must include 'default', which every ratio uses
--pythons LIST comma-separated Python versions (default: every 'default'-tier
row of python-images.txt, i.e. 3.10 through 3.14)
row of python-images.txt, i.e. 3.10 through 3.15)
--quick --rounds 50 --repeats 2
--platform P docker platform; defaults to this machine's architecture
--out DIR where to write results (default examples/benchmark/out)
Expand All @@ -395,26 +394,13 @@ From the repository root: `make bench` runs `run.sh` with no arguments,
`make bench-quick` runs a reduced smoke matrix, and `make bench-test` runs the
self-tests below.

## Python 3.15
## A new Python version

Opt-in, and allowed to fail -- the benchmark's own choice, not CI's: the repository's CI
runs 3.15 as a blocking, supported leg (#1633). There is no released
`python:3.15-slim-bookworm`; the
pins file names a `3.15-rc` digest, which freezes one particular release candidate
and will name an older one as time passes. Measure it with:

```shell
examples/benchmark/run.sh --pythons 3.10,3.11,3.12,3.13,3.14,3.15
```

If that image cannot be built or the run fails in it, the result is a `3.15` column
of `--` with the reason attached, not a missing column and not an aborted run.

As it happens the pinned rc **does** build and measure: every pinned requirement
installs on it, and `default`, `dpkt`, `scapy` and `pcap_ct` all produced
figures. `pypcap`, `pypcapfile` and `pyshark` come back unmeasured with the same
reasons they give on 3.14. That is a fact about one release candidate on one day
rather than a promise about 3.15, which is why the row stays opt-in.
A version joins the default matrix once its `python:<version>-slim-bookworm` image
is released; the pins file says how to add the row. A release candidate can be
measured before then as an `opt-in` row, named with `--pythons`. If its image cannot
be built or the run fails in it, the result is a column of `--` with the reason
attached, not a missing column and not an aborted run.

## Self-tests

Expand Down
19 changes: 8 additions & 11 deletions examples/benchmark/python-images.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,22 +54,19 @@
# can say the engine is unsupported on the interpreter rather than that it failed
# here.
#
# ## 3.15
# ## Adding a version
#
# Opt-in, and allowed to fail. There is no released `python:3.15-slim-bookworm`;
# `3.15-rc-slim-bookworm` is a release candidate and the tag moves with every rc, so
# the digest below freezes one particular candidate and will name an older rc as
# time passes -- bump it deliberately, or drop the row when 3.15 ships. Opt-in is
# this file's own tier, not CI's: the repository's CI runs 3.15 as a blocking,
# supported leg (#1633). `run.sh --pythons ...,3.15` measures it, a plain
# `run.sh` does not, and a
# version that cannot be built is reported as a missing column with the reason
# rather than dropped.
# A new interpreter gets a row once `python:<version>-slim-bookworm` is a released
# image, pinned by the digest of that day's index. A release candidate stays out of
# the default tier: its tag moves with every rc, so a pinned rc names an older
# candidate as time passes. `opt-in` exists for such a row -- `run.sh --pythons`
# measures it, a plain `run.sh` does not -- and a version that cannot be built is
# reported as a missing column with the reason rather than dropped.
#
# version base image pypcap tier
3.10 python:3.10-slim-bookworm@sha256:68d914ec641a0b69267ce65184d000a2bc3a9ee2590ab702b82250ab2385735a pypcap default
3.11 python:3.11-slim-bookworm@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84 pypcap default
3.12 python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254 no-pypcap default
3.13 python:3.13-slim-bookworm@sha256:ed86c82274b3c69b52fb5820f358f0bd7df0b603332063cb5c6e32bd220c3e6e no-pypcap default
3.14 python:3.14-slim-bookworm@sha256:9ab8d9c8514b44f90cf0029dd42fdd7e9e211e639c8b995304cc04568dee900f no-pypcap default
3.15 python:3.15-rc-slim-bookworm@sha256:984bee69a9c4abf834e9c458a0d81ef10332a3c9d3a8666e566f46cc23935599 no-pypcap opt-in
3.15 python:3.15-slim-bookworm@sha256:575f8bc12ec26542e0678bc6b60e385218248019d433157ec327264a78c3a853 no-pypcap default
Loading
Loading