Skip to content

perf(foundation): find pcap distributions without parsing METADATA (#2110) - #2112

Merged
JarryShaw merged 1 commit into
mainfrom
perf/2110-pcap-dist-lookup
Oct 23, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
perf/2110-pcap-dist-lookup

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • perf — performance improvements

Description of your pull request and other information

Closes #2110. pcap_ct extraction was about 2.4× slower on 3.15. The cause is pcapkit's installed_distributions() in _pcap_backend.py, which runs three times per extraction. On 3.15, importlib.metadata.distribution() → Distribution.from_name wraps its search in _prefer_valid, which parses every candidate's METADATA (importlib_metadata #489). One call costs 115 µs on 3.14 and 397 µs on 3.15. libpcap 1.10.3, pcap-ct 1.3.0b3 and a ctypes call cost the same on both versions.

  • Fix: next(iter(importlib.metadata.distributions(name=name)), None) matches on the dist-info directory and reads no file. It finds a package in exactly the cases the old call did; checked on 3.8.20 and 3.15.1 for pcap-ct, pcap_ct, pypcap and pip. The probe() docstring no longer claims a repeat call costs only a sys.modules lookup.

  • Measured (Docker images from docs: regenerate the engine benchmark with Python 3.15 measured #2109's harness, 6-packet in.pcap):

    Python Before (ms/packet) After (ms/packet)
    3.15.1 0.2617 0.1066
    3.14.7 0.1097 0.1138 (within noise)
  • Test: puts an empty pcap_ct dist-info on sys.path and makes any METADATA read raise. Without the fix it fails on both 3.14 and 3.15. The backend, pcap_ct and pypcap engine tests pass on both.

Follow-up: the Extractor still probes three times per extraction (_select_engine calls unsupported_reason twice, plus the constructor). Tracked separately.

…2110)

installed_distributions() runs three times per pcap_ct extraction. On
Python 3.15 importlib.metadata.distribution() parses every candidate's
METADATA (Distribution._prefer_valid) to rank valid dists first, which
took 115 -> 397 us per call and made pcap_ct 0.110 -> 0.261 ms/packet.

Ask importlib.metadata.distributions(name=...) for the first match
instead: it finds a distribution exactly when distribution() does
(checked on 3.8 and 3.15) and reads no file. pcap_ct on 3.15 is back to
0.107 ms/packet; 3.14 is unchanged.

The new test puts an empty pcap_ct dist-info on sys.path, makes any
METADATA read raise, and asserts pcap-ct is still found.
@JarryShaw JarryShaw added perf Pull requests that improve performance (perf: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 23, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at b90787270 (Sonnet review).

  • Semantics: read in CPython source for 3.8, 3.10 and 3.15. distribution(name) and distributions(name=...) share one lookup and one name normalisation (lowercase, -→_). On 3.15 they differ only in _prefer_valid, which reorders candidates but still returns a dist without METADATA. So found and not-found results are identical, and egg-info and editable installs take the same path. The caller only records the name.
  • Attribution: 3.15's _prefer_valid reads .metadata on every candidate. Its docstring cites "importlib_resources#489", most likely meaning importlib_metadata fix(ipv6-route): compute Hdr Ext Len in 8-octet units on both sides #489, whose thread matches.
  • Tests:
    • With _pcap_backend.py reverted, the new test fails.
    • test_pcap_backend (14), the pcap_ct and pypcap engine tests and test_core pass.
    • The 10k-iteration benchmark on 3.14 shows no regression (137.6 µs before, 135.8 µs after).
  • UNVERIFIED here: the 3.15 speed-up itself, which needs Docker or a 3.15 interpreter; the author measured 0.2617 → 0.1066 ms/packet.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 23, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage: 91.54% (unit tier, Python 3.15, b90787270, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19599 880 2618 866 92.12%
pcapkit/corekit 2565 71 812 37 96.56%
pcapkit/dumpkit 282 3 102 3 98.44%
pcapkit/foundation 2929 23 1024 25 98.79%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17414 219 4574 185 98.14%
pcapkit/toolkit 869 36 298 10 95.20%
pcapkit/utilities 473 7 136 5 98.03%
pcapkit/vendor 4432 1988 1018 167 51.87%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Ready to merge at b90787270.

  • Review: the Sonnet review is GOOD TO GO.
  • CI: every blocking check passed (Required checks passed SUCCESS). The 3.8–3.10 KTLO legs carry no warnings.
  • Ordering: the tests/foundation plain-unittest ordering leg passed on a scratch merge into main: 714 tests, 0 failures.

This closes #2110. Once it merges, I'll re-measure the 3.15 pcap_ct cell in #2109's table.

@JarryShaw
JarryShaw merged commit 82b901b into main Oct 23, 2026
65 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 23, 2026
@JarryShaw
JarryShaw deleted the perf/2110-pcap-dist-lookup branch October 23, 2026 22:09
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 23, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

perf Pull requests that improve performance (perf: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

perf: pcap_ct extraction is about 2.3x slower on Python 3.15 than on 3.12-3.14

1 participant