Repository navigation
perf(foundation): find pcap distributions without parsing METADATA (#2110) - #2112
Merged
Merged
Conversation
…2110) installed_distributions() runs three times per pcap_ct extraction. On Python 3.15 importlib.metadata.distribution() parses every candidate's METADATA (Distribution._prefer_valid) to rank valid dists first, which took 115 -> 397 us per call and made pcap_ct 0.110 -> 0.261 ms/packet. Ask importlib.metadata.distributions(name=...) for the first match instead: it finds a distribution exactly when distribution() does (checked on 3.8 and 3.15) and reads no file. pcap_ct on 3.15 is back to 0.107 ms/packet; 3.14 is unchanged. The new test puts an empty pcap_ct dist-info on sys.path, makes any METADATA read raise, and asserts pcap-ct is still found.
Owner
Author
|
GOOD TO GO at
|
Contributor
|
Coverage: 91.54% (unit tier, Python 3.15,
Per-file detail: the |
Owner
Author
|
Ready to merge at
This closes #2110. Once it merges, I'll re-measure the 3.15 |
4 of 5 tasks
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
make pylint,make mypy,make isort)make testpasses, and a test case covers the changeWhat is the purpose of your pull request?
perf— performance improvementsDescription of your pull request and other information
Closes #2110. pcap_ct extraction was about 2.4× slower on 3.15. The cause is pcapkit's
installed_distributions()in_pcap_backend.py, which runs three times per extraction. On 3.15,importlib.metadata.distribution()→Distribution.from_namewraps its search in_prefer_valid, which parses every candidate'sMETADATA(importlib_metadata #489). One call costs 115 µs on 3.14 and 397 µs on 3.15. libpcap 1.10.3, pcap-ct 1.3.0b3 and a ctypes call cost the same on both versions.Fix:
next(iter(importlib.metadata.distributions(name=name)), None)matches on thedist-infodirectory and reads no file. It finds a package in exactly the cases the old call did; checked on 3.8.20 and 3.15.1 forpcap-ct,pcap_ct,pypcapandpip. Theprobe()docstring no longer claims a repeat call costs only asys.moduleslookup.Measured (Docker images from docs: regenerate the engine benchmark with Python 3.15 measured #2109's harness, 6-packet
in.pcap):Test: puts an empty
pcap_ctdist-info onsys.pathand makes anyMETADATAread raise. Without the fix it fails on both 3.14 and 3.15. The backend, pcap_ct and pypcap engine tests pass on both.Follow-up: the Extractor still probes three times per extraction (
_select_enginecallsunsupported_reasontwice, plus the constructor). Tracked separately.