Skip to content

perf(foundation): probe the pcap backend once per extraction (#2113) - #2121

Merged
JarryShaw merged 1 commit into
mainfrom
perf/2113-probe-once
Oct 24, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
perf/2113-probe-once

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • perf — performance improvements

Description of your pull request and other information

Closes #2113. An extraction with engine='pypcap' or 'pcap_ct' probed the pcap backend three times: _select_engine called unsupported_reason() in both the constructor and run(), and the engine constructor called _pcap_backend.probe() again. The default engine never probes.

  • Fix: _pcap_backend.Memo is a context manager backed by a ContextVar, so threads don't share it. While one is entered, probe() reuses its result. Extractor keeps one memo per extraction (_exprb) and enters it around both unsupported_reason() calls and the engine construction.
  • Not a process-wide cache: probe() is documented as uncached, and many tests mock the distribution scan, sys.modules or import_module and expect a fresh answer on each call. Outside a memo nothing changes. Engine choice and warning text and stacklevel stay the same.
  • One behaviour change: with auto=False, run() reuses the constructor's probe, which matches the format decision that probe already made.
  • Measured (3.14, stub pcap_ct module, real distribution scan): 0.797 → 0.343 ms per extraction, with scans going from 3 to 1. The default engine is unchanged.
  • Test: test_pcap_backend_probe_once_2113_unit asserts one scan per extraction for each pcap engine, the collision warning firing once, and no caching across extractions or across direct calls. Without the fix it fails with 3 != 1. The foundation ordering leg runs 722 tests, 0 failures.

@JarryShaw JarryShaw added perf Pull requests that improve performance (perf: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 23, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at 5e5737db5 (Sonnet review).

  • Memo unwinding: an exception inside a nested with memo: with memo: unwinds the token stack, and _MEMO returns to None.
  • No stale probe across extractions: the memo lives on the Extractor. __iter__ and __next__ never probe. _exprb always exists by the time run() enters it.
  • Behaviour unchanged: the probe-once test fails with the change reverted. The backend, pypcap, pcap_ct, pypcapfile and test_core tests pass.
  • Known limit, no change: one Extractor shared across threads could exit its memo out of order.
  • UNVERIFIED: the Memo docs entry was read, not rendered.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 23, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 23, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Ready to merge at 5e5737db5.

  • Review: the Sonnet review is GOOD TO GO.
  • CI: every blocking check passed (Required checks passed SUCCESS). The 3.8–3.10 KTLO legs carry no warnings.
  • Ordering: the tests/foundation plain-unittest ordering leg passed on a scratch merge into main (82b901b91). It ran 722 tests with 0 failures.

This closes #2113.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage: 91.54% (unit tier, Python 3.15, 5e5737db5, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19599 880 2618 866 92.12%
pcapkit/corekit 2580 75 814 37 96.46%
pcapkit/dumpkit 282 3 102 3 98.44%
pcapkit/foundation 2968 23 1034 25 98.80%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17414 219 4574 185 98.14%
pcapkit/toolkit 869 36 298 10 95.20%
pcapkit/utilities 473 7 136 5 98.03%
pcapkit/vendor 4432 1988 1018 167 51.87%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw
JarryShaw merged commit 30a0d6b into main Oct 24, 2026
65 checks passed
@JarryShaw
JarryShaw deleted the perf/2113-probe-once branch October 24, 2026 00:23
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 24, 2026
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 24, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

perf Pull requests that improve performance (perf: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

perf(foundation): the Extractor probes the pcap backend three times per extraction

1 participant