Skip to content

fix(engines): the dpkt engine applies the first PCAP-NG interface's linktype and timestamp resolution to every frame - #1396

Merged
JarryShaw merged 1 commit into
mainfrom
fix/1379-dpkt-pcapng-multi-idb
Oct 8, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/1379-dpkt-pcapng-multi-idb

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner
  • Searched for similar pull requests
  • Followed the coding style (make pylint, make mypy, make isort) — pylint/mypy on engines/dpkt.py, no new findings bar house-style local imports; isort via tests/project
  • make test passes, and a test case covers the change — modules listed below
  • Added a changelog entry under docs/source/changelog/ and regenerated CHANGELOG.md, if the change is user-visible — N/A, added centrally

What is the purpose of your pull request?

  • fix — corrects a defect

Description of your pull request and other information

Closes #1379

dpkt.pcapng.Reader reads only the first IDB, so every frame took interface 0's link type and if_tsresol. The engine now reads PCAP-NG with a new engines.dpkt.PCAPNGReader. It parses SHBs and IDBs with pcapkit's own PCAPNG protocol, one section at a time. Packet blocks are still decoded by dpkt. Each packet takes the link type, if_tsresol and if_tsoffset of the interface it names, using the default engine's Decimal arithmetic. _extmp is now this reader on PCAP-NG, not dpkt's. SPBs are still skipped (#1361 note updated in 3rdparty.rst).

Before → after (dpkt vs default, EPB/PB frames): profile.pcapng 20/40 timestamps wrong (ns interface read as µs, e.g. 1500000000.0005 vs 1500000.0000005) → 0. test.pcapng frame 2 (RAW, ns) parsed as Ethernet with the wrong timestamp → matches. many_interfaces.pcapng frames 3–4 (NULL, if 10) parsed as Ethernet → raw. A synthesised LE+BE two-section capture loses its last frame on main (3 ≠ 4).

Tests: new tests/foundation/engines/test_dpkt_pcapng_interfaces_unit.py (4 passed). All 4 fail with engines/dpkt.py reverted to origin/main: two behavioural failures (timestamp list, 3 != 4), two ImportError. test_runtime_engines.py now patches PCAPNGReader rather than dpkt.pcapng.Reader. All 12 tests/foundation/engines modules, the dpkt integration and extraction modules, and the foundation ordering leg (522 OK) pass. tests/project: 391 passed, 1 skipped.

@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 8, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review verdict on ac02fa9d0: GOOD TO GO (Sonnet reviewed; author Opus).

  • Parity reproduced on all 6 .pcapng fixtures. dpkt and the default engine show 0 timestamp mismatches. profile.pcapng is 40/40 (main had 20 wrong). The NULL and RAW frames now come out as RawPacket, not Ethernet. test.pcapng gives 4 frames against 5; the missing one is the Simple Packet Block, which is the documented docs(engines): the dpkt engine silently drops PCAP-NG Simple Packet Blocks #1361 non-goal.
  • Edge cases match the default engine. These were synthetic captures, in both byte orders: negative if_tsoffset, if_tsresol 0x80 and 3, an obsolete Packet Block, an unknown block type, and an IDB arriving after packets. A packet with no IDB, or a block length under 12, raises FormatError.
  • Memory. The reader streams block by block, the same as dpkt's.
  • Not breaking. The only user of datalink() is the engine itself.
  • Docs and tests. sphinx -b dummy resolves the new autoclass and refs. The revert proof fails all 4 new tests, which I re-ran myself at this head (4/0). The foundation leg is 522/0.

Nits, not blocking:

  • The new tests cover only the happy paths. Negative offset, obsolete PB and IDB-after-packet work, but no test pins them.
  • On a truncated final block, dpkt drops it silently while the default engine still yields the frame. This is undocumented.
  • 3rdparty.rst has an over-long line near :73.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 8, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

CI found a real defect on Python 3.10: back to NEEDS CHANGES. pcapkit/foundation/engines/dpkt.py:166 calls decimal.localcontext(prec=64). That keyword form exists only from Python 3.11, so all 3 new dpkt tests, plus test_extraction_record_header_unit::…test_pcapng_output_opens_with_the_section_header, fail on 3.10 with TypeError: 'prec' is an invalid keyword argument. The repo already has a shim, pcapkit.utilities.compat.localcontext, which frame.py and pcapng.py use. The fix is to import that instead.

The red PyPCAP/PyPCAPFile parity Python 3.11 leg is unrelated: it is the apt mirror flake, failing at the "Install libpcap headers" step.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: good-to-go Cross-review at the current head says ready; CI state is separate labels Oct 8, 2026
…inktype and timestamp resolution to every frame

dpkt.pcapng.Reader reads only the first Interface Description Block, so
every packet got interface 0's link type and if_tsresol, and a later
section's interfaces were never read. The DPKT engine now reads PCAP-NG
with its own PCAPNGReader: SHBs and IDBs are parsed by pcapkit's PCAPNG
protocol per section, packet blocks are still decoded by dpkt, and each
packet takes the link type, if_tsresol and if_tsoffset of the interface
it names, with the default engine's timestamp arithmetic.

Closes #1379
@JarryShaw
JarryShaw force-pushed the fix/1379-dpkt-pcapng-multi-idb branch from ac02fa9 to 5fcf07d Compare October 8, 2026 01:59
@JarryShaw

Copy link
Copy Markdown
Owner Author

Delta verdict on 5fcf07d8e: GOOD TO GO. I checked git range-diff against ac02fa9d0. The only change is the import of pcapkit.utilities.compat.localcontext and its use at dpkt.py:166. The rest of the patch is identical, so the earlier cross-review still applies. The author re-ran the dpkt modules and tests/project (391) on 3.14. The Python 3.10 leg of this run is the real confirmation, and "ready" waits on it.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Coverage: 89.42% (unit tier, Python 3.14, 5fcf07d8e, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 4 93.55%
pcapkit/const 18791 968 2352 869 90.90%
pcapkit/corekit 2018 88 628 24 94.78%
pcapkit/dumpkit 234 2 84 2 98.74%
pcapkit/foundation 2637 98 928 48 94.78%
pcapkit/interface 112 7 40 5 92.11%
pcapkit/protocols 16262 203 4296 165 98.17%
pcapkit/toolkit 539 74 168 3 84.58%
pcapkit/utilities 429 4 122 4 98.55%
pcapkit/vendor 4409 2342 1006 157 43.25%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw
JarryShaw merged commit 979dc37 into main Oct 8, 2026
38 checks passed
@JarryShaw
JarryShaw deleted the fix/1379-dpkt-pcapng-multi-idb branch October 8, 2026 02:37
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 8, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(engines): the dpkt engine applies the first PCAP-NG interface's linktype and timestamp resolution to every frame

1 participant