Skip to content

fix(toolkit): pyshark skips partial TCP headers and traces first fragments (#1611, #1612) - #1613

Merged
JarryShaw merged 1 commit into
mainfrom
fix/pyshark-bad-tcp-and-fragments
Oct 10, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/pyshark-bad-tcp-and-fragments

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Please follow the guide below

What is the purpose of your pull request?

Tick the commit type your subject line carries.

  • fix — corrects a defect
  • feat — adds a feature
  • perf — changes performance, not behaviour
  • refactor — changes neither behaviour nor performance
  • test — tests only
  • docs — documentation only
  • ci — workflows or build tooling
  • release — bumps the version or rolls up a distribution
  • chore — anything else

Description of your pull request and other information

Closes #1611
Closes #1612

#1611. tcp_traceflow now traces a pyshark TCP layer only if it has tcp.urgent_pointer. Octets 18–19 hold the last field of the fixed header. I checked dissect_tcp in epan/dissectors/packet-tcp.c for Wireshark 2.6.0, 3.0.0 and 4.6.0. In all three, tshark adds that field whenever the Data Offset is at least 5 and all 20 octets were captured. It is added after every field tcp_traceflow reads.

  • Data Offset 0–4 or a header cut inside its first 14 octets no longer raises AttributeError.
  • A frame cut at 14–19 TCP octets used to be traced. It is now skipped, as in the default engine.

#1612. FileCapture now runs tshark with ip.defragment and ipv6.defragment set to FALSE (PyShark.OVERRIDE_PREFS). As a result, TCP in a fragmented datagram is traced at its first fragment, both IPv4 and IPv6: (1, 2)/(6, 7), no longer (1, 5)/(6, 10).

  • Side effect in tshark's TCP analysis. With reassembly off, tshark sizes a fragmented segment from its first fragment alone, so it sees a gap in the sequence. Later frames of that stream, fragments or not, then gain tcp.analysis.lost_segment, and their tcp.completeness changes. Those fields reach the frame records packet2dict builds. tcp_traceflow reads none of them, so tracing is unaffected, and a new test checks this on such a stream.
  • Fixtures. The 23 fixture captures carry no fragmented TCP. On them, every field of every frame is the same with reassembly on and off, except the first and last fragments in ipv6.pcap.

Tests. tests/toolkit/test_pyshark_unit.py:

  • adds fake-based tests that run on 3.14;
  • adds real-tshark tests that compare against the default engine frame by frame, and skip unless pyshark runs.

The first-fragment test from #1605 no longer excludes pyshark. test_runtime_engines.py now asserts the FileCapture call with override_prefs=PyShark.OVERRIDE_PREFS. Every new test fails with its fix reverted, except one guard: PySharkFixedHeaderTests::test_a_whole_header_is_traced passes on main too. The engine gap table in the docs is updated.

@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 89.72% (unit tier, Python 3.14, 396140c88, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 3 20 3 95.16%
pcapkit/const 18858 956 2366 870 91.04%
pcapkit/corekit 2366 67 760 35 96.48%
pcapkit/dumpkit 275 3 100 3 98.40%
pcapkit/foundation 2884 147 1010 36 94.07%
pcapkit/interface 138 7 52 5 93.68%
pcapkit/protocols 17089 227 4566 192 98.05%
pcapkit/toolkit 903 123 320 6 85.36%
pcapkit/utilities 494 4 146 4 98.75%
pcapkit/vendor 4409 2342 1006 157 43.25%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict: NEEDS CHANGES at c9b32c05c. This was a Sonnet cross-review; the author ran on Opus. Only the wording needs to change; the code holds.

What the review confirmed:

Two fixes:

  1. The OVERRIDE_PREFS comment (pcapkit/foundation/engines/pyshark.py, ~line 110) overstates the change. It says turning tshark's reassembly off changes no frame that is not a fragment. That is false for a stream that carries a fragmented TCP segment. tshark then sizes the segment from the first fragment, so later non-fragment frames gain tcp.analysis.lost_segment and a different tcp.completeness. I re-measured this with tshark 4.6.9 on such a capture. tcp_traceflow reads neither field, so tracing is unaffected, but the comment and the PR body should say so and limit the claim to the fixtures.
  2. The body says every new test fails with its fix reverted. PySharkFixedHeaderTests::test_a_whole_header_is_traced passes on main. Name it as a guard.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 10, 2026
…ments (#1611, #1612)

- tcp_traceflow read tcp.flags_syn whether or not tshark had reported it, so
  a Data Offset below 5 or a header cut short raised a bare AttributeError and
  aborted the extraction; a header cut at 14-19 octets was traced where the
  default engine skips it. A TCP layer is now traced only if it carries
  tcp.urgent_pointer, the last field of the fixed header.
- The engine left tshark's IP reassembly on, so TCP in a fragmented datagram
  was traced at its last fragment. FileCapture now gets ip.defragment and
  ipv6.defragment FALSE through override_prefs (PyShark.OVERRIDE_PREFS).
  That changes tshark's TCP sequence analysis on later frames of such a
  stream, which tcp_traceflow does not read.
- Tests: fake and real-tshark tests in tests/toolkit/test_pyshark_unit.py,
  including a stream around fragmented segments; the 1581 first-fragment
  test no longer excludes pyshark; the FileCapture call assertion in
  test_runtime_engines.py names the new prefs.
- Docs: the engine gap table says tshark's IP reassembly is off.

Narrow runs pass on Python 3.11 (pyshark 0.6, tshark 4.6.9) and 3.14.
@JarryShaw
JarryShaw force-pushed the fix/pyshark-bad-tcp-and-fragments branch from c9b32c0 to 396140c Compare October 10, 2026 17:52
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 10, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict: GOOD TO GO at 396140c88. This is the round-2 delta review, by Sonnet; the author ran on Opus.

The product code is unchanged since round 1, which the review already confirmed.

  • The reworded OVERRIDE_PREFS comment is accurate. On the new test's stream and on the review's own capture, turning reassembly off changes only tshark's TCP conversation-state fields on non-fragment frames. tcp_traceflow reads none of them.
  • The new stream test exercises those frames. It fails for IPv4 and IPv6 with the engine fix reverted.
  • The body names the guard correctly.
  • test_pyshark_unit.py passes on both Pythons: 38 passed on 3.11 with tshark; 38 passed, 6 skipped on 3.14.

Non-blocking:

  • The comment's field list omits tshark's expert annotations (_ws.expert*), which change with tcp.analysis.flags.
  • "Later frames gain lost_segment" holds for some of those frames, not all.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 10, 2026
@JarryShaw
JarryShaw merged commit 162c287 into main Oct 10, 2026
39 checks passed
@JarryShaw
JarryShaw deleted the fix/pyshark-bad-tcp-and-fragments branch October 10, 2026 22:59
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 10, 2026
JarryShaw added a commit that referenced this pull request Oct 14, 2026
Third-party engines are best effort (#1687), so the pyshark code that
exists only to match the default engine comes out:

- revert #1686, #1670 and #1642: the second tshark pass that read a
  frame's octets, the link type read from the capture file, the 64-layer
  bound, reading past tshark's 802.1Q limit, and keying a source-routed
  segment by ip.cur_rt
- revert #1613's tshark preferences that turned IP reassembly off, with
  its FileCapture assertion in test_runtime_engines.py and its sentence
  in the engines index
- keep #1613's partial-TCP-header guard and #1605's keying by the IP
  header that carries the segment
- re-land #1671's crash fix: a TCP layer whose IP layer lacks ip.src or
  ip.dst is not traced, rather than raising AttributeError
- delete the three test modules those reverts added and the agreement
  checks in test_pyshark_unit.py; the #1611 and #1671 checks assert
  constants

The pyshark test modules, test_runtime_engines.py, test_engine_runtime.py
and the project guards pass on 3.11 (tshark 4.6.9) and 3.14.
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

1 participant