Skip to content

fix(toolkit): key tunnelled TCP by the innermost IP header (#1581) #1605

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JarryShaw merged 1 commit into from
Oct 10, 2026
Merged

fix(toolkit): key tunnelled TCP by the innermost IP header (#1581) #1605

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/source/pcapkit/foundation/engines/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@ offers, and the ones that cannot say so rather than quietly doing less:
| :class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` | no IPv6 decoder, hence no IPv6 reassembly -- disabled with an |
| | :class:`~pcapkit.utilities.warnings.AttributeWarning`, and |
| | :func:`~pcapkit.toolkit.pypcapfile.ipv6_reassembly` raises; |
| | TCP over IPv6, and TCP tunnelled in IP (e.g. 6in4, 4in4), are |
| | left out of TCP reassembly and flow tracing, each with an |
| | TCP over IPv6, tunnelled in IP (e.g. 6in4) or not, is left |
| | out of TCP reassembly and flow tracing, with an |
| | :class:`~pcapkit.utilities.warnings.AttributeWarning` once |
| | per capture; IPv4 behind 802.1Q/802.1ad VLAN tags is read as |
| | it is untagged; PCAP savefiles only, otherwise |
Expand Down
8 changes: 5 additions & 3 deletions docs/source/pcapkit/toolkit/3rdparty.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,11 +185,13 @@ cannot be used for it.
IPv6 fragment".

Nor does it decode a VLAN tag or a tunnel. The adapters read an IPv4 packet
behind 802.1Q and 802.1ad tags as `PyPCAPFile`_ reads an untagged one, while
TCP over IPv6 and TCP tunnelled in IP (e.g. 6in4, 4in4, 6in6) are left out of
behind 802.1Q and 802.1ad tags as `PyPCAPFile`_ reads an untagged one, and
TCP tunnelled in IP (e.g. 4in4, 4in6) from its innermost IPv4 packet, keyed by
that packet's addresses as the default engine keys it, while TCP over IPv6,
tunnelled (e.g. 6in4, 6in6) or not, is left out of
:func:`~pcapkit.toolkit.pypcapfile.tcp_reassembly` and
:func:`~pcapkit.toolkit.pypcapfile.tcp_traceflow`, with an
:class:`~pcapkit.utilities.warnings.AttributeWarning` once per capture for each.
:class:`~pcapkit.utilities.warnings.AttributeWarning` once per capture.

.. autofunction:: pcapkit.toolkit.pypcapfile.ipv4_reassembly

Expand Down
75 changes: 57 additions & 18 deletions pcapkit/toolkit/dpkt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,17 @@
#: .. _DPKT: https://dpkt.readthedocs.io
BUFFER_ATTR = '__pcapkit_buffer__'

#: IP protocol numbers of the IP-in-IP tunnels the default engine dissects
#: (see :attr:`Internet.__proto__
#: <pcapkit.protocols.internet.internet.Internet.__proto__>`), over IPv4 and
#: IPv6 alike: IPv4 (protocol 4, :rfc:`2003`) and IPv6 (protocol 41,
#: :rfc:`2473` and :rfc:`4213`). `DPKT`_ also decodes an IPv4 packet's
#: protocol 0 as IPv4, its ``IP_PROTO_IP``, where the default engine reads an
#: IPv6 Hop-by-Hop Options header, so :func:`_tcp_carrier` does not enter it.
#:
#: .. _DPKT: https://dpkt.readthedocs.io
IP_TUNNELS = frozenset((int(Enum_TransType.IPv4), int(Enum_TransType.IPv6)))


class RecordTimestamp:
"""A record's capture timestamp, with what a PCAP frame record needs of it.
Expand Down Expand Up @@ -420,6 +431,44 @@ def _network(packet: 'Packet') -> 'Optional[IP | IP6]':
return getattr(packet, 'ip6', None)


def _tcp_carrier(packet: 'Packet') -> 'tuple[Optional[IP | IP6], Optional[TCP]]':
"""Fetch the IP layer of a DPKT packet that carries its TCP segment, and the segment.

Args:
packet: DPKT packet, i.e. the outermost layer.

Returns:
The innermost IPv4 or IPv6 packet of the network layer (see
:func:`_network`) and of the IP-in-IP tunnels it carries, and its
:class:`dpkt.tcp.TCP` payload, if it has one; else :data:`None` for the
latter, and for both with no network layer. A tunnel is a packet whose
protocol, or for IPv6 the next header after its extension headers, is
one of the :data:`IP_TUNNELS`.

Note:
`DPKT`_ decodes an IPv4 or IPv6 packet inside another (:rfc:`2003`,
:rfc:`2473`, :rfc:`4213`) as its ``data``. The TCP endpoints are the
innermost packet's addresses, as for the default engine
(:func:`pcapkit.toolkit.pcap.tcp_segment`); the outer ones are the
tunnel's (:issue:`1581`).

.. _DPKT: https://dpkt.readthedocs.io

"""
import dpkt # isort:skip

ip = _network(packet)
if ip is None:
return None, None
while getattr(ip, 'p', None) in IP_TUNNELS and isinstance(ip.data, (dpkt.ip.IP, dpkt.ip6.IP6)):
ip = ip.data

tcp = getattr(ip, 'tcp', None) # type: Optional[TCP]
if tcp is None and type(getattr(ip, 'data', None)).__name__ == 'TCP':
tcp = cast('TCP', ip.data)
return ip, tcp


def ipv6_hdr_len(ipv6: 'IP6') -> 'int':
"""Calculate length of headers before IPv6 Fragment header.

Expand Down Expand Up @@ -670,7 +719,8 @@ def tcp_reassembly(packet: 'Packet', timestamp: 'float', *,
Data for TCP reassembly.

* If the ``packet`` can be used for TCP reassembly. A packet can be reassembled
if it contains TCP layer (:class:`dpkt.tcp.TCP`).
if it contains TCP layer (:class:`dpkt.tcp.TCP`), keyed by the IP packet that
carries it -- in an IP-in-IP tunnel, the innermost (see :func:`_tcp_carrier`).
* If the ``packet`` can be reassembled, then the :obj:`dict` mapping of data for TCP
reassembly (:term:`reasm.tcp.packet`) will be returned; otherwise, returns :data:`None`.

Expand All @@ -687,14 +737,8 @@ def tcp_reassembly(packet: 'Packet', timestamp: 'float', *,
:class:`pcapkit.foundation.reassembly.tcp.TCP`

"""
ip = _network(packet)
if ip is None:
return None

tcp = getattr(ip, 'tcp', None) # type: Optional[TCP]
if tcp is None and type(getattr(ip, 'data', None)).__name__ == 'TCP':
tcp = cast('TCP', ip.data)
if tcp is not None:
ip, tcp = _tcp_carrier(packet)
if ip is not None and tcp is not None:
flags = bin(tcp.flags)[2:].zfill(8)
wire, _ = _layer2bytes(packet, tcp)
raw_len = len(tcp.data) # payload length, header excludes
Expand Down Expand Up @@ -740,7 +784,8 @@ def tcp_traceflow(packet: 'Packet', timestamp: 'float', *,
Data for TCP reassembly.

* If the ``packet`` can be used for TCP flow tracing. A packet can be reassembled
if it contains TCP layer (:class:`dpkt.tcp.TCP`).
if it contains TCP layer (:class:`dpkt.tcp.TCP`), keyed by the IP packet that
carries it -- in an IP-in-IP tunnel, the innermost (see :func:`_tcp_carrier`).
* If the ``packet`` can be reassembled, then the :obj:`dict` mapping of data for TCP
flow tracing (:term:`trace.tcp.packet`) will be returned; otherwise, returns :data:`None`.

Expand All @@ -757,14 +802,8 @@ def tcp_traceflow(packet: 'Packet', timestamp: 'float', *,
:class:`pcapkit.foundation.traceflow.tcp.TCP`

"""
ip = _network(packet)
if ip is None:
return None

tcp = getattr(ip, 'tcp', None) # type: Optional[TCP]
if tcp is None and type(getattr(ip, 'data', None)).__name__ == 'TCP':
tcp = cast('TCP', ip.data)
if tcp is not None:
ip, tcp = _tcp_carrier(packet)
if ip is not None and tcp is not None:
flags = bin(tcp.flags)[2:].zfill(8)
wire, _ = _layer2bytes(packet, tcp)

Expand Down
66 changes: 51 additions & 15 deletions pcapkit/toolkit/pcap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,11 @@
from pcapkit.foundation.reassembly.data.ip import Packet as IP_Packet
from pcapkit.foundation.reassembly.data.tcp import Packet as TCP_Packet
from pcapkit.foundation.traceflow.data.tcp import Packet as TF_TCP_Packet
from pcapkit.protocols.internet.ip import IP as Protocol_IP
from pcapkit.protocols.misc.null import NoPayload
from pcapkit.protocols.misc.raw import Raw
from pcapkit.protocols.protocol import ProtocolBase
from pcapkit.protocols.transport.tcp import TCP

if TYPE_CHECKING:
from ipaddress import IPv4Address, IPv6Address
Expand All @@ -32,8 +36,6 @@
from pcapkit.protocols.internet.ipv6 import IPv6
from pcapkit.protocols.internet.ipv6_frag import IPv6_Frag
from pcapkit.protocols.misc.pcap import Frame
from pcapkit.protocols.protocol import ProtocolBase
from pcapkit.protocols.transport.tcp import TCP

__all__ = ['ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'tcp_traceflow',
'tcp_segment', 'TCPSegment']
Expand All @@ -46,7 +48,7 @@ class TCPSegment(NamedTuple):

"""

#: Info of the IP layer carrying the segment.
#: Info of the IP layer carrying the segment, the innermost one in a tunnel.
ip: 'Data_IPv4 | Data_IPv6'
#: TCP source port.
srcport: 'int'
Expand All @@ -68,6 +70,38 @@ class TCPSegment(NamedTuple):
payload: 'bytes'


def _carrier(frame: 'ProtocolBase') -> 'tuple[Optional[IPv4 | IPv6], Optional[TCP]]':
"""Fetch the TCP layer of a frame and the IP layer that carries it.

Args:
frame: PCAP frame or PCAP-NG block.

Returns:
The IPv4 or IPv6 layer nearest above the frame's TCP layer, and that TCP
layer; with no TCP layer, the last IPv4 or IPv6 layer of the frame and
:data:`None`. With no IP layer, neither.

Note:
The nearest IP layer is the innermost one of an IP-in-IP tunnel
(:rfc:`2003`, :rfc:`2473`, :rfc:`4213`), whose addresses are the TCP
endpoints', as in Wireshark's TCP conversations; the outermost, which
``frame['IP']`` finds, is the tunnel's (:issue:`1581`). It is found by
walking down from the outermost, so the headers between them do not
intervene: IPv6 dissects its own extension headers, and an AH or other
header inside IPv4 is a layer of its own but not an IP layer.

"""
if 'IP' not in frame:
return None, None
ip = cast('IPv4 | IPv6', frame['IP'])
layer = ip.payload
while isinstance(layer, ProtocolBase) and not isinstance(layer, (NoPayload, TCP)):
if isinstance(layer, Protocol_IP):
ip = cast('IPv4 | IPv6', layer)
layer = layer.payload
return ip, (cast('TCP', frame['TCP']) if 'TCP' in frame else None)


def tcp_segment(frame: 'ProtocolBase') -> 'Optional[TCPSegment]':
"""The TCP segment ``frame`` carries, or :data:`None` if it carries none.

Expand All @@ -78,11 +112,13 @@ def tcp_segment(frame: 'ProtocolBase') -> 'Optional[TCPSegment]':
frame: PCAP frame or PCAP-NG block.

Returns:
The fields of the TCP layer if :mod:`pcapkit` parsed one. Otherwise, the
fields of the fixed 20-octet TCP header (:rfc:`9293#section-3.1`) if the
IP payload is a :class:`~pcapkit.protocols.misc.raw.Raw` the TCP parser
rejected, of protocol TCP, from a datagram that is not a later fragment,
with at least those 20 octets captured; else :data:`None`.
The fields of the TCP layer if :mod:`pcapkit` parsed one, with those of
the IP layer that carries it -- in a tunnel, the innermost
(:issue:`1581`). Otherwise, the fields of the fixed 20-octet TCP header
(:rfc:`9293#section-3.1`) if the payload of the innermost IP layer is a
:class:`~pcapkit.protocols.misc.raw.Raw` the TCP parser rejected, of
protocol TCP, from a datagram that is not a later fragment, with at
least those 20 octets captured; else :data:`None`.

Note:
A capture snapped inside the TCP header leaves a Data Offset that runs
Expand All @@ -92,14 +128,17 @@ def tcp_segment(frame: 'ProtocolBase') -> 'Optional[TCPSegment]':
are all in the fixed header, so the segment still belongs to its flow,
as it does for :mod:`dpkt`, :mod:`scapy` and Wireshark (:issue:`1518`).
The header and payload are then the captured octets either side of the
Data Offset.
Data Offset. Inside a tunnel, this holds of the IP layer carrying the
segment, as it does without one.

"""
if 'TCP' in frame:
tcp = cast('TCP', frame['TCP'])
ip, tcp = _carrier(frame)
if ip is None:
return None
if tcp is not None:
tcp_info = tcp.info
return TCPSegment(
ip=cast('IPv4 | IPv6', frame['IP']).info,
ip=ip.info,
srcport=tcp_info.srcport.port,
dstport=tcp_info.dstport.port,
seq=tcp_info.seq,
Expand All @@ -111,9 +150,6 @@ def tcp_segment(frame: 'ProtocolBase') -> 'Optional[TCPSegment]':
payload=tcp.packet.payload,
)

if 'IP' not in frame:
return None
ip = cast('IPv4 | IPv6', frame['IP'])
ip_info = ip.info
raw = ip.payload
# NOTE: a Raw with no error is one the caller asked for, by stopping the
Expand Down
Loading
Loading