diff --git a/docs/source/contributing/conventions/reassembly-integration.rst b/docs/source/contributing/conventions/reassembly-integration.rst index 93ccf40328..109073ae4e 100644 --- a/docs/source/contributing/conventions/reassembly-integration.rst +++ b/docs/source/contributing/conventions/reassembly-integration.rst @@ -4,19 +4,25 @@ Reassembly Integration ----------------------- Where a new reassembly engine under :mod:`pcapkit.foundation.reassembly` plugs into -the rest of the package. The owner's ruling, on this session's DTLS reassembly work. +the rest of the package. The owner's ruling, given on the DTLS reassembly work of +:issue:`2440`. The Rule ~~~~~~~~ #. **A protocol class does not handle or integrate reassembly itself.** The protocol dissector stays a parser; it does not reach into - :mod:`pcapkit.foundation.reassembly` directly. + :mod:`pcapkit.foundation.reassembly` directly. A public method that parses a + reassembled unit from its own octets is still a parser, so a protocol class may + offer one (:issue:`2496`). #. **A new engine needs a new** :mod:`pcapkit.toolkit` **helper** that declares which output engines (|dpkt|_, |pyshark|_, ``scapy``, the default) support it. #. **A new engine's reassembled outcome needs a new** :mod:`pcapkit.foundation.analysis` **method** to parse that outcome into a usable - result, rather than leaving the caller to interpret raw reassembled bytes. + result, rather than leaving the caller to interpret raw reassembled bytes. The + package mirrors :mod:`pcapkit.foundation.reassembly`: a module per engine, whose + public names the package, :mod:`pcapkit.foundation` and :mod:`pcapkit.all` + export (:issue:`2496`). #. **Reassembly always follows the RFC's own definition**, never an algorithm this package invents. Where the RFC states no required algorithm -- buffer the whole message, accept overlaps -- the engine does the same and adds no timeout, cap or @@ -25,33 +31,35 @@ The Rule Compliance, Measured ~~~~~~~~~~~~~~~~~~~~~ -**Point 1 is an existing, enforced precedent.** Neither -:class:`~pcapkit.protocols.internet.ipv4.IPv4` nor -:class:`~pcapkit.protocols.internet.ipv6.IPv6` nor -:class:`~pcapkit.protocols.transport.tcp.TCP` imports or calls into +**Point 1 holds for every engine.** None of +:class:`~pcapkit.protocols.internet.ipv4.IPv4`, +:class:`~pcapkit.protocols.internet.ipv6.IPv6`, +:class:`~pcapkit.protocols.transport.tcp.TCP` and +:class:`~pcapkit.protocols.application.dtls.DTLS` imports or calls into :mod:`pcapkit.foundation.reassembly` at all. The real path is :mod:`pcapkit.foundation.engines` calling a ``pcapkit.toolkit.`` extraction function on an already-parsed frame, feeding the result to the reassembly instance the -:class:`~pcapkit.foundation.extraction.Extractor` holds. - -**Points 2 and 3 describe infrastructure that does not exist yet, for any engine.** -No single :mod:`pcapkit.toolkit` helper enumerates which output engines support a given -reassembly engine -- support is established ad hoc, per engine backend, in three -different shapes (define the extraction functions; define them raising -:exc:`~pcapkit.utilities.exceptions.UnsupportedCall`; or disable reassembly at the -engine level, as :mod:`pcapkit.foundation.engines.pyshark` does). No -:mod:`pcapkit.foundation.analysis` module exists; what parses a reassembled outcome -today is the pre-existing, general-purpose +:class:`~pcapkit.foundation.extraction.Extractor` holds; for DTLS that function is +:func:`pcapkit.toolkit.pcap.dtls_reassembly` (:issue:`2496`). +:meth:`DTLS.read_handshake ` +is the parser method the rule allows. + +**No engine meets point 2.** No single :mod:`pcapkit.toolkit` helper enumerates which +output engines support a given reassembly engine. Support is established ad hoc, per +engine backend, by defining the extraction functions, by defining them raising +:exc:`~pcapkit.utilities.exceptions.UnsupportedCall`, or by disabling reassembly at the +engine level, as :mod:`pcapkit.foundation.engines.pyshark` does. DTLS defines them for +the default engine and is disabled at the engine level on every other. + +**Point 3 holds for DTLS alone.** +:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` parses a reassembled DTLS +handshake message (:issue:`2496`). IP and TCP have no such method: what parses their +outcome is the general-purpose :meth:`Protocol.analyze ` dispatcher, reused via :class:`~pcapkit.foundation.reassembly.data.data.Deferred`, not a method -written for reassembly specifically. So points 2 and 3 are a forward-looking ask, not -yet a pattern any engine follows. - -**The DTLS reassembly work on** :pr:`2451` **violates point 1.** -:class:`~pcapkit.protocols.application.dtls.DTLS` imports -:class:`~pcapkit.foundation.reassembly.dtls.DTLS` directly and calls it from -:meth:`DTLS.read`, which no other protocol class does. Its own module docstring notes -the further divergence from the IP/TCP shape: it is "not toolkit-fed", reports its -reassembled body eagerly instead of through a :class:`Deferred -`, and sets no -``__protocol_type__``. +written for reassembly specifically. + +DTLS differs from the IP/TCP shape where the rule is silent: its reassembled body is +reported eagerly, with no deferred ``packet``, and +:class:`~pcapkit.foundation.reassembly.dtls.DTLS` sets no ``__protocol_type__`` +(:issue:`2531`). diff --git a/docs/source/contributing/pep.rst b/docs/source/contributing/pep.rst index eed7379040..a8368d741f 100644 --- a/docs/source/contributing/pep.rst +++ b/docs/source/contributing/pep.rst @@ -247,10 +247,12 @@ without keys. **Done: reassembling a handshake message DTLS has fragmented across several records** (:rfc:`6347#section-4.2.3`, :issue:`2440`), via -:class:`~pcapkit.foundation.reassembly.dtls.DTLS`, keyed on the UDP 4-tuple. -Pass ``reassembly=`` and ``flow=`` to -:class:`~pcapkit.protocols.application.dtls.DTLS` to opt in; the completing -fragment's body decodes to the full reassembled message, while every other +:class:`~pcapkit.foundation.reassembly.dtls.DTLS`, keyed on the UDP 4-tuple +and the epoch. ``Extractor(reassembly=True, dtls=True)`` opts in on the +default engine, which feeds it through +:func:`pcapkit.toolkit.pcap.dtls_reassembly`, and +:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes each +reassembled message, of any length (:issue:`2496`). In the frame itself every fragment's body stays that one fragment's raw octets -- see :attr:`DTLSHandshake.is_fragmented `. An unfragmented message is unaffected either way. diff --git a/docs/source/pcapkit/foundation/analysis/dtls.rst b/docs/source/pcapkit/foundation/analysis/dtls.rst new file mode 100644 index 0000000000..75ce3cdecd --- /dev/null +++ b/docs/source/pcapkit/foundation/analysis/dtls.rst @@ -0,0 +1,12 @@ +=============================== +DTLS Handshake-Message Analysis +=============================== + +.. module:: pcapkit.foundation.analysis.dtls + +:mod:`pcapkit.foundation.analysis.dtls` contains +:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` only, which decodes a +DTLS handshake message that :class:`~pcapkit.foundation.reassembly.dtls.DTLS` +reassembled (:rfc:`6347#section-4.2.3`). + +.. autofunction:: pcapkit.foundation.analysis.dtls.dtls_handshake diff --git a/docs/source/pcapkit/foundation/analysis/index.rst b/docs/source/pcapkit/foundation/analysis/index.rst new file mode 100644 index 0000000000..ca5b2d3123 --- /dev/null +++ b/docs/source/pcapkit/foundation/analysis/index.rst @@ -0,0 +1,17 @@ +============================ +Reassembled Payload Analysis +============================ + +.. module:: pcapkit.foundation.analysis + +:mod:`pcapkit.foundation.analysis` parses what a +:mod:`pcapkit.foundation.reassembly` engine reassembled into a usable result, +so that a caller does not interpret the raw reassembled octets itself +(:ref:`reassembly-integration`). + +.. versionadded:: 1.6.0 + +.. toctree:: + :maxdepth: 2 + + dtls diff --git a/docs/source/pcapkit/foundation/engines/index.rst b/docs/source/pcapkit/foundation/engines/index.rst index a4ec00a68c..d7c731927f 100644 --- a/docs/source/pcapkit/foundation/engines/index.rst +++ b/docs/source/pcapkit/foundation/engines/index.rst @@ -115,6 +115,11 @@ their row says otherwise: | | Scapy binds it to on the host it runs on, and no TCP or | | | Fragment header behind an AH, Mobility, HIP or Shim6 header | +-----------------------------------------------------------------+---------------------------------------------------------------+ +| :class:`~pcapkit.foundation.engines.dpkt.DPKT`, | no DTLS reassembly, since | +| :class:`~pcapkit.foundation.engines.scapy.Scapy` and | :func:`~pcapkit.toolkit.pcap.dtls_reassembly` reads | +| :class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` | :mod:`pcapkit`'s own DTLS layer -- disabled with an | +| | :class:`~pcapkit.utilities.warnings.AttributeWarning` | ++-----------------------------------------------------------------+---------------------------------------------------------------+ | :class:`~pcapkit.foundation.engines.dpkt.DPKT`, | no PCAP output, since each frame reaches the writer as a | | :class:`~pcapkit.foundation.engines.scapy.Scapy`, | plain :obj:`dict` -- ``json`` is written instead, with a | | :class:`~pcapkit.foundation.engines.pyshark.PyShark`, | :class:`~pcapkit.utilities.warnings.FormatWarning` | diff --git a/docs/source/pcapkit/foundation/index.rst b/docs/source/pcapkit/foundation/index.rst index fc1a32762e..242b94dbc3 100644 --- a/docs/source/pcapkit/foundation/index.rst +++ b/docs/source/pcapkit/foundation/index.rst @@ -6,8 +6,9 @@ Library Foundation :mod:`pcapkit.foundation` is a collection of foundations for :mod:`pcapkit`, including PCAP file extraction tool :class:`~pcapkit.foundation.extraction.Extractor` and the -:mod:`~pcapkit.foundation.engines` it reads through, TCP/IP reassembly -implementations, TCP flow tracer +:mod:`~pcapkit.foundation.engines` it reads through, TCP/IP and DTLS +reassembly implementations and the :mod:`~pcapkit.foundation.analysis` of what +they reassemble, TCP flow tracer :class:`~pcapkit.foundation.traceflow.traceflow.TraceFlow`, and registry management APIs for :mod:`pcapkit`. @@ -17,5 +18,6 @@ management APIs for :mod:`pcapkit`. extraction engines/index reassembly/index + analysis/index traceflow/index registry diff --git a/docs/source/pcapkit/foundation/reassembly/dtls.rst b/docs/source/pcapkit/foundation/reassembly/dtls.rst index bb0484bde9..ec7ff2ed23 100644 --- a/docs/source/pcapkit/foundation/reassembly/dtls.rst +++ b/docs/source/pcapkit/foundation/reassembly/dtls.rst @@ -54,8 +54,9 @@ Terminology is :class:`~pcapkit.foundation.reassembly.data.dtls.Packet` -- one fragment's own ``bufid``, ``message_seq``, ``fragment_offset``, ``fragment_length`` and raw ``payload``, built by - :meth:`DTLS.read ` from - a single :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`. + :func:`pcapkit.toolkit.pcap.dtls_reassembly` or + :func:`pcapkit.toolkit.pcapng.dtls_reassembly` from a single + :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`. reasm.dtls.datagram Data structure for **reassembled DTLS handshake message** (element of @@ -64,7 +65,8 @@ Terminology ``completed``, the original packet identifier, the packet numbers that built it, and ``body``, the reassembled payload (a contiguous :obj:`bytes`, or -- under ``strict=True`` and an incomplete message -- - a :obj:`tuple` of the octet runs actually received). + a :obj:`tuple` of the octet runs actually received), which + :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes. reasm.dtls.buffer Data structure for internal buffering diff --git a/docs/source/pcapkit/protocols/application/dtls.rst b/docs/source/pcapkit/protocols/application/dtls.rst index 4bb57106d4..8aa3c61c86 100644 --- a/docs/source/pcapkit/protocols/application/dtls.rst +++ b/docs/source/pcapkit/protocols/application/dtls.rst @@ -49,11 +49,13 @@ the two formats are told apart by that octet alone. Known Limitations ----------------- -**A handshake message split across DTLS records is not reassembled** -(:rfc:`6347#section-4.2.3`, :issue:`2440`). Such a message is flagged +**A handshake message split across DTLS records is not decoded here** +(:rfc:`6347#section-4.2.3`, :issue:`2440`). Each fragment is flagged :attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.is_fragmented` and its :attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.body` -kept as raw octets. +kept as raw octets. ``Extractor(reassembly=True, dtls=True)`` reassembles the +message, and :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes it +with :meth:`~pcapkit.protocols.application.dtls.DTLS.read_handshake`. **A unified-header record with a connection ID is not decoded** (:rfc:`9147#section-4.1`, :issue:`2467`). Its length is negotiated, not carried @@ -83,6 +85,7 @@ ones on the wire. .. automethod:: read .. automethod:: make .. automethod:: unpack + .. automethod:: read_handshake .. automethod:: __length_hint__ diff --git a/docs/source/pcapkit/toolkit/builtin.rst b/docs/source/pcapkit/toolkit/builtin.rst index 086262ace0..20cc495e67 100644 --- a/docs/source/pcapkit/toolkit/builtin.rst +++ b/docs/source/pcapkit/toolkit/builtin.rst @@ -12,7 +12,8 @@ The reassembly and flow tracing adapters return the data their :mod:`pcapkit.foundation` counterpart consumes, or :data:`None` if the frame cannot be used for it. The TCP adapters here and in :mod:`pcapkit.toolkit.pcapng` read the segment with -:func:`~pcapkit.toolkit.pcap.tcp_segment`. +:func:`~pcapkit.toolkit.pcap.tcp_segment`, and the DTLS ones the record with +:func:`~pcapkit.toolkit.pcap._dtls_fragments`. .. autofunction:: pcapkit.toolkit.pcap.ipv4_reassembly @@ -20,6 +21,8 @@ cannot be used for it. The TCP adapters here and in .. autofunction:: pcapkit.toolkit.pcap.tcp_reassembly +.. autofunction:: pcapkit.toolkit.pcap.dtls_reassembly + .. autofunction:: pcapkit.toolkit.pcap.tcp_traceflow Auxiliary Functions @@ -29,6 +32,8 @@ Auxiliary Functions .. autofunction:: pcapkit.toolkit.pcap._fragment +.. autofunction:: pcapkit.toolkit.pcap._dtls_fragments + .. autoclass:: pcapkit.toolkit.pcap.TCPSegment :members: :show-inheritance: @@ -49,6 +54,8 @@ cannot be used for it. .. autofunction:: pcapkit.toolkit.pcapng.tcp_reassembly +.. autofunction:: pcapkit.toolkit.pcapng.dtls_reassembly + .. autofunction:: pcapkit.toolkit.pcapng.tcp_traceflow Auxiliary Functions diff --git a/pcapkit/all.py b/pcapkit/all.py index 90b845478c..cb097a3e61 100644 --- a/pcapkit/all.py +++ b/pcapkit/all.py @@ -84,6 +84,9 @@ 'TCP_Reassembly', # TCP Reassembly 'DTLS_Reassembly', # DTLS Reassembly + # pcapkit.foundation.analysis + 'dtls_handshake', # DTLS Analysis + # pcapkit.foundation.registry 'register_protocol', 'register_protocol_code', @@ -137,8 +140,9 @@ 'Data', 'data', # Protocol Data # pcapkit.toolkit - 'ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'tcp_traceflow', - 'pcapng_ipv4_reassembly', 'pcapng_ipv6_reassembly', 'pcapng_tcp_reassembly', 'pcapng_tcp_traceflow', + 'ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'dtls_reassembly', 'tcp_traceflow', + 'pcapng_ipv4_reassembly', 'pcapng_ipv6_reassembly', 'pcapng_tcp_reassembly', 'pcapng_dtls_reassembly', + 'pcapng_tcp_traceflow', 'pcapng_block2frame', # default engine 'dpkt_ipv6_hdr_len', 'dpkt_packet2chain', 'dpkt_packet2dict', 'dpkt_ipv4_reassembly', 'dpkt_ipv6_reassembly', 'dpkt_tcp_reassembly', 'dpkt_tcp_traceflow', diff --git a/pcapkit/foundation/__init__.py b/pcapkit/foundation/__init__.py index aee568ac99..e210fb2761 100644 --- a/pcapkit/foundation/__init__.py +++ b/pcapkit/foundation/__init__.py @@ -9,9 +9,11 @@ :mod:`pcapkit`, including PCAP file extraction tool :class:`~pcapkit.foundation.extraction.Extractor`, flow tracing :mod:`~pcapkit.foundation.traceflow`, registry management -APIs for :mod:`pcapkit`, and TCP/IP reassembly implementations. +APIs for :mod:`pcapkit`, TCP/IP and DTLS reassembly implementations, and +the :mod:`~pcapkit.foundation.analysis` of what they reassemble. """ +from pcapkit.foundation.analysis import * from pcapkit.foundation.extraction import Extractor from pcapkit.foundation.reassembly import * from pcapkit.foundation.registry import * @@ -23,6 +25,8 @@ 'IPv4_Reassembly', 'IPv6_Reassembly', 'TCP_Reassembly', 'DTLS_Reassembly', 'ReassemblyManager', + 'dtls_handshake', + 'TCP_TraceFlow', 'TraceFlowManager', diff --git a/pcapkit/foundation/analysis/__init__.py b/pcapkit/foundation/analysis/__init__.py new file mode 100644 index 0000000000..62fde606f1 --- /dev/null +++ b/pcapkit/foundation/analysis/__init__.py @@ -0,0 +1,21 @@ +# -*- coding: utf-8 -*- +# pylint: disable=unused-import +"""Reassembled Payload Analysis +================================== + +.. module:: pcapkit.foundation.analysis + +:mod:`pcapkit.foundation.analysis` parses what a +:mod:`pcapkit.foundation.reassembly` engine reassembled into a usable result, +so that a caller does not interpret the raw reassembled octets itself +(:ref:`reassembly-integration`). + +.. versionadded:: 1.6.0 + +""" +# Analysis of DTLS handshake messages +from pcapkit.foundation.analysis.dtls import dtls_handshake + +__all__ = [ + 'dtls_handshake', # DTLS Analysis +] diff --git a/pcapkit/foundation/analysis/dtls.py b/pcapkit/foundation/analysis/dtls.py new file mode 100644 index 0000000000..ff23176976 --- /dev/null +++ b/pcapkit/foundation/analysis/dtls.py @@ -0,0 +1,67 @@ +# -*- coding: utf-8 -*- +"""DTLS Handshake-Message Analysis +===================================== + +.. module:: pcapkit.foundation.analysis.dtls + +:mod:`pcapkit.foundation.analysis.dtls` contains +:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` only, which decodes a +DTLS handshake message that :class:`~pcapkit.foundation.reassembly.dtls.DTLS` +reassembled (:rfc:`6347#section-4.2.3`). + +.. versionadded:: 1.6.0 + +""" +from typing import TYPE_CHECKING, cast + +from pcapkit.protocols.application.dtls import DTLS +from pcapkit.protocols.data.application.dtls import DTLSHandshake as Data_DTLSHandshake +from pcapkit.utilities.exceptions import ProtocolError + +if TYPE_CHECKING: + from typing import Optional + + from pcapkit.foundation.reassembly.data.dtls import Datagram as DTLS_Datagram + +__all__ = ['dtls_handshake'] + + +def dtls_handshake(datagram: 'DTLS_Datagram') -> 'Optional[Data_DTLSHandshake]': + """Parse a DTLS handshake message reassembled by + :class:`~pcapkit.foundation.reassembly.dtls.DTLS`. + + The message is read as if it had arrived unfragmented, ``fragment_offset`` + 0 and ``fragment_length`` its ``length`` (:rfc:`6347#section-4.2.3`), by + :meth:`DTLS.read_handshake + `. So a + ``ClientHello`` or ``ServerHello`` body decodes as an unfragmented one + does, whatever its length, and any other body stays raw octets. + + Args: + datagram: Reassembled message (:term:`reasm.dtls.datagram`). + + Returns: + The message, unfragmented. Its body stays the raw reassembled octets if + they do not decode as its ``msg_type``. :data:`None` if ``datagram`` is + incomplete, as its missing octets cannot be decoded. + + """ + if not datagram.completed: + return None + # NOTE: a complete message is always one contiguous run. + body = cast('bytes', datagram.body) + message = (bytes([int(datagram.msg_type)]) + datagram.length.to_bytes(3, 'big') + + datagram.message_seq.to_bytes(2, 'big') + bytes(3) # fragment_offset 0 + + datagram.length.to_bytes(3, 'big') + body) + try: + return DTLS.read_handshake(message) + except ProtocolError: + return Data_DTLSHandshake( + msg_type=datagram.msg_type, + length=datagram.length, + body=body, + message_seq=datagram.message_seq, + fragment_offset=0, + fragment_length=datagram.length, + is_fragmented=False, + ) diff --git a/pcapkit/foundation/engines/dpkt.py b/pcapkit/foundation/engines/dpkt.py index 577b8fb7c3..e9483077c2 100644 --- a/pcapkit/foundation/engines/dpkt.py +++ b/pcapkit/foundation/engines/dpkt.py @@ -16,6 +16,7 @@ from pcapkit.const.pcapng.option_type import OptionType as Enum_OptionType from pcapkit.const.reg.linktype import LinkType as Enum_LinkType from pcapkit.foundation.engines.engine import EngineBase +from pcapkit.foundation.reassembly import ReassemblyManager from pcapkit.protocols.schema.misc.pcap.header import RESOLUTION from pcapkit.protocols.schema.misc.pcapng import (BYTE_ORDER_MAGIC, DEFAULT_RESOLUTION, GENERIC_BLOCK_MINIMUM) @@ -480,7 +481,8 @@ def run(self) -> 'None': is provided as the DPKT engine does not support such operations; or if :attr:`self.extractor._exctx ` is provided, as the DPKT engine does not parse with :mod:`pcapkit`'s own - protocol implementations. + protocol implementations; or if DTLS reassembly is enabled, for the + same reason. Raises: FormatError: If the file format is not supported, i.e., not a PCAP @@ -503,6 +505,14 @@ def run(self) -> 'None': "'context=' is ignored", AttributeWarning, stacklevel=stacklevel()) + if ext._flag_r and ext._dtls: + ext._dtls = False + ext._reasm = ReassemblyManager(ipv4=ext._reasm.ipv4, ipv6=ext._reasm.ipv6, + tcp=ext._reasm.tcp, dtls=None) + warn("'Extractor(engine=dpkt)' object does not support DTLS reassembly, " + "which reads pcapkit's own DTLS layer; so 'dtls=True' will be ignored", + AttributeWarning, stacklevel=stacklevel()) + # setup verbose handler, unless the caller gave one of their own (#1874) if ext._vfunc_default: from pcapkit.toolkit.dpkt import packet2chain # isort:skip diff --git a/pcapkit/foundation/engines/pcap.py b/pcapkit/foundation/engines/pcap.py index 1f4b14641a..67a18735da 100644 --- a/pcapkit/foundation/engines/pcap.py +++ b/pcapkit/foundation/engines/pcap.py @@ -147,7 +147,7 @@ def read_frame(self) -> 'Frame': - extract frames and each layer of packets; - make :class:`~pcapkit.corekit.infoclass.Info` object out of frame properties; - write to output file with corresponding dumper; - - reassemble IP and/or TCP datagram; + - reassemble IP and/or TCP datagram, and/or DTLS handshake message; - trace TCP flows if any; - record frame :class:`~pcapkit.corekit.infoclass.Info` object to frame storage. @@ -155,8 +155,8 @@ def read_frame(self) -> 'Frame': Parsed frame instance. """ - from pcapkit.toolkit.pcap import (ipv4_reassembly, ipv6_reassembly, tcp_reassembly, - tcp_traceflow) + from pcapkit.toolkit.pcap import (dtls_reassembly, ipv4_reassembly, ipv6_reassembly, + tcp_reassembly, tcp_traceflow) ext = self._extractor # read frame header @@ -197,6 +197,11 @@ def read_frame(self) -> 'Frame': data_tcp = tcp_reassembly(frame) if data_tcp is not None: ext._reasm.tcp(data_tcp) + if ext._dtls: + data_dtls = dtls_reassembly(frame) + if data_dtls is not None: + for packet_dtls in data_dtls: + ext._reasm.dtls(packet_dtls) # trace flows if ext._flag_t: diff --git a/pcapkit/foundation/engines/pcap_ct.py b/pcapkit/foundation/engines/pcap_ct.py index 511d3184bd..e2bbf61cdd 100644 --- a/pcapkit/foundation/engines/pcap_ct.py +++ b/pcapkit/foundation/engines/pcap_ct.py @@ -375,11 +375,12 @@ def run(self) -> 'None': f'but {ext._ifnm!r} is not one; libpcap opens savefiles ' 'by name and cannot read an in-memory stream') - if ext._flag_r and (ext._ipv4 or ext._ipv6 or ext._tcp): + if ext._flag_r and (ext._ipv4 or ext._ipv6 or ext._tcp or ext._dtls): ext._flag_r = False - ext._reasm = ReassemblyManager(ipv4=None, ipv6=None, tcp=None) + ext._reasm = ReassemblyManager(ipv4=None, ipv6=None, tcp=None, dtls=None) warn("'Extractor(engine=pcap_ct)' object does not support reassembly; " - f"so 'ipv4={ext._ipv4}', 'ipv6={ext._ipv6}' and 'tcp={ext._tcp}' will be ignored", + f"so 'ipv4={ext._ipv4}', 'ipv6={ext._ipv6}', 'tcp={ext._tcp}' and " + f"'dtls={ext._dtls}' will be ignored", AttributeWarning, stacklevel=stacklevel()) if ext._flag_t and ext._tcp: diff --git a/pcapkit/foundation/engines/pcapng.py b/pcapkit/foundation/engines/pcapng.py index 1c0cfe05fe..81462e1b81 100644 --- a/pcapkit/foundation/engines/pcapng.py +++ b/pcapkit/foundation/engines/pcapng.py @@ -202,7 +202,7 @@ def read_frame(self) -> 'P_PCAPNG': - if not, save the block into the internal context storage and repeat; - if yes, save the related information into the internal context storage; - write the parsed block into output file; - - reassemble IP and/or TCP fragments; + - reassemble IP and/or TCP fragments, and/or DTLS handshake messages; - trace TCP flows if any; - record frame information if any. @@ -210,8 +210,8 @@ def read_frame(self) -> 'P_PCAPNG': Parsed PCAP-NG block. """ - from pcapkit.toolkit.pcapng import (ipv4_reassembly, ipv6_reassembly, tcp_reassembly, - tcp_traceflow) + from pcapkit.toolkit.pcapng import (dtls_reassembly, ipv4_reassembly, ipv6_reassembly, + tcp_reassembly, tcp_traceflow) ext = self._extractor while True: @@ -317,6 +317,11 @@ def read_frame(self) -> 'P_PCAPNG': data_tcp = tcp_reassembly(block) if data_tcp is not None: ext._reasm.tcp(data_tcp) + if ext._dtls: + data_dtls = dtls_reassembly(block) + if data_dtls is not None: + for packet_dtls in data_dtls: + ext._reasm.dtls(packet_dtls) # trace flows if ext._flag_t: diff --git a/pcapkit/foundation/engines/pypcap.py b/pcapkit/foundation/engines/pypcap.py index 21c2eb0455..c7085c5966 100644 --- a/pcapkit/foundation/engines/pypcap.py +++ b/pcapkit/foundation/engines/pypcap.py @@ -302,11 +302,12 @@ def run(self) -> 'None': f'but {ext._ifnm!r} is not one; libpcap opens savefiles ' 'by name and cannot read an in-memory stream') - if ext._flag_r and (ext._ipv4 or ext._ipv6 or ext._tcp): + if ext._flag_r and (ext._ipv4 or ext._ipv6 or ext._tcp or ext._dtls): ext._flag_r = False - ext._reasm = ReassemblyManager(ipv4=None, ipv6=None, tcp=None) + ext._reasm = ReassemblyManager(ipv4=None, ipv6=None, tcp=None, dtls=None) warn("'Extractor(engine=pypcap)' object does not support reassembly; " - f"so 'ipv4={ext._ipv4}', 'ipv6={ext._ipv6}' and 'tcp={ext._tcp}' will be ignored", + f"so 'ipv4={ext._ipv4}', 'ipv6={ext._ipv6}', 'tcp={ext._tcp}' and " + f"'dtls={ext._dtls}' will be ignored", AttributeWarning, stacklevel=stacklevel()) if ext._flag_t and ext._tcp: diff --git a/pcapkit/foundation/engines/pypcapfile.py b/pcapkit/foundation/engines/pypcapfile.py index 092bba9997..c1b969b450 100644 --- a/pcapkit/foundation/engines/pypcapfile.py +++ b/pcapkit/foundation/engines/pypcapfile.py @@ -226,6 +226,8 @@ def run(self) -> 'None': own protocol implementations. * if IPv6 reassembly is enabled, as |pcapfile|_ has no IPv6 decoder. + * if DTLS reassembly is enabled, as it reads :mod:`pcapkit`'s own + DTLS layer. * if |pcapfile|_ has no decoder for the savefile's link layer type, in which case frames are left undecoded. @@ -256,10 +258,19 @@ def run(self) -> 'None': if ext._flag_r and ext._ipv6: ext._ipv6 = False - ext._reasm = ReassemblyManager(ipv4=ext._reasm.ipv4, ipv6=None, tcp=ext._reasm.tcp) + ext._reasm = ReassemblyManager(ipv4=ext._reasm.ipv4, ipv6=None, tcp=ext._reasm.tcp, + dtls=ext._reasm.dtls) warn("'Extractor(engine=pypcapfile)' object does not support IPv6 reassembly; " "so 'ipv6=True' will be ignored", AttributeWarning, stacklevel=stacklevel()) + if ext._flag_r and ext._dtls: + ext._dtls = False + ext._reasm = ReassemblyManager(ipv4=ext._reasm.ipv4, ipv6=ext._reasm.ipv6, + tcp=ext._reasm.tcp, dtls=None) + warn("'Extractor(engine=pypcapfile)' object does not support DTLS reassembly, " + "which reads pcapkit's own DTLS layer; so 'dtls=True' will be ignored", + AttributeWarning, stacklevel=stacklevel()) + stream = NamedStream(ext._ifile, ext._ifnm) sfile = pcapfile.savefile.load_savefile(stream, layers=0, lazy=True) self._dlink = Enum_LinkType.get(sfile.header.ll_type) diff --git a/pcapkit/foundation/engines/pyshark.py b/pcapkit/foundation/engines/pyshark.py index c2e27f91f8..82987f04ca 100644 --- a/pcapkit/foundation/engines/pyshark.py +++ b/pcapkit/foundation/engines/pyshark.py @@ -242,12 +242,13 @@ def run(self) -> 'None': "'context=' is ignored", AttributeWarning, stacklevel=stacklevel()) - if ext._flag_r and (ext._ipv4 or ext._ipv6 or ext._tcp): + if ext._flag_r and (ext._ipv4 or ext._ipv6 or ext._tcp or ext._dtls): ext._flag_r = False logger.debug('pyshark: reassembly unsupported, disabling it') - ext._reasm = ReassemblyManager(ipv4=None, ipv6=None, tcp=None) + ext._reasm = ReassemblyManager(ipv4=None, ipv6=None, tcp=None, dtls=None) warn("'Extractor(engine=pyshark)' object does not support reassembly; " - f"so 'ipv4={ext._ipv4}', 'ipv6={ext._ipv6}' and 'tcp={ext._tcp}' will be ignored", + f"so 'ipv4={ext._ipv4}', 'ipv6={ext._ipv6}', 'tcp={ext._tcp}' and " + f"'dtls={ext._dtls}' will be ignored", AttributeWarning, stacklevel=stacklevel()) # setup verbose handler, unless the caller gave one of their own diff --git a/pcapkit/foundation/engines/scapy.py b/pcapkit/foundation/engines/scapy.py index e6571b7f05..05153d8824 100644 --- a/pcapkit/foundation/engines/scapy.py +++ b/pcapkit/foundation/engines/scapy.py @@ -39,6 +39,7 @@ from typing import TYPE_CHECKING, cast from pcapkit.foundation.engines.engine import EngineBase +from pcapkit.foundation.reassembly import ReassemblyManager from pcapkit.protocols.schema.misc.pcap.frame import RECORD_HEADER_LEN from pcapkit.protocols.schema.misc.pcap.header import RESOLUTION from pcapkit.protocols.schema.misc.pcapng import DEFAULT_RESOLUTION @@ -369,7 +370,8 @@ def run(self) -> 'None': is provided as the Scapy engine does not support such operations; or if :attr:`self.extractor._exctx ` is provided, as the Scapy engine does not parse with :mod:`pcapkit`'s own - protocol implementations. + protocol implementations; or if DTLS reassembly is enabled, for the + same reason. """ ext = self._extractor @@ -385,6 +387,14 @@ def run(self) -> 'None': "'context=' is ignored", AttributeWarning, stacklevel=stacklevel()) + if ext._flag_r and ext._dtls: + ext._dtls = False + ext._reasm = ReassemblyManager(ipv4=ext._reasm.ipv4, ipv6=ext._reasm.ipv6, + tcp=ext._reasm.tcp, dtls=None) + warn("'Extractor(engine=scapy)' object does not support DTLS reassembly, " + "which reads pcapkit's own DTLS layer; so 'dtls=True' will be ignored", + AttributeWarning, stacklevel=stacklevel()) + # setup verbose handler, unless the caller gave one of their own (#1874) if ext._vfunc_default: from pcapkit.toolkit.scapy import packet2chain # isort:skip diff --git a/pcapkit/foundation/extraction.py b/pcapkit/foundation/extraction.py index 39404fb107..25048a7382 100644 --- a/pcapkit/foundation/extraction.py +++ b/pcapkit/foundation/extraction.py @@ -59,6 +59,7 @@ from typing_extensions import Literal from pcapkit.corekit.context import ProtocolContext + from pcapkit.foundation.reassembly.dtls import DTLS as DTLS_Reassembly from pcapkit.foundation.reassembly.ipv4 import IPv4 as IPv4_Reassembly from pcapkit.foundation.reassembly.ipv6 import IPv6 as IPv6_Reassembly from pcapkit.foundation.reassembly.tcp import TCP as TCP_Reassembly @@ -165,6 +166,11 @@ class Extractor(Generic[_P]): ``verbose`` is called on every engine, where the third-party engines replaced it with their own printer (:issue:`1874`). + .. versionchanged:: 1.6.0 + + ``dtls`` is inserted after ``tcp``, so a positional argument from + ``buffer_size`` onwards moves one place along. + """ if TYPE_CHECKING: #: Input file name. @@ -247,6 +253,9 @@ class Extractor(Generic[_P]): #: TCP flag. It indicates if the TCP reassembly and/or flow tracing #: is enabled. _tcp: 'bool' + #: DTLS flag. It indicates if the DTLS handshake-message reassembly is + #: enabled. + _dtls: 'bool' #: Extract til protocol. _exptl: 'Protocols' @@ -339,6 +348,7 @@ class Extractor(Generic[_P]): 'ipv4': ModuleDescriptor('pcapkit.foundation.reassembly.ipv4', 'IPv4'), 'ipv6': ModuleDescriptor('pcapkit.foundation.reassembly.ipv6', 'IPv6'), 'tcp': ModuleDescriptor('pcapkit.foundation.reassembly.tcp', 'TCP'), + 'dtls': ModuleDescriptor('pcapkit.foundation.reassembly.dtls', 'DTLS'), } # type: dict[str, ModuleDescriptor[ReassemblyBase] | Type[ReassemblyBase]] #: Flow tracing support mapping for extracting frames. The values should be a module @@ -423,6 +433,7 @@ def reassembly(self) -> 'ReassemblyData': * ``ipv4`` -- tuple of IPv4 payload fragment (:term:`reasm.ipv4.datagram`) * ``ipv6`` -- tuple of IPv6 payload fragment (:term:`reasm.ipv6.datagram`) * ``tcp`` -- tuple of TCP payload fragment (:term:`reasm.tcp.datagram`) + * ``dtls`` -- tuple of DTLS handshake message (:term:`reasm.dtls.datagram`) Raises: UnsupportedCall: If :attr:`self._flag_r ` @@ -439,6 +450,7 @@ def reassembly(self) -> 'ReassemblyData': ipv4=tuple(self._reasm.ipv4.datagram) if self._ipv4 else None, ipv6=tuple(self._reasm.ipv6.datagram) if self._ipv6 else None, tcp=tuple(self._reasm.tcp.datagram) if self._tcp else None, + dtls=tuple(self._reasm.dtls.datagram) if self._dtls else None, ) return data raise UnsupportedCall("'Extractor(reassembly=False)' object has no attribute 'reassembly'") @@ -1187,6 +1199,7 @@ def __init__(self, trace_byteorder: 'Literal["big", "little"]' = sys.byteorder, trace_nanosecond: 'bool' = False, # trace settings # pylint: disable=line-too-long trace_bidirectional: 'bool' = True, trace_analyse: 'bool' = False, # trace settings # pylint: disable=line-too-long ip: 'bool' = False, ipv4: 'bool' = False, ipv6: 'bool' = False, tcp: 'bool' = False, # reassembly/trace settings # pylint: disable=line-too-long + dtls: 'bool' = False, # reassembly settings # pylint: disable=line-too-long buffer_size: 'int' = io.DEFAULT_BUFFER_SIZE, buffer_save: 'bool' = False, buffer_path: 'Optional[str]' = None, # buffer settings # pylint: disable=line-too-long no_eof: 'bool' = False, # EOF settings # pylint: disable=line-too-long context: 'Optional[ContextRegistry | ProtocolContext | Mapping[str, ProtocolContext] | Iterable[ProtocolContext]]' = None) -> 'None': # context settings # pylint: disable=line-too-long @@ -1221,7 +1234,8 @@ def __init__(self, each protocol's own default -- 60 seconds for IPv4 (:rfc:`1122#section-3.3.2`) and IPv6 (:rfc:`8200#section-4.5`), disabled for TCP, which no specification gives a deadline. Pass - :data:`math.inf` to disable it everywhere + :data:`math.inf` to disable it everywhere; DTLS reassembly takes + none (:rfc:`6347#section-4.2.3`) trace: if trace TCP traffic flows trace_fout: path name for flow tracer if necessary @@ -1242,6 +1256,8 @@ def __init__(self, ipv6: if perform IPv6 reassembly (must be used with ``reassembly=True``) tcp: if perform TCP reassembly and/or flow tracing (must be used with ``reassembly=True`` or ``trace=True``) + dtls: if perform DTLS handshake-message reassembly (must be used + with ``reassembly=True``); the default engine alone supports it buffer_size: buffer size for reading input file (for :class:`~pcapkit.corekit.io.SeekableReader` only) buffer_save: if save buffer to file (for :class:`~pcapkit.corekit.io.SeekableReader` only) @@ -1375,13 +1391,14 @@ def __init__(self, self._ipv4 = ipv4 or ip # IPv4 Reassembly self._ipv6 = ipv6 or ip # IPv6 Reassembly self._tcp = tcp # TCP Reassembly + self._dtls = dtls # DTLS Reassembly self._exptl = protocol or 'null' # extract til protocol self._exlyr = cast('Layers', (layer or 'none').lower()) # extract til layer self._exctx = ContextRegistry.make(context) # caller supplied context if reassembly: - reasm_obj_ipv4 = reasm_obj_ipv6 = reasm_obj_tcp = None + reasm_obj_ipv4 = reasm_obj_ipv6 = reasm_obj_tcp = reasm_obj_dtls = None if self._ipv4: logger.debug('IPv4 reassembly enabled') @@ -1410,11 +1427,21 @@ def __init__(self, self.__reassembly__['tcp'] = reasm_cls_tcp # update mapping upon import reasm_obj_tcp = cast('TCP_Reassembly', reasm_cls_tcp(strict=reasm_strict, store=reasm_store, timeout=reasm_timeout)) + if self._dtls: + logger.debug('DTLS reassembly enabled') + + reasm_cls_dtls = self.__reassembly__['dtls'] + if isinstance(reasm_cls_dtls, ModuleDescriptor): + reasm_cls_dtls = reasm_cls_dtls.klass + self.__reassembly__['dtls'] = reasm_cls_dtls # update mapping upon import + # NOTE: no ``timeout``: the RFC gives DTLS handshake reassembly none. + reasm_obj_dtls = cast('DTLS_Reassembly', reasm_cls_dtls(strict=reasm_strict, store=reasm_store)) self._reasm = ReassemblyManager( ipv4=reasm_obj_ipv4, ipv6=reasm_obj_ipv6, tcp=reasm_obj_tcp, + dtls=reasm_obj_dtls, ) if trace: diff --git a/pcapkit/foundation/reassembly/__init__.py b/pcapkit/foundation/reassembly/__init__.py index a17ce67ec3..1bc7f83d80 100644 --- a/pcapkit/foundation/reassembly/__init__.py +++ b/pcapkit/foundation/reassembly/__init__.py @@ -46,6 +46,8 @@ class ReassemblyManager(Info): ipv6: 'IPv6_Reassembly' #: TCP reassembly. tcp: 'TCP_Reassembly' + #: DTLS handshake-message reassembly. + dtls: 'DTLS_Reassembly' if TYPE_CHECKING: - def __init__(self, ipv4: 'Optional[IPv4_Reassembly]', ipv6: 'Optional[IPv6_Reassembly]', tcp: 'Optional[TCP_Reassembly]') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long + def __init__(self, ipv4: 'Optional[IPv4_Reassembly]', ipv6: 'Optional[IPv6_Reassembly]', tcp: 'Optional[TCP_Reassembly]', dtls: 'Optional[DTLS_Reassembly]') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long diff --git a/pcapkit/foundation/reassembly/data/data.py b/pcapkit/foundation/reassembly/data/data.py index 08aa65ef7e..4c1443953a 100644 --- a/pcapkit/foundation/reassembly/data/data.py +++ b/pcapkit/foundation/reassembly/data/data.py @@ -15,6 +15,7 @@ from pcapkit.const.reg.apptype import TCP as AppType_TCP from pcapkit.const.reg.transtype import TransType + from pcapkit.foundation.reassembly.data.dtls import Datagram as DTLS_Datagram from pcapkit.foundation.reassembly.data.ip import Datagram as IP_Datagram from pcapkit.foundation.reassembly.data.tcp import Datagram as TCP_Datagram from pcapkit.protocols.protocol import ProtocolBase @@ -360,6 +361,8 @@ class ReassemblyData(Info): ipv6: 'tuple[IP_Datagram, ...]' #: TCP reassembled data. tcp: 'tuple[TCP_Datagram, ...]' + #: DTLS reassembled handshake messages. + dtls: 'tuple[DTLS_Datagram, ...]' if TYPE_CHECKING: - def __init__(self, ipv4: 'Optional[tuple[IP_Datagram, ...]]', ipv6: 'Optional[tuple[IP_Datagram, ...]]', tcp: 'Optional[tuple[TCP_Datagram, ...]]') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long + def __init__(self, ipv4: 'Optional[tuple[IP_Datagram, ...]]', ipv6: 'Optional[tuple[IP_Datagram, ...]]', tcp: 'Optional[tuple[TCP_Datagram, ...]]', dtls: 'Optional[tuple[DTLS_Datagram, ...]]') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long diff --git a/pcapkit/foundation/reassembly/data/dtls.py b/pcapkit/foundation/reassembly/data/dtls.py index 95d681aaf3..6716905f41 100644 --- a/pcapkit/foundation/reassembly/data/dtls.py +++ b/pcapkit/foundation/reassembly/data/dtls.py @@ -106,13 +106,8 @@ class Datagram(Info, Generic[_AT]): msg_type: 'Enum_HandshakeType' #: Total message length, as declared by its fragments. length: 'int' - #: Reassembled message body, undecoded -- decoding a ``ClientHello``/ - #: ``ServerHello`` once reassembled is - #: :meth:`DTLS.read `'s job, - #: which alone holds the bound instance :meth:`SSL._read_tls_client_hello - #: ` and - #: :meth:`SSL._read_tls_server_hello - #: ` need. + #: Reassembled message body, undecoded; + #: :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes it. body: 'bytes | tuple[bytes, ...]' if TYPE_CHECKING: diff --git a/pcapkit/foundation/reassembly/dtls.py b/pcapkit/foundation/reassembly/dtls.py index 1bbdc38273..74efe62d81 100644 --- a/pcapkit/foundation/reassembly/dtls.py +++ b/pcapkit/foundation/reassembly/dtls.py @@ -43,10 +43,11 @@ class DTLS(ReassemblyBase[Packet[_AT], Datagram[_AT], BufferID, Buffer], Generic sequence number resets on each cipher-state change (:rfc:`6347#section-4.1`), so a new epoch opens a fresh buffer. Overlapping fragments are accepted - (:rfc:`6347#section-4.2.3`, :rfc:`9147#section-5.5`). Not toolkit-fed: - a protocol instance never sees the addresses, so a caller supplies the - 4-tuple as ``flow`` to :meth:`DTLS.read - `. + (:rfc:`6347#section-4.2.3`, :rfc:`9147#section-5.5`). The + :class:`~pcapkit.foundation.extraction.Extractor` feeds it through + :func:`pcapkit.toolkit.pcap.dtls_reassembly`, which reads the 4-tuple + off the frame, and :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` + decodes what it reassembles. Example: >>> from pcapkit.foundation.reassembly.dtls import DTLS @@ -106,12 +107,11 @@ def last(self) -> 'Optional[Datagram[_AT]]': one. This is a synchronous alternative to :attr:`datagram`/:meth:`fetch` - for a caller -- :meth:`DTLS.read - ` is the one that - exists -- that wants to know, immediately after feeding one fragment, - whether *that* fragment was the one that completed its message, - rather than waiting for the buffer (which may hold several other - messages still in flight under the same buffer ID) to be flushed. + for a caller that wants to know, immediately after feeding one + fragment, whether *that* fragment was the one that completed its + message, rather than waiting for the buffer (which may hold several + other messages still in flight under the same buffer ID) to be + flushed. """ return self._last diff --git a/pcapkit/interface/core.py b/pcapkit/interface/core.py index 35e9a1e4d5..967fec351d 100644 --- a/pcapkit/interface/core.py +++ b/pcapkit/interface/core.py @@ -82,6 +82,7 @@ def extract(fin: 'Optional[str | IO[bytes]]' = None, fout: 'Optional[str]' = Non trace_byteorder: 'Literal["big", "little"]' = sys.byteorder, trace_nanosecond: 'bool' = False, # trace settings # pylint: disable=line-too-long trace_bidirectional: 'bool' = True, trace_analyse: 'bool' = False, # trace settings # pylint: disable=line-too-long ip: 'bool' = False, ipv4: 'bool' = False, ipv6: 'bool' = False, tcp: 'bool' = False, # reassembly/trace settings # pylint: disable=line-too-long + dtls: 'bool' = False, # reassembly settings # pylint: disable=line-too-long buffer_size: 'int' = io.DEFAULT_BUFFER_SIZE, buffer_save: 'bool' = False, buffer_path: 'Optional[str]' = None, # buffer settings # pylint: disable=line-too-long no_eof: 'bool' = False, # EOF settings # pylint: disable=line-too-long context: 'Optional[ContextRegistry | ProtocolContext | Mapping[str, ProtocolContext] | Iterable[ProtocolContext]]' = None) -> 'Extractor': # context settings # pylint: disable=line-too-long @@ -113,7 +114,7 @@ def extract(fin: 'Optional[str | IO[bytes]]' = None, fout: 'Optional[str]' = Non reasm_store: if store reassembled datagrams reasm_timeout: reassembly timeout in seconds, on the capture's own clock; :data:`None` selects each protocol's default (60 seconds for - IPv4 and IPv6, disabled for TCP) + IPv4 and IPv6, disabled for TCP); DTLS reassembly takes none trace: if trace TCP traffic flows trace_fout: path name for flow tracer if necessary @@ -130,6 +131,8 @@ def extract(fin: 'Optional[str | IO[bytes]]' = None, fout: 'Optional[str]' = Non ipv6: if perform IPv6 reassembly (must be used with ``reassembly=True``) tcp: if perform TCP reassembly and/or flow tracing (must be used with ``reassembly=True`` or ``trace=True``) + dtls: if perform DTLS handshake-message reassembly (must be used with + ``reassembly=True``); the default engine alone supports it buffer_size: buffer size for reading input file (for :class:`~pcapkit.corekit.io.SeekableReader` only) buffer_save: if save buffer to file (for :class:`~pcapkit.corekit.io.SeekableReader` only) @@ -172,6 +175,11 @@ def extract(fin: 'Optional[str | IO[bytes]]' = None, fout: 'Optional[str]' = Non keyword. Flows are traced bidirectionally by default, and IPv4 and IPv6 reassembly buffers expire after 60 seconds of capture time. + .. versionchanged:: 1.6.0 + + ``dtls`` is inserted after ``tcp``, so a positional argument from + ``buffer_size`` onwards moves one place along. + """ if isinstance(layer, type) and issubclass(layer, ProtocolBase): layer = (layer.__layer__ or 'none').lower() # type: ignore[assignment] @@ -180,7 +188,7 @@ def extract(fin: 'Optional[str | IO[bytes]]' = None, fout: 'Optional[str]' = Non store=store, files=files, nofile=nofile, auto=auto, verbose=verbose, extension=extension, engine=engine, layer=layer, protocol=protocol, # type: ignore[arg-type] - ip=ip, ipv4=ipv4, ipv6=ipv6, tcp=tcp, + ip=ip, ipv4=ipv4, ipv6=ipv6, tcp=tcp, dtls=dtls, reassembly=reassembly, reasm_store=reasm_store, reasm_strict=reasm_strict, reasm_timeout=reasm_timeout, trace=trace, trace_fout=trace_fout, trace_format=trace_format, diff --git a/pcapkit/protocols/application/dtls.py b/pcapkit/protocols/application/dtls.py index 327127e23c..2c6018b3ac 100644 --- a/pcapkit/protocols/application/dtls.py +++ b/pcapkit/protocols/application/dtls.py @@ -16,13 +16,15 @@ Known Limitations ----------------- -**A handshake message split across DTLS records decodes only with -reassembly** (:rfc:`6347#section-4.2.3`, :issue:`2440`). Otherwise its body -stays raw octets, flagged +**A handshake message split across DTLS records is not decoded here** +(:rfc:`6347#section-4.2.3`, :issue:`2440`). Each fragment's body stays raw +octets, flagged :attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.is_fragmented`. -Pass a :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine as -``reassembly`` to decode the completing fragment. The caller supplies the UDP -4-tuple as ``flow``. Only the first record of each datagram is reassembled. +``Extractor(reassembly=True, dtls=True)`` reassembles the message +(:class:`~pcapkit.foundation.reassembly.dtls.DTLS`), and +:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes it with +:meth:`DTLS.read_handshake`. Only the first record of each datagram is +reassembled. **A unified-header record with a connection ID is not decoded** (:rfc:`9147#section-4.1`). Its length is negotiated, not carried in the @@ -39,11 +41,9 @@ from typing import TYPE_CHECKING, cast from pcapkit.const.tls.content_type import ContentType as Enum_ContentType -from pcapkit.const.tls.handshake_type import HandshakeType as Enum_HandshakeType from pcapkit.protocols.application.ssl import DTLS13_VERSION, SSL from pcapkit.protocols.data.application.dtls import DTLS as Data_DTLS from pcapkit.protocols.data.application.dtls import DTLSCiphertext as Data_DTLSCiphertext -from pcapkit.protocols.data.application.dtls import DTLSHandshake as Data_DTLSHandshake from pcapkit.protocols.schema.application.dtls import DTLS as Schema_DTLS from pcapkit.protocols.schema.application.dtls import DTLSCiphertext as Schema_DTLSCiphertext from pcapkit.utilities.exceptions import ProtocolError @@ -55,7 +55,7 @@ from aenum import IntEnum as AenumEnum from typing_extensions import Literal - from pcapkit.foundation.reassembly.dtls import DTLS as DTLS_Reassembly + from pcapkit.protocols.data.application.dtls import DTLSHandshake as Data_DTLSHandshake from pcapkit.protocols.protocol import ProtocolBase from pcapkit.protocols.schema.schema import Schema @@ -166,25 +166,11 @@ def id(cls) -> 'tuple[Literal["DTLS"]]': # type: ignore[override] """ return ('DTLS',) - def read(self, length: 'Optional[int]' = None, *, reassembly: 'Optional[DTLS_Reassembly[Any]]' = None, - flow: 'Optional[tuple[Any, int, Any, int]]' = None, frame_num: 'int' = 0, - frame_timestamp: 'float' = 0.0, **kwargs: 'Any') -> 'Data_DTLS': + def read(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLS': """Read Datagram Transport Layer Security (DTLS). Args: length: Length of packet data. - reassembly: A :class:`~pcapkit.foundation.reassembly.dtls.DTLS` - engine to feed fragmented handshake messages to; the fragment - that completes a message is decoded in place. :data:`None` - leaves fragments undecoded. - flow: ``(src, sport, dst, dport)`` of the UDP association, which - with the record's epoch keys the reassembly buffer. Required - when ``reassembly`` is given, so that interleaved associations - do not share a buffer. - frame_num: Packet number, recorded on the reassembled message's - :attr:`~pcapkit.foundation.reassembly.data.dtls.Datagram.index`. - frame_timestamp: Capture timestamp in seconds, recorded on the - fragment; reassembly does not expire buffers by time. **kwargs: Arbitrary keyword arguments. Returns: @@ -192,12 +178,10 @@ def read(self, length: 'Optional[int]' = None, *, reassembly: 'Optional[DTLS_Rea Raises: ProtocolError: If the record cannot be a genuine DTLS record -- - see :meth:`_read_dtls_record` -- or if ``reassembly`` is - given without ``flow``. + see :meth:`_read_dtls_record`. """ - return self._read_dtls_record(length, reassembly=reassembly, flow=flow, frame_num=frame_num, - frame_timestamp=frame_timestamp, **kwargs) + return self._read_dtls_record(length, **kwargs) def make(self, content_type: 'Enum_ContentType | StdlibEnum | AenumEnum | str | int' = Enum_ContentType.application_data, # pylint: disable=line-too-long @@ -306,6 +290,33 @@ def unpack(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLS return self.read(length, **kwargs) return super().unpack(length, **kwargs) + @classmethod + def read_handshake(cls, message: 'bytes') -> 'Data_DTLSHandshake': + """Read one handshake message from its own octets, outside any record. + + A message's ``length`` is 24-bit (:rfc:`6347#section-4.2.2`) but a + record's is 16-bit, so a message reassembled from its fragments may be + too long for any record to carry to :meth:`read`. + + Args: + message: The handshake header, fragment header and body. + + Returns: + The message, decoded as :meth:`read` decodes one in a record. + + Raises: + ProtocolError: If ``message`` is not exactly one handshake message, + or its body is malformed for its ``msg_type``. + + """ + # NOTE: a bare instance, for the ``alias`` the shared decoders name in an error. + self = cls.__new__(cls) + messages = self._read_tls_handshake_messages(message, dtls=True) + if len(messages) != 1: + raise ProtocolError(f'{self.alias}: invalid format: {len(messages)} handshake ' + 'messages where exactly one is expected') + return cast('Data_DTLSHandshake', messages[0]) + ########################################################################## # Data models. ########################################################################## @@ -318,118 +329,6 @@ def __length_hint__(self) -> 'Literal[13]': # Utilities. ########################################################################## - def _reassemble_dtls_handshake(self, content: 'tuple[Any, ...]', reassembly: 'DTLS_Reassembly[Any]', # pylint: disable=too-many-arguments - flow: 'Optional[tuple[Any, int, Any, int]]', epoch: 'int', - frame_num: 'int', frame_timestamp: 'float') -> 'tuple[Any, ...]': - """Feed a ``handshake`` record's fragmented messages to ``reassembly`` - and decode whichever one it just completed. - - Args: - content: This record's decoded ``handshake`` content, as - :meth:`SSL._read_tls_handshake_messages - ` - built it -- a tuple of - :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`. - reassembly: Engine to reassemble a fragmented message with. - flow: Buffer key, as documented on :meth:`read`, which - enforces this unconditionally before this method is ever - reached -- the check below is only for a direct call that - bypasses it. Required: :data:`None` would default every - caller that omits it onto the same buffer ID, corrupting two - interleaved associations' reassembly together. - epoch: This record's epoch, appended to ``flow`` to form the - reassembly buffer ID (:rfc:`6347#section-4.1`). - frame_num: This record's packet number. - frame_timestamp: This record's capture timestamp. - - Returns: - ``content``, with a message that :meth:`reassembly` completes - decoded as a ``ClientHello`` or ``ServerHello`` in place of its raw - octets; any other message is unchanged. A completed message that - does not decode keeps its raw reassembled bytes as ``body`` - instead of raising. - - Raises: - ProtocolError: If ``flow`` is :data:`None`. - - """ - # NOTE: imported here, as pcapkit.foundation.reassembly imports pcapkit.protocols, - # whose package import reaches this module (a cycle at import time). - from pcapkit.foundation.reassembly.data.dtls import \ - Packet as _ReasmPacket # pylint: disable=import-outside-toplevel - - if flow is None: - raise ProtocolError(f'{self.alias}: flow is required when reassembly is given -- ' - 'defaulting it would collide two concurrent DTLS associations ' - 'on the same buffer ID') - bufid = (*flow, epoch) - - messages = [] - for message in content: - if not isinstance(message, Data_DTLSHandshake) or not message.is_fragmented: - messages.append(message) - continue - - # A fragmented message's body is always raw bytes -- - # :meth:`SSL._read_tls_handshake_messages - # ` - # only decodes one when it is *not* fragmented -- so this is a type - # narrowing for mypy's benefit, not a runtime check. - body = cast('bytes', message.body) - reassembly(_ReasmPacket( - bufid=bufid, - num=frame_num, - msg_type=message.msg_type, - length=message.length, - message_seq=message.message_seq, - fragment_offset=message.fragment_offset, - fragment_length=message.fragment_length, - payload=body, - timestamp=frame_timestamp, - )) - - datagram = reassembly.last - if datagram is None or not datagram.completed: - messages.append(message) - continue - - # A malformed record may have ``fragment_offset + fragment_length`` - # past ``length`` (:rfc:`6347#section-4.2.3`). Then - # :meth:`SSL._make_tls_handshake_messages` cannot reliably slice its - # share back out of the reassembled body, so keep the raw bytes. - if message.fragment_offset + message.fragment_length > message.length: - messages.append(message) - continue - - reassembled = datagram.body - if isinstance(reassembled, tuple): # pragma: no cover -- COMPLETE never reports runs - reassembled = b''.join(reassembled) - - try: - if message.msg_type == Enum_HandshakeType.client_hello: - decoded = self._read_tls_client_hello(reassembled, dtls=True) # type: Any - elif message.msg_type == Enum_HandshakeType.server_hello: - decoded = self._read_tls_server_hello(reassembled) - else: - decoded = reassembled - except ProtocolError: - # The reassembled bytes do not decode as a valid message of - # this ``msg_type`` -- keep them raw rather than failing the - # whole record; without ``reassembly`` the same bytes would - # have stayed undecoded too; see :meth:`read`'s docstring. - decoded = reassembled - - messages.append(Data_DTLSHandshake( - msg_type=message.msg_type, - length=message.length, - body=decoded, - message_seq=message.message_seq, - fragment_offset=message.fragment_offset, - fragment_length=message.fragment_length, - is_fragmented=message.is_fragmented, - )) - return tuple(messages) - @classmethod def _make_data(cls, data: 'Data_DTLS | Data_DTLSCiphertext') -> 'dict[str, Any]': # type: ignore[override] """Create key-value pairs from ``data`` for protocol construction. @@ -460,10 +359,7 @@ def _make_data(cls, data: 'Data_DTLS | Data_DTLSCiphertext') -> 'dict[str, Any]' 'payload': cls._make_payload(data), } - def _read_dtls_record(self, length: 'Optional[int]' = None, *, # pylint: disable=unused-argument - reassembly: 'Optional[DTLS_Reassembly[Any]]' = None, - flow: 'Optional[tuple[Any, int, Any, int]]' = None, frame_num: 'int' = 0, - frame_timestamp: 'float' = 0.0, **kwargs: 'Any') -> 'Data_DTLS | Data_DTLSCiphertext': + def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLS | Data_DTLSCiphertext': # pylint: disable=unused-argument """Read a DTLS record, identifying its concrete tier as it goes. A unified-header record is handed to :meth:`_read_dtls_ciphertext`. @@ -489,10 +385,6 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, *, # pylint: disabl Args: length: Length of packet data. - reassembly: Reassembly engine, as documented on :meth:`read`. - flow: Reassembly buffer key, as documented on :meth:`read`. - frame_num: Packet number, as documented on :meth:`read`. - frame_timestamp: Capture timestamp, as documented on :meth:`read`. **kwargs: Arbitrary keyword arguments. Returns: @@ -503,8 +395,7 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, *, # pylint: disabl record -- an unrecognised content type, a legacy version outside DTLS's range, or a declared ``length`` that runs past the end of the buffer -- or if the record fragment is - malformed for its content type; or if ``reassembly`` is given - without ``flow``. + malformed for its content type. """ if isinstance(self.__header__, Schema_DTLSCiphertext): @@ -514,13 +405,6 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, *, # pylint: disabl available = length if length is not None else len(self) self._validate_tls_record(schema, available=available, dtls=True) - # NOTE: checked for every content type, not only handshake records that reach - # _reassemble_dtls_handshake, so interleaved associations never share a buffer. - if reassembly is not None and flow is None: - raise ProtocolError(f'{self.alias}: flow is required when reassembly is given -- ' - 'defaulting it would collide two concurrent DTLS associations ' - 'on the same buffer ID') - # NOTE: DTLS 1.3 is checked first, at 0xFEFF too: an initial ClientHello may carry # {254, 255} (RFC 9147, section 4). Unknown versions keep the generic alias. if schema.version in _DTLS13_LEGACY_VERSIONS and self._tls_negotiates(schema, DTLS13_VERSION, dtls=True): @@ -534,12 +418,6 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, *, # pylint: disabl content = self._read_tls_content(schema.content_type, schema.fragment, dtls=True, epoch=schema.epoch) - # NOTE: a handshake record past epoch 0 is ciphertext, so ``content`` is raw - # bytes rather than a tuple of messages, and is passed through unchanged. - if (reassembly is not None and schema.content_type == Enum_ContentType.handshake - and isinstance(content, tuple)): - content = self._reassemble_dtls_handshake(content, reassembly, flow, schema.epoch, - frame_num, frame_timestamp) dtls = Data_DTLS( content_type=schema.content_type, version=schema.version, diff --git a/pcapkit/protocols/application/ssl.py b/pcapkit/protocols/application/ssl.py index 56d8c120bc..d87023843d 100644 --- a/pcapkit/protocols/application/ssl.py +++ b/pcapkit/protocols/application/ssl.py @@ -424,12 +424,10 @@ def _read_tls_handshake_messages(self, fragment: 'bytes', *, ``fragment``. Notes: - Only a whole ``ClientHello``/``ServerHello`` body is decoded: an - unfragmented one, or a fragmented DTLS message completed by a - :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine passed as - :meth:`DTLS.read `'s - ``reassembly`` (:issue:`2440`). Other bodies stay undecoded - (:issue:`2437`). + Only an unfragmented ``ClientHello``/``ServerHello`` body is + decoded; other bodies stay undecoded (:issue:`2437`). A fragmented + DTLS message is decoded once reassembled, by + :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` (:issue:`2440`). """ from pcapkit.protocols.data.application.dtls import \ @@ -487,13 +485,20 @@ def _make_tls_handshake_messages(self, messages: 'tuple[Any, ...]', *, Args: messages: Handshake messages, as returned by :meth:`_read_tls_handshake_messages`, or already-encoded - :obj:`bytes` per message. + :obj:`bytes` per message. A fragmented DTLS message's body is + its fragment's octets, or the whole message, of which only the + fragment's range is written. dtls: Whether to write DTLS's fragment header after each message's ``msg_type``/``length`` header. Returns: Encoded record fragment. + Raises: + ProtocolError: If a fragmented DTLS message's body does not yield + exactly ``fragment_length`` octets, which a fragment carries + (:rfc:`6347#section-4.2.3`). + """ out = bytearray() for message in messages: @@ -516,13 +521,15 @@ def _make_tls_handshake_messages(self, messages: 'tuple[Any, ...]', *, if is_fragmented: fragment_length = message.fragment_length - # ``body_bytes`` is this fragment's own octets, unless decoding the - # completing fragment replaced it with the full reassembled message - # (see :meth:`DTLS._reassemble_dtls_handshake`). Slice back to this - # record's own byte range so parse-then-rebuild stays byte-exact. - if len(body_bytes) != fragment_length: + # NOTE: a fragment's body is exactly ``fragment_length`` octets (RFC 6347, + # section 4.2.3); a whole-message body is cut down to this fragment's range. + if len(body_bytes) != fragment_length and len(body_bytes) == length: start = message.fragment_offset body_bytes = body_bytes[start:start + fragment_length] + if len(body_bytes) != fragment_length: + raise ProtocolError(f'{self.alias}: invalid format: handshake fragment body ' + f'of {len(body_bytes)} octets, fragment_length ' + f'{fragment_length}') else: fragment_length = len(body_bytes) diff --git a/pcapkit/protocols/data/application/dtls.py b/pcapkit/protocols/data/application/dtls.py index 6dc9459c1f..c5c9bb0101 100644 --- a/pcapkit/protocols/data/application/dtls.py +++ b/pcapkit/protocols/data/application/dtls.py @@ -34,12 +34,11 @@ class DTLSHandshake(Handshake): :attr:`body` decodes to a :class:`~pcapkit.protocols.data.application.tls.ClientHello` or :class:`~pcapkit.protocols.data.application.tls.ServerHello` when - :attr:`is_fragmented` is :data:`False`, or when a - :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine passed to - :meth:`DTLS.read ` completed - the message on this fragment (:issue:`2440`). Otherwise a fragmented - message's :attr:`body` is the fragment's raw octets. :attr:`is_fragmented` - reports whether the message was split on the wire, either way. + :attr:`is_fragmented` is :data:`False`. A fragmented message's :attr:`body` + is the fragment's raw octets; + :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes the + message once :class:`~pcapkit.foundation.reassembly.dtls.DTLS` has + reassembled it (:issue:`2440`). """ diff --git a/pcapkit/toolkit/__init__.py b/pcapkit/toolkit/__init__.py index 5869b0914c..b102704afa 100644 --- a/pcapkit/toolkit/__init__.py +++ b/pcapkit/toolkit/__init__.py @@ -10,10 +10,12 @@ """ # tools for default engine -from pcapkit.toolkit.pcap import ipv4_reassembly, ipv6_reassembly, tcp_reassembly, tcp_traceflow +from pcapkit.toolkit.pcap import (dtls_reassembly, ipv4_reassembly, ipv6_reassembly, tcp_reassembly, + tcp_traceflow) from pcapkit.toolkit.pcapng import ipv4_reassembly as pcapng_ipv4_reassembly from pcapkit.toolkit.pcapng import ipv6_reassembly as pcapng_ipv6_reassembly from pcapkit.toolkit.pcapng import tcp_reassembly as pcapng_tcp_reassembly +from pcapkit.toolkit.pcapng import dtls_reassembly as pcapng_dtls_reassembly from pcapkit.toolkit.pcapng import tcp_traceflow as pcapng_tcp_traceflow from pcapkit.toolkit.pcapng import block2frame as pcapng_block2frame @@ -57,8 +59,9 @@ __all__ = [ # default engine - 'ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'tcp_traceflow', - 'pcapng_ipv4_reassembly', 'pcapng_ipv6_reassembly', 'pcapng_tcp_reassembly', 'pcapng_tcp_traceflow', + 'ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'dtls_reassembly', 'tcp_traceflow', + 'pcapng_ipv4_reassembly', 'pcapng_ipv6_reassembly', 'pcapng_tcp_reassembly', 'pcapng_dtls_reassembly', + 'pcapng_tcp_traceflow', 'pcapng_block2frame', # # DPKT engine diff --git a/pcapkit/toolkit/pcap.py b/pcapkit/toolkit/pcap.py index a43fcd5633..255f7ad223 100644 --- a/pcapkit/toolkit/pcap.py +++ b/pcapkit/toolkit/pcap.py @@ -9,16 +9,22 @@ :mod:`pcapkit.foundation` counterpart consumes, or :data:`None` if the frame cannot be used for it. The TCP adapters here and in :mod:`pcapkit.toolkit.pcapng` read the segment with -:func:`~pcapkit.toolkit.pcap.tcp_segment`. +:func:`~pcapkit.toolkit.pcap.tcp_segment`, and the DTLS ones the record with +:func:`~pcapkit.toolkit.pcap._dtls_fragments`. """ from typing import TYPE_CHECKING, NamedTuple, cast from pcapkit.const.ipv6.extension_header import ExtensionHeader as Enum_ExtensionHeader from pcapkit.const.reg.transtype import TransType as Enum_TransType +from pcapkit.const.tls.content_type import ContentType as Enum_ContentType +from pcapkit.foundation.reassembly.data.dtls import Packet as DTLS_Packet from pcapkit.foundation.reassembly.data.ip import Packet as IP_Packet from pcapkit.foundation.reassembly.data.tcp import Packet as TCP_Packet from pcapkit.foundation.traceflow.data.tcp import Packet as TF_TCP_Packet +from pcapkit.protocols.application.dtls import DTLS +from pcapkit.protocols.data.application.dtls import DTLS as Data_DTLS +from pcapkit.protocols.data.application.dtls import DTLSHandshake as Data_DTLSHandshake from pcapkit.protocols.internet.ip import IP as Protocol_IP from pcapkit.protocols.internet.ipv4 import IPv4 from pcapkit.protocols.internet.ipv6 import IPv6 @@ -27,6 +33,7 @@ from pcapkit.protocols.protocol import ProtocolBase from pcapkit.protocols.schema.transport.tcp import TCP_DATA_OFFSET_OCTET, TCP_MIN_HEADER_LEN from pcapkit.protocols.transport.tcp import TCP +from pcapkit.protocols.transport.udp import UDP if TYPE_CHECKING: from ipaddress import IPv4Address, IPv6Address @@ -38,8 +45,8 @@ from pcapkit.protocols.internet.ipv6_frag import IPv6_Frag from pcapkit.protocols.misc.pcap import Frame -__all__ = ['ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'tcp_traceflow', - 'tcp_segment', 'TCPSegment'] +__all__ = ['ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'dtls_reassembly', + 'tcp_traceflow', 'tcp_segment', 'TCPSegment'] class TCPSegment(NamedTuple): @@ -382,6 +389,92 @@ def tcp_reassembly(frame: 'Frame') -> 'TCP_Packet | None': return None +def _dtls_fragments(frame: 'ProtocolBase', *, num: 'int', + timestamp: 'float') -> 'Optional[tuple[DTLS_Packet, ...]]': + """Make data for DTLS handshake-message reassembly from a frame's DTLS record. + + The DTLS adapters of this module and of :mod:`pcapkit.toolkit.pcapng` read + the record with this. + + Args: + frame: PCAP frame or PCAP-NG block. + num: Original packet range number. + timestamp: Capture timestamp, in seconds since the Unix epoch. + + Returns: + One record per fragmented handshake message in the frame's first + :class:`~pcapkit.protocols.application.dtls.DTLS` layer, if UDP carries + it and it is a cleartext ``handshake`` record holding one; otherwise + :data:`None`. The buffer ID is the IP and UDP endpoints nearest above + the record, with its epoch (:rfc:`6347#section-4.1`). + + """ + ip = prev = None # type: Optional[ProtocolBase] + layer = frame # type: ProtocolBase + while isinstance(layer, ProtocolBase) and not isinstance(layer, (NoPayload, DTLS)): + if isinstance(layer, Protocol_IP): + ip = layer + prev, layer = layer, layer.payload + if not isinstance(layer, DTLS) or not isinstance(prev, UDP) or ip is None: + return None + + dtls_info = layer.info + # NOTE: a record past epoch 0 is ciphertext, so its content is bytes, not messages. + if (not isinstance(dtls_info, Data_DTLS) or dtls_info.content_type != Enum_ContentType.handshake + or not isinstance(dtls_info.content, tuple)): + return None + + ip_info = cast('Data_IPv4 | Data_IPv6', ip.info) + udp_info = prev.info + bufid = ( + ip_info.src, # source IP address + udp_info.srcport.port, # source port + ip_info.dst, # destination IP address + udp_info.dstport.port, # destination port + dtls_info.epoch, # DTLS epoch + ) + packets = tuple( + DTLS_Packet( # type: ignore[type-var] + bufid=bufid, # buffer identifier + num=num, # original packet range number + msg_type=message.msg_type, # handshake message type + length=message.length, # total message length + message_seq=message.message_seq, # message sequence number + fragment_offset=message.fragment_offset, # fragment offset + fragment_length=message.fragment_length, # fragment length + payload=cast('bytes', message.body), # raw bytes type fragment body + timestamp=timestamp, # capture timestamp + ) + for message in dtls_info.content + if isinstance(message, Data_DTLSHandshake) and message.is_fragmented + ) + return packets or None + + +def dtls_reassembly(frame: 'Frame') -> 'Optional[tuple[DTLS_Packet, ...]]': + """Make data for DTLS handshake-message reassembly. + + A frame can be reassembled if UDP carries a DTLS layer + (:class:`~pcapkit.protocols.application.dtls.DTLS`) whose cleartext + ``handshake`` record holds a fragmented message + (:attr:`DTLSHandshake.is_fragmented + `, + :rfc:`6347#section-4.2.3`). Only the frame's first record is read. + + Args: + frame: PCAP frame. + + Returns: + Data for DTLS reassembly (:term:`reasm.dtls.packet`), one per fragmented + message, if the ``frame`` can be reassembled; otherwise :data:`None`. + + See Also: + :class:`pcapkit.foundation.reassembly.dtls.DTLS` + + """ + return _dtls_fragments(frame, num=frame.info.number, timestamp=float(frame.info.time_epoch)) + + def tcp_traceflow(frame: 'Frame', *, data_link: 'LinkType') -> 'TF_TCP_Packet | None': """Trace packet flow for TCP. diff --git a/pcapkit/toolkit/pcapng.py b/pcapkit/toolkit/pcapng.py index 3523f535b0..255493d2dd 100644 --- a/pcapkit/toolkit/pcapng.py +++ b/pcapkit/toolkit/pcapng.py @@ -21,16 +21,17 @@ from pcapkit.protocols.internet.ipv4 import IPv4 from pcapkit.protocols.internet.ipv6 import IPv6 from pcapkit.protocols.schema.misc.pcap.header import RESOLUTION -from pcapkit.toolkit.pcap import _fragment, tcp_segment +from pcapkit.toolkit.pcap import _dtls_fragments, _fragment, tcp_segment if TYPE_CHECKING: from ipaddress import IPv4Address, IPv6Address from typing import Optional + from pcapkit.foundation.reassembly.data.dtls import Packet as DTLS_Packet from pcapkit.protocols.internet.ipv6_frag import IPv6_Frag from pcapkit.protocols.misc.pcapng import PCAPNG, Packet -__all__ = ['ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', +__all__ = ['ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'dtls_reassembly', 'tcp_traceflow', 'block2frame'] @@ -190,6 +191,32 @@ def tcp_reassembly(frame: 'PCAPNG') -> 'TCP_Packet | None': return None +def dtls_reassembly(frame: 'PCAPNG') -> 'Optional[tuple[DTLS_Packet, ...]]': + """Make data for DTLS handshake-message reassembly. + + A frame can be reassembled if UDP carries a DTLS layer + (:class:`~pcapkit.protocols.application.dtls.DTLS`) whose cleartext + ``handshake`` record holds a fragmented message + (:attr:`DTLSHandshake.is_fragmented + `, + :rfc:`6347#section-4.2.3`). Only the frame's first record is read (see + :func:`pcapkit.toolkit.pcap._dtls_fragments`). + + Args: + frame: PCAP-NG frame. + + Returns: + Data for DTLS reassembly (:term:`reasm.dtls.packet`), one per fragmented + message, if the ``frame`` can be reassembled; otherwise :data:`None`. + + See Also: + :class:`pcapkit.foundation.reassembly.dtls.DTLS` + + """ + frame_info = cast('Packet', frame.info) + return _dtls_fragments(frame, num=frame_info.number, timestamp=float(frame_info.timestamp_epoch)) + + def tcp_traceflow(frame: 'PCAPNG', *, nanosecond: 'bool' = False) -> 'TF_TCP_Packet | None': """Trace packet flow for TCP. diff --git a/tests/foundation/engines/test_pcap_ct_engine.py b/tests/foundation/engines/test_pcap_ct_engine.py index f7337d238a..e248a1cd52 100644 --- a/tests/foundation/engines/test_pcap_ct_engine.py +++ b/tests/foundation/engines/test_pcap_ct_engine.py @@ -144,7 +144,8 @@ def setUp(self) -> None: def make_extractor(self, **overrides): sink = OutputSink() - reasm = types.SimpleNamespace(ipv4=mock.Mock(), ipv6=mock.Mock(), tcp=mock.Mock()) + reasm = types.SimpleNamespace(ipv4=mock.Mock(), ipv6=mock.Mock(), tcp=mock.Mock(), + dtls=mock.Mock()) trace = types.SimpleNamespace(tcp=mock.Mock()) values = { '_ifile': io.BytesIO(b'capture'), @@ -163,6 +164,7 @@ def make_extractor(self, **overrides): '_ipv4': False, '_ipv6': False, '_tcp': False, + '_dtls': False, '_reasm': reasm, '_trace': trace, '_frame': [], @@ -452,8 +454,25 @@ def test_run_disables_reassembly_and_flow_tracing(self) -> None: self.assertIsNone(extractor._reasm.ipv4) self.assertIsNone(extractor._reasm.ipv6) self.assertIsNone(extractor._reasm.tcp) + self.assertIsNone(extractor._reasm.dtls) self.assertIsNone(extractor._trace.tcp) + def test_run_disables_reassembly_requested_for_dtls_alone(self) -> None: + from pcapkit.utilities.warnings import AttributeWarning + + extractor, _ = self.make_extractor(_flag_r=True, _dtls=True) + engine = self.engine(extractor) + with mock.patch.object(engine._expkg, 'pcap', return_value=FakeHandle()): + with mock.patch('pcapkit.foundation.engines.pcap_ct.warn') as warn: + engine.run() + + messages = [call.args[0] for call in warn.call_args_list + if call.args[1] is AttributeWarning] + self.assertTrue(any("'dtls=True' will be ignored" in message for message in messages), + messages) + self.assertFalse(extractor._flag_r) + self.assertIsNone(extractor._reasm.dtls) + def test_run_leaves_disabled_reassembly_and_tracing_untouched(self) -> None: extractor, _ = self.make_extractor(_flag_r=True, _flag_t=True) original_reasm, original_trace = extractor._reasm, extractor._trace diff --git a/tests/foundation/engines/test_pypcap_engine.py b/tests/foundation/engines/test_pypcap_engine.py index 89a56ddbb9..1f037ea8de 100644 --- a/tests/foundation/engines/test_pypcap_engine.py +++ b/tests/foundation/engines/test_pypcap_engine.py @@ -87,7 +87,8 @@ def setUp(self) -> None: def make_extractor(self, **overrides): sink = OutputSink() - reasm = types.SimpleNamespace(ipv4=mock.Mock(), ipv6=mock.Mock(), tcp=mock.Mock()) + reasm = types.SimpleNamespace(ipv4=mock.Mock(), ipv6=mock.Mock(), tcp=mock.Mock(), + dtls=mock.Mock()) trace = types.SimpleNamespace(tcp=mock.Mock()) values = { '_ifile': io.BytesIO(b'capture'), @@ -106,6 +107,7 @@ def make_extractor(self, **overrides): '_ipv4': False, '_ipv6': False, '_tcp': False, + '_dtls': False, '_reasm': reasm, '_trace': trace, '_frame': [], @@ -368,8 +370,25 @@ def test_run_disables_reassembly_and_flow_tracing(self) -> None: self.assertIsNone(extractor._reasm.ipv4) self.assertIsNone(extractor._reasm.ipv6) self.assertIsNone(extractor._reasm.tcp) + self.assertIsNone(extractor._reasm.dtls) self.assertIsNone(extractor._trace.tcp) + def test_run_disables_reassembly_requested_for_dtls_alone(self) -> None: + from pcapkit.utilities.warnings import AttributeWarning + + extractor, _ = self.make_extractor(_flag_r=True, _dtls=True) + engine = self.engine(extractor) + with mock.patch.object(engine._expkg, 'pcap', return_value=FakeHandle()): + with mock.patch('pcapkit.foundation.engines.pypcap.warn') as warn: + engine.run() + + messages = [call.args[0] for call in warn.call_args_list + if call.args[1] is AttributeWarning] + self.assertTrue(any("'dtls=True' will be ignored" in message for message in messages), + messages) + self.assertFalse(extractor._flag_r) + self.assertIsNone(extractor._reasm.dtls) + def test_run_leaves_disabled_reassembly_and_tracing_untouched(self) -> None: extractor, _ = self.make_extractor(_flag_r=True, _flag_t=True) original_reasm, original_trace = extractor._reasm, extractor._trace diff --git a/tests/foundation/engines/test_pypcapfile_engine.py b/tests/foundation/engines/test_pypcapfile_engine.py index a5b69865c4..ade9bd3820 100644 --- a/tests/foundation/engines/test_pypcapfile_engine.py +++ b/tests/foundation/engines/test_pypcapfile_engine.py @@ -171,7 +171,8 @@ def setUp(self) -> None: def make_extractor(self, **overrides): sink = OutputSink() - reasm = types.SimpleNamespace(ipv4=mock.Mock(), ipv6=mock.Mock(), tcp=mock.Mock()) + reasm = types.SimpleNamespace(ipv4=mock.Mock(), ipv6=mock.Mock(), tcp=mock.Mock(), + dtls=mock.Mock()) trace = types.SimpleNamespace(tcp=mock.Mock()) values = { '_ifile': io.BytesIO(b'capture'), @@ -190,6 +191,7 @@ def make_extractor(self, **overrides): '_ipv4': False, '_ipv6': False, '_tcp': False, + '_dtls': False, '_reasm': reasm, '_trace': trace, '_frame': [], @@ -305,6 +307,25 @@ def test_run_disables_ipv6_reassembly_but_keeps_ipv4_and_tcp(self) -> None: self.assertIs(extractor._reasm.tcp, tcp) self.assertIsNone(extractor._reasm.ipv6) + def test_run_disables_dtls_reassembly_but_keeps_the_rest(self) -> None: + from pcapkit.utilities.warnings import AttributeWarning + + extractor, _ = self.make_extractor(_flag_r=True, _ipv4=True, _tcp=True, _dtls=True) + ipv4, tcp = extractor._reasm.ipv4, extractor._reasm.tcp + engine = self.engine(extractor) + with mock.patch('pcapkit.foundation.engines.pypcapfile.warn') as warn: + engine.run() + + messages = [call.args[0] for call in warn.call_args_list + if call.args[1] is AttributeWarning] + self.assertTrue(any("'dtls=True' will be ignored" in message for message in messages), + messages) + self.assertTrue(extractor._flag_r) + self.assertFalse(extractor._dtls) + self.assertIsNone(extractor._reasm.dtls) + self.assertIs(extractor._reasm.ipv4, ipv4) + self.assertIs(extractor._reasm.tcp, tcp) + def test_run_leaves_reassembly_alone_when_ipv6_was_not_requested(self) -> None: extractor, _ = self.make_extractor(_flag_r=True, _ipv4=True, _tcp=True) original = extractor._reasm diff --git a/tests/foundation/engines/test_runtime_engines.py b/tests/foundation/engines/test_runtime_engines.py index 03b6cd707a..41ac27502e 100644 --- a/tests/foundation/engines/test_runtime_engines.py +++ b/tests/foundation/engines/test_runtime_engines.py @@ -35,7 +35,8 @@ def make_extractor(**overrides): from pcapkit.corekit.context import ContextRegistry sink = OutputSink() - reasm = types.SimpleNamespace(ipv4=mock.Mock(), ipv6=mock.Mock(), tcp=mock.Mock()) + reasm = types.SimpleNamespace(ipv4=mock.Mock(), ipv6=mock.Mock(), tcp=mock.Mock(), + dtls=mock.Mock()) trace = types.SimpleNamespace(tcp=mock.Mock()) values = { # ``Extractor`` always hands the engines a buffered (peekable) reader, @@ -56,6 +57,7 @@ def make_extractor(**overrides): '_ipv4': True, '_ipv6': True, '_tcp': True, + '_dtls': False, '_reasm': reasm, '_trace': trace, '_frame': [], diff --git a/tests/foundation/reassembly/data/test_models.py b/tests/foundation/reassembly/data/test_models.py index d7105b2cb0..e551271345 100644 --- a/tests/foundation/reassembly/data/test_models.py +++ b/tests/foundation/reassembly/data/test_models.py @@ -46,10 +46,11 @@ def test_ip_data_models_and_package_aliases(self) -> None: self.assertEqual(buffer.timestamp, 1000.0) self.assertEqual(buffer.conflict, []) - storage = ReassemblyData((datagram,), (), ()) + storage = ReassemblyData((datagram,), (), (), ()) self.assertEqual(storage.ipv4, (datagram,)) self.assertEqual(storage.ipv6, ()) self.assertEqual(storage.tcp, ()) + self.assertEqual(storage.dtls, ()) def test_completion_is_a_string_that_reads_as_a_bool(self) -> None: """The contract ``Datagram.completed`` has to keep as a StrEnum. diff --git a/tests/foundation/reassembly/test_dtls_handshake_reassembly_2440_unit.py b/tests/foundation/reassembly/test_dtls_handshake_reassembly_2440_unit.py index 3473125188..0249d3fb9e 100644 --- a/tests/foundation/reassembly/test_dtls_handshake_reassembly_2440_unit.py +++ b/tests/foundation/reassembly/test_dtls_handshake_reassembly_2440_unit.py @@ -1,22 +1,31 @@ # -*- coding: utf-8 -*- -"""DTLS handshake-message reassembly (GitHub issue #2440). +"""DTLS handshake-message reassembly (GitHub issues #2440 and #2496). -Covers the new :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine in -isolation, fed fragments directly, and the wiring into -:meth:`~pcapkit.protocols.application.dtls.DTLS.read`, which decodes a -fragmented ``ClientHello``/``ServerHello`` once its reassembly engine reports -the message complete. +Covers the :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine in +isolation, fed fragments directly, and the extraction path: the default engine +feeds it through :func:`pcapkit.toolkit.pcap.dtls_reassembly`, and +:func:`pcapkit.foundation.analysis.dtls.dtls_handshake` decodes a fragmented +``ClientHello``/``ServerHello`` once the engine reports the message complete. -Every case builds its fragments in memory and reads no capture. Classes are -imported inside each test, after :func:`~tests._support.reimport_once_per_class`. +Every case builds its fragments and captures in memory and reads no sample +capture. Classes are imported inside each test, after +:func:`~tests._support.reimport_once_per_class`. """ from __future__ import annotations +import io +import ipaddress +import struct import unittest +import warnings +from typing import TYPE_CHECKING -from tests._support import reimport_once_per_class +from tests._support import close_extractor, reimport_once_per_class + +if TYPE_CHECKING: + from typing import Any #: Test tier (#1958): hand-picked round-trip cases, not a sweep. __tier__ = 'ALL' @@ -71,6 +80,55 @@ def _split(body: bytes, bounds: 'tuple[int, ...]') -> 'list[tuple[int, int]]': return [(edges[i], edges[i + 1] - edges[i]) for i in range(len(edges) - 1)] +#: ``(src, sport, dst, dport)`` of the UDP association the captures below carry. +FLOW = (ipaddress.ip_address('192.0.2.1'), 1234, ipaddress.ip_address('198.51.100.1'), 5684) + + +def _frame(record: bytes, flow: 'tuple[Any, int, Any, int]' = FLOW) -> bytes: + """An Ethernet frame of IPv4 and UDP carrying the DTLS ``record`` along ``flow``.""" + src, sport, dst, dport = flow + segment = struct.pack('!HHHH', sport, dport, 8 + len(record), 0) + record + return (b'\xbb' * 6 + b'\xaa' * 6 + b'\x08\x00' + + struct.pack('!BBHHHBBH4s4s', 0x45, 0, 20 + len(segment), 1, 0x4000, 64, 17, 0, + src.packed, dst.packed) + + segment) + + +def _pcap(frames: 'list[bytes]') -> bytes: + """A little-endian microsecond Ethernet PCAP holding ``frames``, one second apart.""" + out = struct.pack(' bytes: + body += b'\x00' * (-len(body) % 4) + return struct.pack(' bytes: + """A little-endian PCAP-NG holding ``frames`` on an Ethernet interface, one second apart.""" + out = (_block(0x0A0D0D0A, struct.pack('> 32, ticks & 0xFFFFFFFF, + len(octets), len(octets)) + octets) + return out + + +def _extract(test: 'unittest.TestCase', capture: bytes, **kwargs: 'Any') -> 'Any': + """Run the default engine over ``capture``, every frame stored.""" + from pcapkit import extract + + with warnings.catch_warnings(): + warnings.simplefilter('ignore') + extractor = extract(fin=io.BytesIO(capture), nofile=True, store=True, **kwargs) + test.addCleanup(close_extractor, extractor) + return extractor + + class TestDTLSHandshakeReassembly(unittest.TestCase): """The reassembly engine, fed fragments directly.""" @@ -407,262 +465,196 @@ def test_delayed_fragments_complete_with_no_deadline(self) -> None: self.assertEqual(engine.last.body, body) -class TestDTLSReadReassemblyWiring(unittest.TestCase): - """:meth:`DTLS.read`'s ``reassembly``/``flow`` arguments, exercised - through the real protocol class rather than the engine alone.""" +class TestDTLSExtractorReassembly(unittest.TestCase): + """``Extractor(reassembly=True, dtls=True)`` over real frames, and + :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` over what it reassembles.""" def setUp(self) -> None: reimport_once_per_class(self) + @staticmethod + def _records(msg_type: int, body: bytes, frags: 'list[tuple[int, int]]', *, + message_seq: int = 5) -> 'list[bytes]': + """One DTLS record per ``(offset, length)`` fragment of ``body``.""" + return [_dtls_record(22, _dtls_handshake(msg_type, body[fo:fo + length], + message_seq=message_seq, fragment_offset=fo, + fragment_length=length, total_length=len(body)), + seq=num) + for num, (fo, length) in enumerate(frags, 1)] + def test_fragmented_client_hello_decodes_once_reassembled(self) -> None: - from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly - from pcapkit.protocols.application.dtls import DTLS + from pcapkit.foundation.analysis import dtls_handshake body = _client_hello_body(cookie=b'\xab\xcd\xef\x01') - half = len(body) // 2 - wire1 = _dtls_record(22, _dtls_handshake(1, body[:half], message_seq=5, fragment_offset=0, - fragment_length=half, total_length=len(body)), seq=2) - wire2 = _dtls_record(22, _dtls_handshake(1, body[half:], message_seq=5, fragment_offset=half, - fragment_length=len(body) - half, total_length=len(body)), seq=3) - - engine = DTLSReassembly() - flow = ('10.0.0.1', 1234, '10.0.0.2', 5684) - - parsed1 = DTLS(wire1, len(wire1), reassembly=engine, flow=flow, frame_num=1, frame_timestamp=0.0) - msg1 = parsed1.info.content[0] - self.assertTrue(msg1.is_fragmented) - self.assertIsInstance(msg1.body, bytes) - - parsed2 = DTLS(wire2, len(wire2), reassembly=engine, flow=flow, frame_num=2, frame_timestamp=0.1) - msg2 = parsed2.info.content[0] - self.assertTrue(msg2.is_fragmented) - self.assertNotIsInstance(msg2.body, bytes) - self.assertEqual(msg2.body.cookie, b'\xab\xcd\xef\x01') + records = self._records(1, body, _split(body, (len(body) // 2,))) + for container, write in (('pcap', _pcap), ('pcapng', _pcapng)): + with self.subTest(container=container): + extractor = _extract(self, write([_frame(r) for r in records]), + reassembly=True, dtls=True) + + # the frames themselves keep each fragment's raw octets + for frame in extractor.frame: + message = frame['DTLS'].info.content[0] + self.assertTrue(message.is_fragmented) + self.assertIsInstance(message.body, bytes) + + datagram, = extractor.reassembly.dtls + self.assertTrue(datagram.completed) + self.assertEqual(datagram.index, (1, 2)) + self.assertEqual((datagram.id.src, datagram.id.sport, datagram.id.dst, + datagram.id.dport, datagram.id.epoch), (*FLOW, 0)) + decoded = dtls_handshake(datagram) + self.assertFalse(decoded.is_fragmented) + self.assertEqual((decoded.fragment_offset, decoded.fragment_length), + (0, len(body))) + self.assertEqual(decoded.body.cookie, b'\xab\xcd\xef\x01') def test_fragmented_server_hello_decodes_once_reassembled(self) -> None: - from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly - from pcapkit.protocols.application.dtls import DTLS + from pcapkit.foundation.analysis import dtls_handshake body = _server_hello_body(extensions=b'\x00\x0d\x00\x02\xab\xcd') third = len(body) // 3 frags = _split(body, (third, 2 * third)) - bufid_flow = ('1.1.1.1', 100, '2.2.2.2', 5684) - - engine = DTLSReassembly() - decoded = None - # deliver out of order: [2, 0, 1] - for num, idx in enumerate((2, 0, 1), 1): - fo, length = frags[idx] - wire = _dtls_record(22, _dtls_handshake(2, body[fo:fo + length], message_seq=9, - fragment_offset=fo, fragment_length=length, total_length=len(body)), - seq=10 + num) - parsed = DTLS(wire, len(wire), reassembly=engine, flow=bufid_flow, - frame_num=num, frame_timestamp=float(num)) - msg = parsed.info.content[0] - if not isinstance(msg.body, bytes): - decoded = msg.body - self.assertIsNotNone(decoded) - self.assertEqual(decoded.random, bytes(range(32, 64))) - - def test_reassembly_without_flow_raises_rather_than_corrupting(self) -> None: - """Omitting ``flow`` while passing ``reassembly`` must not silently - default onto a shared buffer key -- two interleaved associations - would otherwise corrupt each other's reassembly.""" - from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly - from pcapkit.protocols.application.dtls import DTLS - from pcapkit.utilities.exceptions import ProtocolError - - body = _client_hello_body() - half = len(body) // 2 - wire = _dtls_record(22, _dtls_handshake(1, body[:half], message_seq=5, fragment_offset=0, - fragment_length=half, total_length=len(body)), seq=2) - - engine = DTLSReassembly() - with self.assertRaises(ProtocolError): - DTLS(wire, len(wire), reassembly=engine, frame_num=1, frame_timestamp=0.0) + # delivered out of order: [2, 0, 1] + records = self._records(2, body, [frags[2], frags[0], frags[1]], message_seq=9) + extractor = _extract(self, _pcap([_frame(r) for r in records]), reassembly=True, dtls=True) + + datagram, = extractor.reassembly.dtls + self.assertEqual(datagram.body, body) + self.assertEqual(dtls_handshake(datagram).body.random, bytes(range(32, 64))) + + def test_interleaved_associations_do_not_share_a_buffer(self) -> None: + """Two associations sending the same ``message_seq`` interleaved each + reassemble their own message: the buffer ID is read off each frame's + own IP and UDP layers.""" + from pcapkit.foundation.analysis import dtls_handshake + + other = (FLOW[0], 4321, FLOW[2], FLOW[3]) + body_a = _client_hello_body(cookie=b'\x0a' * 4) + body_b = _client_hello_body(cookie=b'\x0b' * 4) + half = len(body_a) // 2 + rec_a = self._records(1, body_a, _split(body_a, (half,))) + rec_b = self._records(1, body_b, _split(body_b, (half,))) + frames = [_frame(rec_a[0]), _frame(rec_b[0], other), _frame(rec_a[1]), _frame(rec_b[1], other)] + extractor = _extract(self, _pcap(frames), reassembly=True, dtls=True) + + cookies = {d.id.sport: dtls_handshake(d).body.cookie for d in extractor.reassembly.dtls} + self.assertEqual(cookies, {1234: b'\x0a' * 4, 4321: b'\x0b' * 4}) def test_encrypted_handshake_record_skips_reassembly_unharmed(self) -> None: - """A ``handshake``-typed record at epoch >= 1 is ciphertext -- - :meth:`SSL._read_tls_content ` - returns it as raw :obj:`bytes` rather than parsed messages. Every - real DTLS handshake ends with exactly such a record (the encrypted - ``Finished``), so a ``reassembly`` engine must not choke on it.""" - from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly - from pcapkit.protocols.application.dtls import DTLS - + """A ``handshake``-typed record at epoch >= 1 is ciphertext -- every + real DTLS handshake ends with one, the encrypted ``Finished`` -- so it + stays raw bytes and feeds nothing.""" ciphertext = b'\xa3\x5d\x12\xee' * 12 - wire = _dtls_record(22, ciphertext, epoch=1, seq=7) - - engine = DTLSReassembly() - flow = ('10.0.0.1', 1234, '10.0.0.2', 5684) - - parsed = DTLS(wire, len(wire), reassembly=engine, flow=flow, - frame_num=1, frame_timestamp=0.0) - self.assertEqual(parsed.info.content, ciphertext) - - def test_completing_record_rebuilds_byte_exact(self) -> None: - """The record that completes a reassembled message must still - round-trip exactly through ``from_data``/``make``: - the fragment header has to keep describing *this* record's own byte - range even though ``body`` holds the full decoded message, or - re-encoding it drifts out of sync with the bytes written after it.""" - from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + extractor = _extract(self, _pcap([_frame(_dtls_record(22, ciphertext, epoch=1, seq=7))]), + reassembly=True, dtls=True) + self.assertEqual(extractor.frame[0]['DTLS'].info.content, ciphertext) + self.assertEqual(extractor.reassembly.dtls, ()) + + def test_every_fragment_record_rebuilds_byte_exact(self) -> None: + """Reassembly leaves the frames alone, so every fragment's record -- + the one completing its message included -- round-trips exactly + through ``from_data``.""" from pcapkit.protocols.application.dtls import DTLS body = _client_hello_body(cookie=b'\xab\xcd\xef\x01') - half = len(body) // 2 - wire1 = _dtls_record(22, _dtls_handshake(1, body[:half], message_seq=5, fragment_offset=0, - fragment_length=half, total_length=len(body)), seq=2) - wire2 = _dtls_record(22, _dtls_handshake(1, body[half:], message_seq=5, fragment_offset=half, - fragment_length=len(body) - half, total_length=len(body)), seq=3) + records = self._records(1, body, _split(body, (len(body) // 2,))) + extractor = _extract(self, _pcap([_frame(r) for r in records]), reassembly=True, dtls=True) - engine = DTLSReassembly() - flow = ('10.0.0.1', 1234, '10.0.0.2', 5684) - - DTLS(wire1, len(wire1), reassembly=engine, flow=flow, frame_num=1, frame_timestamp=0.0) - parsed2 = DTLS(wire2, len(wire2), reassembly=engine, flow=flow, frame_num=2, frame_timestamp=0.1) - - msg2 = parsed2.info.content[0] - # the completing fragment: decoded in place, not raw bytes. - self.assertNotIsInstance(msg2.body, bytes) - - rebuilt = bytes(DTLS.from_data(parsed2.info)) - self.assertEqual(rebuilt, wire2) + self.assertTrue(extractor.reassembly.dtls[0].completed) + for frame, record in zip(extractor.frame, records): + self.assertEqual(bytes(DTLS.from_data(frame['DTLS'].info)), record) def test_overrunning_completing_fragment_keeps_raw_bytes_and_round_trips(self) -> None: """A fragment whose ``fragment_offset + fragment_length`` overruns the - declared ``length`` (5 + 10 > 10) completes its message, but its record - must stay as it arrived. Replacing its body with the clipped message - corrupts the round trip, and - :meth:`SSL._make_tls_handshake_messages - `'s - exactness check cannot be relied on to catch it.""" - from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + declared ``length`` (5 + 10 > 10) completes its message, clipped, but + its record stays exactly as it arrived.""" from pcapkit.protocols.application.dtls import DTLS length = 10 - first = b'A' * 5 second = b'B' * 10 # this record's own 10 octets, but offset 5 + 10 > length 10 - wire1 = _dtls_handshake(11, first, message_seq=3, fragment_offset=0, - fragment_length=5, total_length=length) - wire2 = _dtls_handshake(11, second, message_seq=3, fragment_offset=5, - fragment_length=10, total_length=length) - record1 = _dtls_record(22, wire1, seq=2) - record2 = _dtls_record(22, wire2, seq=3) - - engine = DTLSReassembly() - flow = ('19.0.0.1', 1, '19.0.0.2', 5684) - - DTLS(record1, len(record1), reassembly=engine, flow=flow, frame_num=1, frame_timestamp=0.0) - parsed2 = DTLS(record2, len(record2), reassembly=engine, flow=flow, frame_num=2, - frame_timestamp=0.1) - - # the engine itself did complete the message (clipping the overrun) - # but this record is left exactly as it arrived. - self.assertIsNotNone(engine.last) - self.assertTrue(engine.last.completed) - - msg2 = parsed2.info.content[0] - self.assertIsInstance(msg2.body, bytes) - self.assertEqual(msg2.body, second) - self.assertEqual(msg2.fragment_offset, 5) - self.assertEqual(msg2.fragment_length, 10) - - rebuilt = bytes(DTLS.from_data(parsed2.info)) - self.assertEqual(rebuilt, record2) + record1 = _dtls_record(22, _dtls_handshake(11, b'A' * 5, message_seq=3, fragment_offset=0, + fragment_length=5, total_length=length), seq=2) + record2 = _dtls_record(22, _dtls_handshake(11, second, message_seq=3, fragment_offset=5, + fragment_length=10, total_length=length), seq=3) + extractor = _extract(self, _pcap([_frame(record1), _frame(record2)]), + reassembly=True, dtls=True) + + datagram, = extractor.reassembly.dtls + self.assertTrue(datagram.completed) + self.assertEqual(datagram.body, b'A' * 5 + b'B' * 5) + + message = extractor.frame[1]['DTLS'].info.content[0] + self.assertEqual((message.body, message.fragment_offset, message.fragment_length), + (second, 5, 10)) + self.assertEqual(bytes(DTLS.from_data(extractor.frame[1]['DTLS'].info)), record2) def test_undecodable_reassembled_body_falls_back_to_raw_bytes(self) -> None: - """A completed message whose reassembled bytes do not decode as a - valid message of its ``msg_type`` keeps those raw bytes instead of - raising (dtls.py's ``except ProtocolError`` fallback).""" - from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly - from pcapkit.protocols.application.dtls import DTLS + """A completed message whose reassembled bytes do not decode as its + ``msg_type`` keeps them raw instead of raising.""" + from pcapkit.foundation.analysis import dtls_handshake - # far too short to be a genuine ClientHello (needs >= 34 octets - # before even the session_id length is reached), but a perfectly - # in-bounds pair of fragments for the reassembler itself. + # far too short to be a genuine ClientHello, but in bounds for the reassembler body = b'\x01\x02\x03\x04\x05' - wire1 = _dtls_record(22, _dtls_handshake(1, body[:3], message_seq=1, fragment_offset=0, - fragment_length=3, total_length=len(body)), seq=2) - wire2 = _dtls_record(22, _dtls_handshake(1, body[3:], message_seq=1, fragment_offset=3, - fragment_length=len(body) - 3, total_length=len(body)), seq=3) - - engine = DTLSReassembly() - flow = ('20.0.0.1', 1, '20.0.0.2', 5684) - - DTLS(wire1, len(wire1), reassembly=engine, flow=flow, frame_num=1, frame_timestamp=0.0) - parsed2 = DTLS(wire2, len(wire2), reassembly=engine, flow=flow, frame_num=2, frame_timestamp=0.1) - - self.assertIsNotNone(engine.last) - self.assertTrue(engine.last.completed) - - msg2 = parsed2.info.content[0] - # decode was attempted (the fragment is complete and in-bounds) and - # failed -- raw reassembled bytes kept, not raised. - self.assertIsInstance(msg2.body, bytes) - self.assertEqual(msg2.body, body) - - def test_flow_required_check_fires_for_non_handshake_records_too(self) -> None: - """``flow`` is required the moment ``reassembly`` is given, - regardless of this record's content type, not only for a - cleartext ``handshake`` record.""" - from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly - from pcapkit.protocols.application.dtls import DTLS - from pcapkit.utilities.exceptions import ProtocolError - - engine = DTLSReassembly() - - # an ``alert`` record -- never reaches ``_reassemble_dtls_handshake`` - # at all, so a handshake-only check would never see it. - alert_wire = _dtls_record(21, bytes([1, 0]), seq=2) - with self.assertRaises(ProtocolError): - DTLS(alert_wire, len(alert_wire), reassembly=engine, frame_num=1, frame_timestamp=0.0) - - # an encrypted (epoch >= 1) ``handshake`` record -- ``content`` is - # raw ciphertext :obj:`bytes` here, not a :obj:`tuple`, so a check - # keyed on the tuple would skip it too. - encrypted_wire = _dtls_record(22, b'\xaa' * 16, epoch=1, seq=3) - with self.assertRaises(ProtocolError): - DTLS(encrypted_wire, len(encrypted_wire), reassembly=engine, frame_num=2, - frame_timestamp=0.0) - - def test_without_reassembly_argument_body_stays_raw(self) -> None: - """With no engine, the body stays raw whatever the fragment.""" + records = self._records(1, body, _split(body, (3,)), message_seq=1) + extractor = _extract(self, _pcap([_frame(r) for r in records]), reassembly=True, dtls=True) + + datagram, = extractor.reassembly.dtls + self.assertTrue(datagram.completed) + decoded = dtls_handshake(datagram) + self.assertEqual(decoded.body, body) + self.assertFalse(decoded.is_fragmented) + + def test_records_without_fragmented_messages_feed_nothing(self) -> None: + """An alert, application data, and a handshake record whose message + arrives whole reach no reassembly buffer.""" + whole = _client_hello_body() + records = [ + _dtls_record(21, bytes([1, 0]), seq=2), + _dtls_record(23, b'\x17' * 16, seq=3), + _dtls_record(22, _dtls_handshake(1, whole, message_seq=0, fragment_offset=0, + fragment_length=len(whole)), seq=4), + ] + extractor = _extract(self, _pcap([_frame(r) for r in records]), reassembly=True, dtls=True) + self.assertEqual(extractor.reassembly.dtls, ()) + + def test_frames_are_the_same_with_and_without_reassembly(self) -> None: + """What the extractor yields per frame does not depend on DTLS reassembly.""" + body = _client_hello_body(cookie=b'\xab\xcd\xef\x01') + capture = _pcap([_frame(r) for r in self._records(1, body, _split(body, (len(body) // 2,)))]) + plain = _extract(self, capture) + reassembled = _extract(self, capture, reassembly=True, dtls=True) + self.assertEqual([f['DTLS'].info for f in plain.frame], + [f['DTLS'].info for f in reassembled.frame]) + + def test_without_reassembly_body_stays_raw(self) -> None: + """Parsed alone, a fragment's body stays raw whatever the fragment.""" from pcapkit.protocols.application.dtls import DTLS body = _client_hello_body() half = len(body) // 2 wire = _dtls_record(22, _dtls_handshake(1, body[half:], message_seq=5, fragment_offset=half, - fragment_length=len(body) - half, total_length=len(body)), seq=2) + fragment_length=len(body) - half, + total_length=len(body)), seq=2) - parsed = DTLS(wire, len(wire)) - msg = parsed.info.content[0] + msg = DTLS(wire, len(wire)).info.content[0] self.assertTrue(msg.is_fragmented) self.assertIsInstance(msg.body, bytes) - def test_incomplete_reassembly_through_read_stays_raw_and_does_not_crash(self) -> None: - from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly - from pcapkit.protocols.application.dtls import DTLS + def test_incomplete_reassembly_stays_raw_and_does_not_crash(self) -> None: + from pcapkit.foundation.analysis import dtls_handshake body = _server_hello_body() third = len(body) // 3 frags = _split(body, (third, 2 * third)) - flow = ('3.3.3.3', 200, '4.4.4.4', 5684) - - engine = DTLSReassembly() - for num, idx in enumerate((0, 2), 1): # fragment 1 of 3 never arrives - fo, length = frags[idx] - wire = _dtls_record(22, _dtls_handshake(2, body[fo:fo + length], message_seq=1, - fragment_offset=fo, fragment_length=length, total_length=len(body)), - seq=20 + num) - parsed = DTLS(wire, len(wire), reassembly=engine, flow=flow, - frame_num=num, frame_timestamp=float(num)) - msg = parsed.info.content[0] - self.assertTrue(msg.is_fragmented) - self.assertIsInstance(msg.body, bytes) - - datagram, = engine.fetch() + records = self._records(2, body, [frags[0], frags[2]], message_seq=1) # fragment 1 lost + extractor = _extract(self, _pcap([_frame(r) for r in records]), reassembly=True, dtls=True) + + for frame in extractor.frame: + self.assertIsInstance(frame['DTLS'].info.content[0].body, bytes) + datagram, = extractor.reassembly.dtls self.assertEqual(datagram.completed.name, 'PARTIAL') + self.assertIsNone(dtls_handshake(datagram)) if __name__ == '__main__': diff --git a/tests/foundation/reassembly/test_dtls_toolkit_analysis_2496_unit.py b/tests/foundation/reassembly/test_dtls_toolkit_analysis_2496_unit.py new file mode 100644 index 0000000000..54f932186c --- /dev/null +++ b/tests/foundation/reassembly/test_dtls_toolkit_analysis_2496_unit.py @@ -0,0 +1,340 @@ +# -*- coding: utf-8 -*- +"""DTLS reassembly is fed by the toolkit and parsed by the analysis (#2496). + +:class:`~pcapkit.protocols.application.dtls.DTLS` stays a parser: the default +engine reads each frame's fragmented handshake messages with +:func:`pcapkit.toolkit.pcap.dtls_reassembly` (or its PCAP-NG twin), +:class:`~pcapkit.foundation.reassembly.dtls.DTLS` reassembles them, and +:func:`pcapkit.foundation.analysis.dtls.dtls_handshake` decodes the result. The +engines that do not parse with pcapkit's own protocols turn DTLS reassembly off. + +Every capture is built in memory; no sample capture is read. + +""" + +from __future__ import annotations + +import ast +import importlib.util +import inspect +import ipaddress +import struct +import types +import unittest +import warnings +from typing import TYPE_CHECKING +from unittest import mock + +from tests._support import reimport_once_per_class +from tests.foundation.reassembly.test_dtls_handshake_reassembly_2440_unit import ( + FLOW, _client_hello_body, _dtls_handshake, _dtls_record, _extract, _frame, _pcap, _pcapng, + _split) + +if TYPE_CHECKING: + from typing import Any + +#: A ``ClientHello`` body, and the two records it is fragmented across. +BODY = _client_hello_body(cookie=b'\xab\xcd\xef\x01') +RECORDS = [ + _dtls_record(22, _dtls_handshake(1, BODY[fo:fo + length], message_seq=4, fragment_offset=fo, + fragment_length=length, total_length=len(BODY)), seq=num) + for num, (fo, length) in enumerate(_split(BODY, (len(BODY) // 2,)), 1) +] + + +def _udp(payload: bytes, sport: int = FLOW[1], dport: int = FLOW[3]) -> bytes: + return struct.pack('!HHHH', sport, dport, 8 + len(payload), 0) + payload + + +def _ipv4(payload: bytes, src: bytes, dst: bytes, protocol: int) -> bytes: + return struct.pack('!BBHHHBBH4s4s', 0x45, 0, 20 + len(payload), 1, 0x4000, 64, protocol, 0, + src, dst) + payload + + +def _ether(packet: bytes, ethertype: int = 0x0800) -> bytes: + return b'\xbb' * 6 + b'\xaa' * 6 + ethertype.to_bytes(2, 'big') + packet + + +class DTLSToolkitAdapterTests(unittest.TestCase): + """:func:`dtls_reassembly` reads a frame's fragmented messages, and only those.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + + def _adapt(self, frames: 'list[bytes]', **kwargs: 'Any') -> 'list[Any]': + """Each frame's PCAP adapter result, checked equal to the PCAP-NG adapter's.""" + from pcapkit.toolkit import pcap as toolkit_pcap + from pcapkit.toolkit import pcapng as toolkit_pcapng + + pcap_frames = _extract(self, _pcap(frames), **kwargs).frame + pcapng_frames = _extract(self, _pcapng(frames), **kwargs).frame + results = [toolkit_pcap.dtls_reassembly(frame) for frame in pcap_frames] + self.assertEqual(results, [toolkit_pcapng.dtls_reassembly(frame) for frame in pcapng_frames]) + return results + + def test_a_fragmented_message_is_read_off_the_frame(self) -> None: + packets = self._adapt([_frame(record) for record in RECORDS]) + half = len(BODY) // 2 + for num, (result, (fo, length)) in enumerate(zip(packets, ((0, half), (half, len(BODY) - half))), 1): + with self.subTest(frame=num): + packet, = result + self.assertEqual(packet.bufid, (*FLOW, 0)) + self.assertEqual((packet.num, packet.timestamp), (num, 1500000000.0 + num - 1)) + self.assertEqual((packet.msg_type, packet.length, packet.message_seq), (1, len(BODY), 4)) + self.assertEqual((packet.fragment_offset, packet.fragment_length), (fo, length)) + self.assertEqual(packet.payload, BODY[fo:fo + length]) + + def test_every_fragmented_message_of_a_record_is_read_in_order(self) -> None: + cert, done, whole = b'C' * 40, b'D' * 20, _client_hello_body() + fragment = (_dtls_handshake(11, cert[:16], message_seq=1, fragment_offset=0, + fragment_length=16, total_length=len(cert)) + + _dtls_handshake(1, whole, message_seq=2, fragment_offset=0, + fragment_length=len(whole)) + + _dtls_handshake(14, done[8:], message_seq=3, fragment_offset=8, + fragment_length=12, total_length=len(done))) + result, = self._adapt([_frame(_dtls_record(22, fragment))]) + self.assertEqual([(p.message_seq, p.payload) for p in result], [(1, cert[:16]), (3, done[8:])]) + + def test_frames_without_a_fragmented_message_report_none(self) -> None: + whole = _client_hello_body() + unified = bytes([0b00101100]) + b'\x00\x07' + (16).to_bytes(2, 'big') + b'\x5a' * 16 + cases = { + 'whole message': _frame(_dtls_record(22, _dtls_handshake( + 1, whole, message_seq=0, fragment_offset=0, fragment_length=len(whole)))), + 'alert': _frame(_dtls_record(21, bytes([1, 0]))), + 'ciphertext handshake': _frame(_dtls_record(22, b'\xa3' * 32, epoch=1)), + 'unified header': _frame(unified), + 'not DTLS': _ether(_ipv4(_udp(RECORDS[0], dport=53), FLOW[0].packed, FLOW[2].packed, 17)), + 'no UDP': _ether(_ipv4(RECORDS[0], FLOW[0].packed, FLOW[2].packed, 253)), + } + for name, frame in cases.items(): + with self.subTest(case=name): + self.assertEqual(self._adapt([frame]), [None]) + + def test_a_dissection_stopped_short_of_dtls_reports_none(self) -> None: + for kwargs in ({'layer': 'transport'}, {'protocol': 'UDP'}): + with self.subTest(**kwargs): + self.assertEqual(self._adapt([_frame(RECORDS[0])], **kwargs), [None]) + + def test_the_ip_layer_nearest_the_udp_one_keys_the_buffer(self) -> None: + inner_src, inner_dst = ipaddress.ip_address('10.0.0.1'), ipaddress.ip_address('10.0.0.2') + inner = _ipv4(_udp(RECORDS[0]), inner_src.packed, inner_dst.packed, 17) + tunnelled = _ether(_ipv4(inner, FLOW[0].packed, FLOW[2].packed, 4)) + + v6_src, v6_dst = ipaddress.ip_address('2001:db8::1'), ipaddress.ip_address('2001:db8::2') + segment = _udp(RECORDS[0]) + over_v6 = _ether(struct.pack('!IHBB16s16s', 6 << 28, len(segment), 17, 64, + v6_src.packed, v6_dst.packed) + segment, 0x86DD) + + for name, frame, (src, dst) in (('IPv4 in IPv4', tunnelled, (inner_src, inner_dst)), + ('IPv6', over_v6, (v6_src, v6_dst))): + with self.subTest(case=name): + (packet,), = self._adapt([frame]) + self.assertEqual(packet.bufid, (src, FLOW[1], dst, FLOW[3], 0)) + + +class DTLSAnalysisTests(unittest.TestCase): + """:func:`dtls_handshake` decodes a reassembled message as an unfragmented one.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + + @staticmethod + def _datagram(body: 'bytes | tuple[bytes, ...]', *, msg_type: int = 1, + completed: str = 'COMPLETE', length: 'int | None' = None) -> 'Any': + from pcapkit.const.tls.handshake_type import HandshakeType + from pcapkit.foundation.reassembly.data.data import Completion + from pcapkit.foundation.reassembly.data.dtls import Datagram, DatagramID + + return Datagram(completed=Completion[completed], id=DatagramID(*FLOW, 0), index=(1, 2), + message_seq=4, msg_type=HandshakeType.get(msg_type), + length=len(body) if length is None else length, body=body) + + def test_a_hello_decodes_as_the_unfragmented_message_does(self) -> None: + from pcapkit.foundation.analysis import dtls_handshake + from pcapkit.protocols.application.dtls import DTLS + + for msg_type, body in ((1, BODY), (11, b'C' * 40)): + with self.subTest(msg_type=msg_type): + whole = _dtls_record(22, _dtls_handshake(msg_type, body, message_seq=4, + fragment_offset=0, fragment_length=len(body))) + expected = DTLS(whole, len(whole)).info.content[0] + self.assertEqual(dtls_handshake(self._datagram(body, msg_type=msg_type)), expected) + + def test_an_undecodable_body_stays_raw(self) -> None: + from pcapkit.foundation.analysis import dtls_handshake + + decoded = dtls_handshake(self._datagram(b'\x01\x02\x03\x04\x05')) + self.assertEqual(decoded.body, b'\x01\x02\x03\x04\x05') + self.assertEqual((decoded.message_seq, decoded.fragment_offset, decoded.fragment_length, + decoded.is_fragmented), (4, 0, 5, False)) + + def test_a_message_too_long_for_one_record_still_decodes(self) -> None: + """A genuine ``ClientHello`` whose extensions take it past what a + record's 16-bit length holds decodes all the same.""" + from pcapkit.foundation.analysis import dtls_handshake + from pcapkit.protocols.data.application.tls import ClientHello, Extension + + body = _client_hello_body(extensions=b'\x00\x15' + (65500).to_bytes(2, 'big') + bytes(65500)) + self.assertGreater(len(body) + 12, 0xFFFF) + decoded = dtls_handshake(self._datagram(body)) + self.assertIsInstance(decoded.body, ClientHello) + self.assertEqual(decoded.body.extensions, (Extension(type=21, data=bytes(65500)),)) + self.assertEqual((decoded.length, decoded.fragment_length, decoded.is_fragmented), + (len(body), len(body), False)) + + def test_the_function_is_exported_as_reassembly_names_are(self) -> None: + import pcapkit.all + import pcapkit.foundation + import pcapkit.foundation.analysis + from pcapkit.foundation.analysis.dtls import dtls_handshake + + for module in (pcapkit.foundation.analysis, pcapkit.foundation, pcapkit.all): + with self.subTest(module=module.__name__): + self.assertIn('dtls_handshake', module.__all__) + self.assertIs(module.dtls_handshake, dtls_handshake) + + def test_an_incomplete_message_is_not_decoded(self) -> None: + from pcapkit.foundation.analysis import dtls_handshake + + for completed, body in (('PARTIAL', (BODY[:10], BODY[20:])), ('PARTIAL', BODY), + ('TIMEOUT', BODY)): + with self.subTest(completed=completed, runs=isinstance(body, tuple)): + self.assertIsNone(dtls_handshake(self._datagram(body, completed=completed, + length=len(BODY)))) + + +class DTLSProtocolStaysAParserTests(unittest.TestCase): + """The protocol class knows nothing of reassembly (convention point 1).""" + + def setUp(self) -> None: + reimport_once_per_class(self) + + def test_the_dtls_module_imports_nothing_from_foundation(self) -> None: + import pcapkit.protocols.application.dtls as module + + tree = ast.parse(inspect.getsource(module)) + imported = [node.module for node in ast.walk(tree) + if isinstance(node, ast.ImportFrom) and node.module] + imported += [alias.name for node in ast.walk(tree) if isinstance(node, ast.Import) + for alias in node.names] + self.assertEqual([name for name in imported if name.startswith('pcapkit.foundation')], []) + + def test_read_takes_no_reassembly_arguments(self) -> None: + from pcapkit.protocols.application.dtls import DTLS + + self.assertEqual(list(inspect.signature(DTLS.read).parameters), ['self', 'length', 'kwargs']) + + +class DTLSExtractorWiringTests(unittest.TestCase): + """The ``dtls`` switch of :class:`~pcapkit.foundation.extraction.Extractor`.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + + def test_dtls_reassembly_is_off_unless_asked_for(self) -> None: + capture = _pcap([_frame(record) for record in RECORDS]) + self.assertIsNone(_extract(self, capture, reassembly=True, ip=True).reassembly.dtls) + self.assertEqual(len(_extract(self, capture, reassembly=True, dtls=True).reassembly.dtls), 1) + + def test_the_reassembly_timeout_does_not_reach_dtls(self) -> None: + """The RFC gives DTLS handshake reassembly no timeout, so fragments a + second apart still complete under a half-second ``reasm_timeout``.""" + extractor = _extract(self, _pcap([_frame(record) for record in RECORDS]), + reassembly=True, dtls=True, reasm_timeout=0.5) + datagram, = extractor.reassembly.dtls + self.assertTrue(datagram.completed) + + def test_the_shipped_reassembly_entry_names_the_dtls_engine(self) -> None: + from pcapkit.foundation.extraction import Extractor + from pcapkit.foundation.reassembly.dtls import DTLS + + self.assertIs(Extractor._shipped_reassembly['dtls'].klass, DTLS) + + +def _extractor(**overrides: 'Any') -> 'Any': + from tests.foundation.engines.test_runtime_engines import make_extractor + + values = {'_flag_r': True, '_flag_t': False, '_ipv4': True, '_ipv6': False, '_tcp': True, + '_dtls': True} + values.update(overrides) + extractor, _ = make_extractor(**values) + return extractor + + +class FakeReader: + def __iter__(self) -> 'FakeReader': + return self + + def __next__(self) -> 'Any': + raise StopIteration + + def datalink(self) -> int: + return 1 + + +class DTLSThirdPartyEngineTests(unittest.TestCase): + """DPKT, Scapy and PyShark do not dissect with pcapkit, so DTLS reassembly is off.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + + def _assert_dtls_dropped(self, extractor: 'Any', warn: 'Any', ipv4: 'Any', tcp: 'Any') -> None: + from pcapkit.utilities.warnings import AttributeWarning + + messages = [call.args[0] for call in warn.call_args_list if call.args[1] is AttributeWarning] + self.assertTrue(any("'dtls=True' will be ignored" in message for message in messages), messages) + self.assertFalse(extractor._dtls) + self.assertTrue(extractor._flag_r) + self.assertIsNone(extractor._reasm.dtls) + self.assertIs(extractor._reasm.ipv4, ipv4) + self.assertIs(extractor._reasm.tcp, tcp) + + @unittest.skipUnless(importlib.util.find_spec('dpkt'), 'DPKT not installed') + def test_dpkt_turns_dtls_reassembly_off(self) -> None: + from pcapkit.foundation.engines.dpkt import DPKT + + extractor = _extractor() + ipv4, tcp = extractor._reasm.ipv4, extractor._reasm.tcp + engine = DPKT(extractor) + with mock.patch('pcapkit.foundation.engines.dpkt._pcap_reader', + return_value=mock.Mock(return_value=FakeReader())), \ + mock.patch('pcapkit.foundation.engines.dpkt.warn') as warn: + engine.run() + self._assert_dtls_dropped(extractor, warn, ipv4, tcp) + + @unittest.skipUnless(importlib.util.find_spec('scapy'), 'Scapy not installed') + def test_scapy_turns_dtls_reassembly_off(self) -> None: + from pcapkit.foundation.engines.scapy import Scapy + + extractor = _extractor() + ipv4, tcp = extractor._reasm.ipv4, extractor._reasm.tcp + with warnings.catch_warnings(): + warnings.simplefilter('ignore') + engine = Scapy(extractor) + with mock.patch('pcapkit.foundation.engines.scapy._reader_type', + return_value=mock.Mock(return_value=mock.Mock())), \ + mock.patch.object(engine._expkg, 'sniff', return_value=[]), \ + mock.patch('pcapkit.foundation.engines.scapy.warn') as warn: + engine.run() + self._assert_dtls_dropped(extractor, warn, ipv4, tcp) + + @unittest.skipUnless(importlib.util.find_spec('pyshark'), 'PyShark not installed') + def test_pyshark_turns_reassembly_off_for_dtls_alone(self) -> None: + from pcapkit.foundation.engines.pyshark import PyShark + from pcapkit.utilities.warnings import AttributeWarning + + extractor = _extractor(_ipv4=False, _tcp=False) + engine = PyShark(extractor) + capture = types.SimpleNamespace(close=lambda: None) + with mock.patch.object(engine._expkg, 'FileCapture', return_value=capture), \ + mock.patch('pcapkit.foundation.engines.pyshark.warn') as warn: + engine.run() + messages = [call.args[0] for call in warn.call_args_list if call.args[1] is AttributeWarning] + self.assertTrue(any("'dtls=True' will be ignored" in message for message in messages), messages) + self.assertFalse(extractor._flag_r) + self.assertIsNone(extractor._reasm.dtls) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/foundation/test_extraction.py b/tests/foundation/test_extraction.py index 440e964925..4471fe8b38 100644 --- a/tests/foundation/test_extraction.py +++ b/tests/foundation/test_extraction.py @@ -117,11 +117,13 @@ def _bare_extractor(self): extractor._ipv4 = True extractor._ipv6 = True extractor._tcp = True + extractor._dtls = True extractor._frame = ['frame'] extractor._reasm = types.SimpleNamespace( ipv4=types.SimpleNamespace(datagram=('ipv4',)), ipv6=types.SimpleNamespace(datagram=('ipv6',)), tcp=types.SimpleNamespace(datagram=('tcp',)), + dtls=types.SimpleNamespace(datagram=('dtls',)), ) # ``finish`` as well as ``index``: _cleanup tells the tracer the capture # has ended, which is where a flow nothing superseded is finalised, so a @@ -154,12 +156,14 @@ def test_properties_success_and_unsupported_paths(self) -> None: self.assertEqual(extractor.reassembly.ipv4, ('ipv4',)) self.assertEqual(extractor.reassembly.ipv6, ('ipv6',)) self.assertEqual(extractor.reassembly.tcp, ('tcp',)) + self.assertEqual(extractor.reassembly.dtls, ('dtls',)) self.assertEqual(extractor.trace.tcp, ('trace',)) self.assertIs(extractor.engine, extractor._exeng) self.assertEqual(extractor.magic_number, b'fake') - extractor._ipv4 = extractor._ipv6 = extractor._tcp = False + extractor._ipv4 = extractor._ipv6 = extractor._tcp = extractor._dtls = False self.assertIsNone(extractor.reassembly.ipv4) + self.assertIsNone(extractor.reassembly.dtls) self.assertIsNone(extractor.trace.tcp) unsupported = object.__new__(Extractor) diff --git a/tests/protocols/application/test_dtls_handshake_message_2496_unit.py b/tests/protocols/application/test_dtls_handshake_message_2496_unit.py new file mode 100644 index 0000000000..72c222fc1e --- /dev/null +++ b/tests/protocols/application/test_dtls_handshake_message_2496_unit.py @@ -0,0 +1,159 @@ +# -*- coding: utf-8 -*- +"""A DTLS handshake message read and written on its own (#2496). + +:meth:`DTLS.read_handshake ` +reads one handshake message outside any record, so a message longer than a +record's 16-bit ``length`` still decodes. A fragmented message is written with +exactly ``fragment_length`` body octets (RFC 6347, section 4.2.3): a body that is +the whole message is cut down to the fragment's range, and any other length is +refused. + +Every case builds its own octets; no sample capture is read. + +""" +from __future__ import annotations + +import unittest + +from tests._support import reimport_once_per_class + +#: Test tier (#1958): hand-picked round-trip cases, not a sweep. +__tier__ = 'ALL' + + +def _client_hello_body(extensions: bytes = b'') -> bytes: + """A DTLS ``ClientHello`` body, its ``cookie`` included.""" + return (b'\xfe\xfd' + bytes(range(32)) + b'\x00' + b'\x04\xab\xcd\xef\x01' + + b'\x00\x02\x13\x01' + b'\x01\x00' + len(extensions).to_bytes(2, 'big') + extensions) + + +def _server_hello_body() -> bytes: + return b'\xfe\xfd' + bytes(range(32, 64)) + b'\x00' + b'\x13\x01' + b'\x00' + b'\x00\x00' + + +def _handshake(msg_type: int, body: bytes, *, message_seq: int = 3, fragment_offset: int = 0, + fragment_length: 'int | None' = None, length: 'int | None' = None) -> bytes: + fragment_length = len(body) if fragment_length is None else fragment_length + length = len(body) if length is None else length + return (bytes([msg_type]) + length.to_bytes(3, 'big') + message_seq.to_bytes(2, 'big') + + fragment_offset.to_bytes(3, 'big') + fragment_length.to_bytes(3, 'big') + body) + + +def _record(fragment: bytes) -> bytes: + """A cleartext DTLS 1.2 ``handshake`` record, epoch and sequence number 0.""" + return b'\x16\xfe\xfd' + bytes(8) + len(fragment).to_bytes(2, 'big') + fragment + + +#: The ``ClientHello`` body the fragment cases cut, and where they cut it. +BODY = _client_hello_body() +HALF = len(BODY) // 2 + + +class ReadHandshakeTests(unittest.TestCase): + """:meth:`DTLS.read_handshake` reads exactly one message, of any length.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + + def test_a_message_reads_as_it_does_in_a_record(self) -> None: + from pcapkit.protocols.application.dtls import DTLS + + for name, message in { + 'ClientHello': _handshake(1, BODY), + 'ServerHello': _handshake(2, _server_hello_body()), + 'other type': _handshake(11, b'C' * 40), + 'fragment': _handshake(1, BODY[:HALF], fragment_length=HALF, length=len(BODY)), + }.items(): + with self.subTest(case=name): + record = _record(message) + expected, = DTLS(record, len(record)).info.content + self.assertEqual(DTLS.read_handshake(message), expected) + + def test_a_message_longer_than_a_record_holds_decodes(self) -> None: + from pcapkit.protocols.application.dtls import DTLS + from pcapkit.protocols.data.application.tls import ClientHello, Extension + + body = _client_hello_body(b'\x00\x15' + (65500).to_bytes(2, 'big') + bytes(65500)) + message = _handshake(1, body) + self.assertGreater(len(message), 0xFFFF) + decoded = DTLS.read_handshake(message) + self.assertIsInstance(decoded.body, ClientHello) + self.assertEqual(decoded.body.extensions, (Extension(type=21, data=bytes(65500)),)) + self.assertEqual((decoded.length, decoded.message_seq, decoded.is_fragmented), + (len(body), 3, False)) + + def test_anything_but_one_message_is_refused(self) -> None: + from pcapkit.protocols.application.dtls import DTLS + from pcapkit.utilities.exceptions import ProtocolError + + whole = _handshake(1, BODY) + for name, octets in { + 'empty': b'', + 'two messages': whole + whole, + 'trailing octets': whole + b'\x00', + 'truncated body': whole[:-1], + 'malformed ClientHello': _handshake(1, b'\x01\x02'), + }.items(): + with self.subTest(case=name), self.assertRaisesRegex(ProtocolError, '^DTLS: '): + DTLS.read_handshake(octets) + + +class FragmentBodyTests(unittest.TestCase): + """A fragment is written with exactly ``fragment_length`` body octets.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + + @staticmethod + def _fragment(body: 'object', fragment_offset: int, fragment_length: int) -> 'object': + from pcapkit.const.tls.handshake_type import HandshakeType + from pcapkit.protocols.data.application.dtls import DTLSHandshake + + return DTLSHandshake(msg_type=HandshakeType.client_hello, length=len(BODY), body=body, + message_seq=3, fragment_offset=fragment_offset, + fragment_length=fragment_length, is_fragmented=True) + + def test_a_whole_message_body_is_cut_to_the_fragment(self) -> None: + from pcapkit.protocols.application.dtls import DTLS + + hello = DTLS.read_handshake(_handshake(1, BODY)).body + for body in (hello, BODY): + for offset, length in ((0, HALF), (HALF, len(BODY) - HALF)): + with self.subTest(decoded=body is hello, fragment_offset=offset): + wire = _record(_handshake(1, BODY[offset:offset + length], + fragment_offset=offset, fragment_length=length, + length=len(BODY))) + made = DTLS(content_type=22, content=(self._fragment(body, offset, length),)) + self.assertEqual(bytes(made), wire) + self.assertEqual(DTLS(wire, len(wire)).info.content[0].body, + BODY[offset:offset + length]) + + def test_a_body_of_any_other_length_is_refused(self) -> None: + from pcapkit.protocols.application.dtls import DTLS + from pcapkit.utilities.exceptions import ProtocolError + + for name, (body, offset, length) in { + 'short': (BODY[:HALF - 1], 0, HALF), + 'long': (BODY[:HALF + 1], 0, HALF), + 'whole message, range overruns it': (BODY, HALF, len(BODY) - HALF + 1), + }.items(): + with self.subTest(case=name), self.assertRaisesRegex(ProtocolError, 'fragment_length'): + DTLS(content_type=22, content=(self._fragment(body, offset, length),)) + + def test_a_parsed_fragment_rebuilds_byte_exact(self) -> None: + """A fragment's own body is never cut, even one whose range overruns + its message (offset 5 + 10 > length 10).""" + from pcapkit.protocols.application.dtls import DTLS + + for name, message in { + 'first half': _handshake(1, BODY[:HALF], fragment_length=HALF, length=len(BODY)), + 'overrunning': _handshake(11, b'B' * 10, fragment_offset=5, fragment_length=10, + length=10), + }.items(): + with self.subTest(case=name): + wire = _record(message) + self.assertEqual(bytes(DTLS.from_data(DTLS(wire, len(wire)).info)), wire) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/toolkit/test_pcap_tcp_segment_public_1570_unit.py b/tests/toolkit/test_pcap_tcp_segment_public_1570_unit.py index ce9f023469..999241465d 100644 --- a/tests/toolkit/test_pcap_tcp_segment_public_1570_unit.py +++ b/tests/toolkit/test_pcap_tcp_segment_public_1570_unit.py @@ -88,7 +88,8 @@ def test_all(self) -> None: from pcapkit.toolkit import pcap self.assertEqual(pcap.__all__, ['ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', - 'tcp_traceflow', 'tcp_segment', 'TCPSegment']) + 'dtls_reassembly', 'tcp_traceflow', 'tcp_segment', + 'TCPSegment']) def test_callers_share_the_public_helper(self) -> None: from pcapkit.toolkit import pcap, pcapng