diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 6a36c5973f..9eeab1455e 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -958,7 +958,7 @@ Changed :rfc:`791` specifies the opposite resolution, and records the disagreement instead (:pr:`482`). * renames with no compatibility alias left behind: ``HoleDiscriptor`` is spelled - :class:`.HoleDescriptor` and its package alias ``TCP_HoleDiscriptor`` is + :class:`~pcapkit.foundation.reassembly.data.tcp.HoleDescriptor` and its package alias ``TCP_HoleDiscriptor`` is ``TCP_HoleDescriptor`` (:pr:`350`). * subclass registration is **opt-in** for :class:`.Engine`, :class:`.Reassembly`, :class:`.TraceFlow` and ``dumpkit``'s :class:`.Dumper` (:issue:`514`). Each registers diff --git a/docs/source/contributing/pep.rst b/docs/source/contributing/pep.rst index f97e882953..ec34f81c2a 100644 --- a/docs/source/contributing/pep.rst +++ b/docs/source/contributing/pep.rst @@ -242,14 +242,15 @@ separate question and is not needed to make the record structure legible, in the same way :class:`ESP ` parses without keys. -**Not done: reassembling a handshake message DTLS has fragmented across -several records** (:rfc:`6347#section-4.2.3`, :issue:`2440`). That needs a -:class:`~pcapkit.foundation.reassembly.reassembly.Reassembly` subclass keyed -on the UDP 5-tuple plus epoch. A fragmented message's body is kept as that one -fragment's raw octets -- see -:attr:`DTLSHandshake.is_fragmented ` --- rather than decoded from a slice of the message. An unfragmented message is -unaffected. +**Done: reassembling a handshake message DTLS has fragmented across several +records** (:rfc:`6347#section-4.2.3`, :issue:`2440`), via +:class:`~pcapkit.foundation.reassembly.dtls.DTLS`, keyed on the UDP 4-tuple. +Pass ``reassembly=`` and ``flow=`` to +:class:`~pcapkit.protocols.application.dtls.DTLS` to opt in; the completing +fragment's body decodes to the full reassembled message, while every other +fragment's body stays that one fragment's raw octets -- see +:attr:`DTLSHandshake.is_fragmented `. +An unfragmented message is unaffected either way. NGAP over DTLS over SCTP (PPID 66, :class:`~pcapkit.const.sctp.payload_protocol_identifier.PayloadProtocolIdentifier`), @@ -1088,9 +1089,9 @@ twice. **DCCP** is the transport-layer implementation, tracked by :issue:`2489`, blocked on :issue:`2479` for its pseudo-header checksum. **Done within wave 2 — TLS/SSL with DTLS.** The `DTLS`_ section above lists -what is decoded (the shared :class:`~pcapkit.protocols.application.ssl.SSL` -base, TLS, DTLS, and handshake message framing) and what is not (DTLS -handshake-fragment reassembly, :issue:`2440`). +what is decoded: the shared :class:`~pcapkit.protocols.application.ssl.SSL` +base, TLS, DTLS, handshake message framing, and handshake-fragment +reassembly (:issue:`2440`). **Done ahead of wave 2 — handlers for the** ``DLT_NULL``, ``DLT_RAW``, ``DLT_LINUX_SLL`` **and** ``DLT_LINUX_SLL2`` **link types.** Each is a diff --git a/docs/source/pcapkit/foundation/reassembly/dtls.rst b/docs/source/pcapkit/foundation/reassembly/dtls.rst new file mode 100644 index 0000000000..bb0484bde9 --- /dev/null +++ b/docs/source/pcapkit/foundation/reassembly/dtls.rst @@ -0,0 +1,121 @@ +================================= +DTLS Handshake-Message Reassembly +================================= + +.. module:: pcapkit.foundation.reassembly.dtls + +:mod:`pcapkit.foundation.reassembly.dtls` contains +:class:`~pcapkit.foundation.reassembly.dtls.DTLS` only, which reconstructs a +DTLS handshake message split across several records back to origin +(:rfc:`6347#section-4.2.3`). + +.. autoclass:: pcapkit.foundation.reassembly.dtls.DTLS + :no-members: + :show-inheritance: + + .. autoproperty:: last + + .. automethod:: reassembly + .. automethod:: submit + .. automethod:: _datagram_from_fragment + .. automethod:: _received_runs + .. automethod:: _update_hole_descriptors + + .. autoattribute:: __protocol_name__ + +Algorithm +========= + +.. seealso:: + + Holes are inclusive byte offsets into the message. They never wrap, and + the upper bound is known from the start: every fragment carries the + message's total ``length`` (:rfc:`6347#section-4.2.3`). + +A DTLS handshake record may carry more than one message fragment -- +:rfc:`6347#section-4.2.3` allows several handshake messages (or fragments of +them) to be coalesced into one record, and :meth:`SSL._read_tls_handshake_messages +` +reads however many fit rather than assuming exactly one. A fragment names the +message it belongs to (``message_seq``), the message's total length +(``length``), and its own byte range within that message (``fragment_offset``, +``fragment_length``). A message is reassembled by punching the arriving +fragment's range out of the message's hole list; it is whole once no hole +remains. + +Terminology +=========== + +.. glossary:: + + reasm.dtls.packet + Data structure for **DTLS handshake-fragment reassembly** + (:meth:`DTLS.reassembly `) + is :class:`~pcapkit.foundation.reassembly.data.dtls.Packet` -- one + fragment's own ``bufid``, ``message_seq``, ``fragment_offset``, + ``fragment_length`` and raw ``payload``, built by + :meth:`DTLS.read ` from + a single :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`. + + reasm.dtls.datagram + Data structure for **reassembled DTLS handshake message** (element of + :attr:`DTLS.datagram `) + is :class:`~pcapkit.foundation.reassembly.data.dtls.Datagram` -- + ``completed``, the original packet identifier, the packet numbers + that built it, and ``body``, the reassembled payload (a contiguous + :obj:`bytes`, or -- under ``strict=True`` and an incomplete message -- + a :obj:`tuple` of the octet runs actually received). + + reasm.dtls.buffer + Data structure for internal buffering + (:attr:`DTLS._buffer `) + is :class:`~pcapkit.foundation.reassembly.data.dtls.Buffer` -- every + message under reconstruction for one + :data:`~pcapkit.foundation.reassembly.data.dtls.BufferID` (one UDP + association at one DTLS epoch), each keyed by its + :attr:`~pcapkit.foundation.reassembly.data.dtls.Packet.message_seq` and + held as a :class:`~pcapkit.foundation.reassembly.data.dtls.Fragment` -- + its hole descriptor list + (:class:`~pcapkit.foundation.reassembly.data.dtls.HoleDescriptor`) and + its zero-filled payload buffer. + +Data Models +=========== + +.. module:: pcapkit.foundation.reassembly.data.dtls +.. currentmodule:: pcapkit.foundation.reassembly.dtls + +.. autoclass:: pcapkit.foundation.reassembly.data.dtls.Packet + :members: + :show-inheritance: + +.. autodata:: pcapkit.foundation.reassembly.data.dtls.BufferID + +.. autoclass:: pcapkit.foundation.reassembly.data.dtls.DatagramID + :members: + :show-inheritance: + +.. autoclass:: pcapkit.foundation.reassembly.data.dtls.Datagram + :members: + :show-inheritance: + +.. autoclass:: pcapkit.foundation.reassembly.data.dtls.HoleDescriptor + :members: + :show-inheritance: + +.. autoclass:: pcapkit.foundation.reassembly.data.dtls.Fragment + :members: + :show-inheritance: + +.. autoclass:: pcapkit.foundation.reassembly.data.dtls.Buffer + :members: + :show-inheritance: + +Type Variables +============== + +.. data:: _AT + :module: pcapkit.foundation.reassembly.data.dtls + :type: ipaddress.IPv4Address | ipaddress.IPv6Address + + IP address type. diff --git a/docs/source/pcapkit/foundation/reassembly/index.rst b/docs/source/pcapkit/foundation/reassembly/index.rst index 486f73559f..f2a3d6bb57 100644 --- a/docs/source/pcapkit/foundation/reassembly/index.rst +++ b/docs/source/pcapkit/foundation/reassembly/index.rst @@ -5,8 +5,8 @@ Packets Reassembly .. module:: pcapkit.foundation.reassembly :mod:`pcapkit.foundation.reassembly` implements datagram reassembly -of IP and TCP packets, based on the algorithms described in :rfc:`791` -and :rfc:`815`. +of IP packets, TCP segments and DTLS handshake messages, based on the +algorithms described in :rfc:`791`, :rfc:`815` and :rfc:`6347`. .. seealso:: @@ -19,6 +19,7 @@ and :rfc:`815`. reassembly ip/index tcp + dtls All reassembly classes are implemented as :class:`~pcapkit.foundation.reassembly.reassembly.ReassemblyBase` subclasses, which are responsible for processing extracted packets and @@ -30,7 +31,7 @@ diagram of the class hierarchy of :mod:`pcapkit.foundation.reassembly`: flowchart LR A{{ReassemblyMeta}} -.->|metaclass| B(ReassemblyBase) - B --> IP & TCP + B --> IP & TCP & DTLS IP --> IPv4 & IPv6 B --> C(Reassembly) @@ -45,6 +46,7 @@ diagram of the class hierarchy of :mod:`pcapkit.foundation.reassembly`: click IPv4 "ip/ipv4.html#pcapkit.foundation.reassembly.ipv4.IPv4" click IPv6 "ip/ipv6.html#pcapkit.foundation.reassembly.ipv6.IPv6" click TCP "tcp.html#pcapkit.foundation.reassembly.tcp.TCP" + click DTLS "dtls.html#pcapkit.foundation.reassembly.dtls.DTLS" Auxiliary Data ============== diff --git a/pcapkit/all.py b/pcapkit/all.py index a5a782c8b3..e0e66c56cc 100644 --- a/pcapkit/all.py +++ b/pcapkit/all.py @@ -82,6 +82,7 @@ 'ReassemblyManager', # Reassembly Manager 'IPv4_Reassembly', 'IPv6_Reassembly', # IP Reassembly 'TCP_Reassembly', # TCP Reassembly + 'DTLS_Reassembly', # DTLS Reassembly # pcapkit.foundation.registry 'register_protocol', diff --git a/pcapkit/foundation/__init__.py b/pcapkit/foundation/__init__.py index b60523b63b..1a8789f9ed 100644 --- a/pcapkit/foundation/__init__.py +++ b/pcapkit/foundation/__init__.py @@ -20,7 +20,7 @@ __all__ = [ 'Extractor', - 'IPv4_Reassembly', 'IPv6_Reassembly', 'TCP_Reassembly', + 'IPv4_Reassembly', 'IPv6_Reassembly', 'TCP_Reassembly', 'DTLS_Reassembly', 'ReassemblyManager', 'TCP_TraceFlow', diff --git a/pcapkit/foundation/reassembly/__init__.py b/pcapkit/foundation/reassembly/__init__.py index 4089d68e68..a17ce67ec3 100644 --- a/pcapkit/foundation/reassembly/__init__.py +++ b/pcapkit/foundation/reassembly/__init__.py @@ -6,24 +6,25 @@ .. module:: pcapkit.foundation.reassembly :mod:`pcapkit.foundation.reassembly`, based on algorithms described -in :rfc:`791` and :rfc:`815`, implements datagram reassembly -of IP and TCP packets. +in :rfc:`791`, :rfc:`815` and :rfc:`6347`, implements datagram +reassembly of IP packets, TCP segments and DTLS handshake messages. """ -# Base Class for Reassembly -from pcapkit.foundation.reassembly.reassembly import Reassembly +# Reassembly for DTLS handshake messages +from pcapkit.foundation.reassembly.dtls import DTLS as DTLS_Reassembly from pcapkit.foundation.reassembly.ip import IP as IP_Reassembly - # Reassembly for IP from pcapkit.foundation.reassembly.ipv4 import IPv4 as IPv4_Reassembly from pcapkit.foundation.reassembly.ipv6 import IPv6 as IPv6_Reassembly - +# Base Class for Reassembly +from pcapkit.foundation.reassembly.reassembly import Reassembly # Reassembly for TCP from pcapkit.foundation.reassembly.tcp import TCP as TCP_Reassembly __all__ = [ 'IPv4_Reassembly', 'IPv6_Reassembly', # IP Reassembly 'TCP_Reassembly', # TCP Reassembly + 'DTLS_Reassembly', # DTLS Reassembly 'ReassemblyManager', # Reassembly Manager, defined below ] diff --git a/pcapkit/foundation/reassembly/data/__init__.py b/pcapkit/foundation/reassembly/data/__init__.py index 2d81c5151e..441dca0ebb 100644 --- a/pcapkit/foundation/reassembly/data/__init__.py +++ b/pcapkit/foundation/reassembly/data/__init__.py @@ -20,6 +20,15 @@ from pcapkit.foundation.reassembly.data.tcp import HoleDescriptor as TCP_HoleDescriptor from pcapkit.foundation.reassembly.data.tcp import Packet as TCP_Packet +# DTLS handshake-message reassembly +from pcapkit.foundation.reassembly.data.dtls import Buffer as DTLS_Buffer +from pcapkit.foundation.reassembly.data.dtls import BufferID as DTLS_BufferID +from pcapkit.foundation.reassembly.data.dtls import Datagram as DTLS_Datagram +from pcapkit.foundation.reassembly.data.dtls import DatagramID as DTLS_DatagramID +from pcapkit.foundation.reassembly.data.dtls import Fragment as DTLS_Fragment +from pcapkit.foundation.reassembly.data.dtls import HoleDescriptor as DTLS_HoleDescriptor +from pcapkit.foundation.reassembly.data.dtls import Packet as DTLS_Packet + __all__ = [ 'ReassemblyData', 'Completion', 'Deferred', @@ -28,4 +37,7 @@ 'TCP_Packet', 'TCP_DatagramID', 'TCP_Datagram', 'TCP_Buffer', 'TCP_Fragment', 'TCP_HoleDescriptor', 'TCP_BufferID', + + 'DTLS_Packet', 'DTLS_DatagramID', 'DTLS_Datagram', 'DTLS_Buffer', + 'DTLS_Fragment', 'DTLS_HoleDescriptor', 'DTLS_BufferID', ] diff --git a/pcapkit/foundation/reassembly/data/dtls.py b/pcapkit/foundation/reassembly/data/dtls.py new file mode 100644 index 0000000000..95d681aaf3 --- /dev/null +++ b/pcapkit/foundation/reassembly/data/dtls.py @@ -0,0 +1,182 @@ +# -*- coding: utf-8 -*- +"""data models for DTLS handshake-message reassembly""" + +from typing import TYPE_CHECKING, Generic, TypeVar + +from pcapkit.corekit.infoclass import Info, info_final +from pcapkit.utilities.compat import Tuple + +__all__ = [ + 'Packet', 'DatagramID', 'Datagram', 'HoleDescriptor', 'Fragment', 'Buffer', 'BufferID', +] + +if TYPE_CHECKING: + from ipaddress import IPv4Address, IPv6Address + from typing import Optional + + from typing_extensions import TypeAlias + + from pcapkit.const.tls.handshake_type import HandshakeType as Enum_HandshakeType + from pcapkit.foundation.reassembly.data.data import Completion + +_AT = TypeVar('_AT', 'IPv4Address', 'IPv6Address') + +#: Buffer ID -- the UDP 4-tuple (addresses and ports; the protocol is always +#: UDP, so it is left out), plus the DTLS **epoch**: the record sequence +#: number resets on every change of cipher state (:rfc:`6347#section-4.1`), so +#: a fragment from a new epoch belongs to a fresh buffer rather than the one +#: its 4-tuple would otherwise name. +BufferID: 'TypeAlias' = Tuple[_AT, int, _AT, int, int] + + +@info_final +class Packet(Info, Generic[_AT]): + """Data model for :term:`DTLS ` handshake-fragment + representation.""" + + #: Buffer ID. + bufid: 'BufferID' + #: Original packet range number. + num: 'int' + #: Handshake message type. + msg_type: 'Enum_HandshakeType' + #: Total message length (:rfc:`6347#section-4.2.2`'s ``length``), fragment + #: headers and bodies excluded -- the same on every fragment of one + #: message. + length: 'int' + #: Message sequence number, the datagram key within :attr:`bufid`. + message_seq: 'int' + #: Fragment offset into the reassembled message. + fragment_offset: 'int' + #: Length of this fragment. + fragment_length: 'int' + #: Raw fragment payload. + payload: 'bytes' + #: Capture timestamp of the fragment, in seconds since the Unix epoch, + #: i.e. the *capture's* clock. Recorded only; it does not expire anything. + timestamp: 'float' + + if TYPE_CHECKING: + def __init__(self, bufid: 'tuple[_AT, int, _AT, int, int]', num: 'int', msg_type: 'Enum_HandshakeType', length: 'int', message_seq: 'int', fragment_offset: 'int', fragment_length: 'int', payload: 'bytes', timestamp: 'float') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long + + +@info_final +class DatagramID(Info, Generic[_AT]): + """Data model for :term:`DTLS ` original packet + identifier.""" + + #: Source address. + src: '_AT' + #: Source port. + sport: 'int' + #: Destination address. + dst: '_AT' + #: Destination port. + dport: 'int' + #: Epoch the message was reassembled under. + epoch: 'int' + + if TYPE_CHECKING: + def __init__(self, src: '_AT', sport: 'int', dst: '_AT', dport: 'int', epoch: 'int') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long + + +@info_final +class Datagram(Info, Generic[_AT]): + """Data model for :term:`DTLS ` reassembled + handshake message. + + :attr:`body` is reported eagerly, not through a + :class:`~pcapkit.foundation.reassembly.data.data.Deferred`: a message only + reaches :meth:`DTLS.submit ` + because it was fragmented, so there is no common unfragmented case to + keep cheap. + + """ + + #: How completely the message was reassembled, and why reassembly + #: stopped; see :class:`~pcapkit.foundation.reassembly.data.data.Completion`. + completed: 'Completion' + #: Original packet identifier. + id: 'DatagramID[_AT]' + #: Packet numbers of the fragments that built this message. + index: 'tuple[int, ...]' + #: Message sequence number. + message_seq: 'int' + #: Handshake message type. + msg_type: 'Enum_HandshakeType' + #: Total message length, as declared by its fragments. + length: 'int' + #: Reassembled message body, undecoded -- decoding a ``ClientHello``/ + #: ``ServerHello`` once reassembled is + #: :meth:`DTLS.read `'s job, + #: which alone holds the bound instance :meth:`SSL._read_tls_client_hello + #: ` and + #: :meth:`SSL._read_tls_server_hello + #: ` need. + body: 'bytes | tuple[bytes, ...]' + + if TYPE_CHECKING: + def __init__(self, completed: 'Completion', id: 'DatagramID[_AT]', index: 'tuple[int, ...]', message_seq: 'int', msg_type: 'Enum_HandshakeType', length: 'int', body: 'bytes | tuple[bytes, ...]') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long,redefined-builtin + + +@info_final +class HoleDescriptor(Info): + """Data model for :term:`DTLS ` hole descriptor. + + Both bounds are **inclusive** byte offsets into the reassembled message, + so a hole covers ``last - first + 1`` octets. They never wrap and need no + open-ended sentinel: every fragment declares the message's total + :attr:`Fragment.length` (:rfc:`6347#section-4.2.3`), so it is known from + the first fragment. + + """ + + #: Offset of the first missing octet. + first: 'int' + #: Offset of the last missing octet, inclusive. + last: 'int' + + if TYPE_CHECKING: + def __init__(self, first: 'int', last: 'int') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long + + +@info_final +class Fragment(Info): + """Data model for :term:`DTLS ` message-under- + construction, one per :attr:`Packet.message_seq` buffered under one + :data:`BufferID`.""" + + #: Numbers of the packets merged into this fragment. + ind: 'list[int]' + #: Handshake message type -- read off the first-arriving fragment and + #: trusted from there, same as :attr:`length`. + msg_type: 'Enum_HandshakeType' + #: Total message length, read off the first-arriving fragment. + length: 'int' + #: Reassembled payload, holes left zero-filled. + raw: 'bytearray' + #: Hole descriptor list; empty once the message is whole. + hdl: 'list[HoleDescriptor]' + + if TYPE_CHECKING: + def __init__(self, ind: 'list[int]', msg_type: 'Enum_HandshakeType', length: 'int', raw: 'bytearray', hdl: 'list[HoleDescriptor]') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long + + +@info_final +class Buffer(Info): + """Data model for :term:`DTLS ` buffer entry -- every + handshake message being reassembled under one :data:`BufferID`, + i.e. one UDP association at one epoch.""" + + #: Messages under reconstruction, by :attr:`Packet.message_seq`. Several + #: may be in flight at once: a ``Certificate``, a ``ServerKeyExchange`` + #: and a ``ServerHelloDone`` can each fragment independently in the same + #: flight. + msg: 'dict[int, Fragment]' + #: Capture timestamp of the first fragment buffered under this buffer ID, + #: in seconds since the Unix epoch. Recorded only: the buffer has no + #: time-based deadline, so it is never expired by time. + timestamp: 'float' + + if TYPE_CHECKING: + def __init__(self, msg: 'dict[int, Fragment]', timestamp: 'float') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long diff --git a/pcapkit/foundation/reassembly/dtls.py b/pcapkit/foundation/reassembly/dtls.py new file mode 100644 index 0000000000..1bbdc38273 --- /dev/null +++ b/pcapkit/foundation/reassembly/dtls.py @@ -0,0 +1,318 @@ +# -*- coding: utf-8 -*- +"""DTLS Handshake-Message Reassembly +======================================= + +.. module:: pcapkit.foundation.reassembly.dtls + +:mod:`pcapkit.foundation.reassembly.dtls` contains +:class:`~pcapkit.foundation.reassembly.dtls.DTLS` only, which reconstructs a +DTLS handshake message split across several records back to origin +(:rfc:`6347#section-4.2.3`). A byte range is punched out of the message's +hole list as each fragment arrives, and the message is whole once no hole +remains. + +.. versionadded:: 1.6.0 + +""" +from typing import TYPE_CHECKING, Generic + +from pcapkit.foundation.reassembly.data.data import Completion +from pcapkit.foundation.reassembly.data.dtls import (_AT, Buffer, BufferID, Datagram, DatagramID, + Fragment, HoleDescriptor, Packet) +from pcapkit.foundation.reassembly.reassembly import ReassemblyBase + +if TYPE_CHECKING: + from typing import Optional + +__all__ = ['DTLS'] + + +class DTLS(ReassemblyBase[Packet[_AT], Datagram[_AT], BufferID, Buffer], Generic[_AT]): # pylint: disable=abstract-method + """Reassembly for DTLS handshake messages. + + Args: + strict: if :data:`True`, report a message that is not completely + reassembled as the tuple of its received runs; otherwise as one + contiguous payload, its holes zero-filled + store: if store reassembled datagram in memory, i.e., + :attr:`self._dtgram ` + (if not, datagram will be discarded after callback) + + Note: + The buffer ID is the UDP 4-tuple plus the DTLS **epoch**: the record + sequence number resets on each cipher-state change + (:rfc:`6347#section-4.1`), so a new epoch opens a fresh buffer. + Overlapping fragments are accepted + (:rfc:`6347#section-4.2.3`, :rfc:`9147#section-5.5`). Not toolkit-fed: + a protocol instance never sees the addresses, so a caller supplies the + 4-tuple as ``flow`` to :meth:`DTLS.read + `. + + Example: + >>> from pcapkit.foundation.reassembly.dtls import DTLS + # Initialise instance: + >>> dtls_reassembly = DTLS() + # Call reassembly: + >>> dtls_reassembly(packet_dict) + # Fetch result: + >>> result = dtls_reassembly.datagram + + """ + + ########################################################################## + # Defaults. + ########################################################################## + + #: Protocol name of current reassembly object. Not set via + #: ``__protocol_type__``, which would import + #: :class:`~pcapkit.protocols.application.dtls.DTLS` at class-definition + #: time; :class:`~pcapkit.foundation.reassembly.reassembly.ReassemblyMeta` + #: looks :attr:`name` up in the protocol registry instead. + __protocol_name__ = 'DTLS' + + # No ``__timeout__``: RFC 6347 section 4.2.3 and RFC 9147 section 5.5 give + # DTLS handshake reassembly no timeout, so it inherits the base default, + # :data:`math.inf`, and a buffer is never expired by time. It is held until + # its messages complete or the capture ends. + + ########################################################################## + # Methods. + ########################################################################## + + def __init__(self, *, strict: 'bool' = True, store: 'bool' = True) -> 'None': + """Initialise packet reassembly. + + Args: + strict: if :data:`True`, report a message that is not completely + reassembled as the tuple of its received runs; otherwise as + one contiguous payload, its holes zero-filled + store: if store reassembled datagram in memory + + """ + super().__init__(strict=strict, store=store) + + #: Optional[Datagram[_AT]]: The message, if any, that the most recent + #: call to :meth:`reassembly` completed; see :attr:`last`. + self._last: 'Optional[Datagram[_AT]]' = None + + ########################################################################## + # Properties. + ########################################################################## + + @property + def last(self) -> 'Optional[Datagram[_AT]]': + """The message reassembled by the most recent call to + :meth:`reassembly`, or :data:`None` if that fragment did not complete + one. + + This is a synchronous alternative to :attr:`datagram`/:meth:`fetch` + for a caller -- :meth:`DTLS.read + ` is the one that + exists -- that wants to know, immediately after feeding one fragment, + whether *that* fragment was the one that completed its message, + rather than waiting for the buffer (which may hold several other + messages still in flight under the same buffer ID) to be flushed. + + """ + return self._last + + ########################################################################## + # Methods. + ########################################################################## + + def reassembly(self, info: 'Packet[_AT]') -> 'None': + """Reassembly procedure. + + Arguments: + info: :term:`info` record of packets to be reassembled + + """ + # clear cache + self._flag_n = False + self.__cached__.clear() + self._last = None + + BUFID = info.bufid # Buffer Identifier + MSN = info.message_seq # Message Sequence Number + FO = info.fragment_offset # Fragment Offset + FL = info.fragment_length # Fragment Length + + # No time-based expiry: a buffer is held until its messages complete, + # so a fragment is never abandoned for arriving late. + buf = self._buffer.get(BUFID) + if buf is None: + buf = Buffer(msg={}, timestamp=info.timestamp) + self._buffer[BUFID] = buf + + frag = buf.msg.get(MSN) + # A fresh buffer for this MSN, or a disagreement with the one held + # there: a different ``length`` or ``msg_type`` means this fragment + # cannot belong to the message under construction. A reused 4-tuple + # restarts ``message_seq`` at 0 (:rfc:`6347#section-4.2.2`). Restarting + # prevents merging disagreeing fragments. Known limitation: a late + # fragment of the displaced message whose length and msg_type match + # the new one can still splice into it. + if frag is None or frag.length != info.length or frag.msg_type != info.msg_type: + if frag is not None: + # The displaced partial is reported as PARTIAL rather than dropped. + self._dtgram.extend(self.submit(Buffer(msg={MSN: frag}, timestamp=buf.timestamp), + bufid=BUFID)) + length = info.length + frag = Fragment( + ind=[], + msg_type=info.msg_type, + length=length, + raw=bytearray(length), + hdl=([HoleDescriptor(first=0, last=length - 1)] if length else []), + ) + buf.msg[MSN] = frag + + frag.ind.append(info.num) + + # The fragment's data length is what its header declares, but only + # the octets the capture actually holds can be written, and a write + # past the message's own declared length is clipped. + held = max(min(len(info.payload), FL, len(frag.raw) - FO), 0) + if held: + stop = FO + held + frag.raw[FO:stop] = info.payload[:held] + self._update_hole_descriptors(frag.hdl, FO, stop - 1) + + # the message is whole once no hole remains + if not frag.hdl: + del buf.msg[MSN] + if not buf.msg: + buf = self._buffer.pop(BUFID) + + datagram = self._datagram_from_fragment(frag, BUFID, MSN, Completion.COMPLETE) + self._dtgram.append(datagram) + self._last = datagram + + for callback in self.__callback_fn__: + callback([datagram]) + + def submit(self, buf: 'Buffer', *, bufid: 'BufferID', checked: 'bool' = False, timeout: 'bool' = False) -> 'list[Datagram[_AT]]': # type: ignore[override] # pylint: disable=arguments-differ,unused-argument,line-too-long + """Submit reassembled payload. + + Arguments: + buf: buffer dict of reassembled packets + bufid: buffer identifier + checked: unused; every message's completeness is decided from its + own hole list, so there is nothing for a caller to assert + ahead of time + timeout: whether this buffer is being submitted because + :meth:`~pcapkit.foundation.reassembly.reassembly.Reassembly.expire` + abandoned it under the reassembly timeout + + Returns: + Every message still held in ``buf``, each reported + :attr:`~pcapkit.foundation.reassembly.data.data.Completion.COMPLETE` + if its hole list is empty, otherwise + :attr:`~pcapkit.foundation.reassembly.data.data.Completion.TIMEOUT` + or :attr:`~pcapkit.foundation.reassembly.data.data.Completion.PARTIAL`. + + """ + ret = [] # type: list[Datagram[_AT]] + for (msn, frag) in buf.msg.items(): + if not frag.hdl: + completion = Completion.COMPLETE + else: + completion = Completion.TIMEOUT if timeout else Completion.PARTIAL + ret.append(self._datagram_from_fragment(frag, bufid, msn, completion)) + + for callback in self.__callback_fn__: + callback(ret) + return ret + + def _datagram_from_fragment(self, frag: 'Fragment', bufid: 'BufferID', message_seq: 'int', + completion: 'Completion') -> 'Datagram[_AT]': + """Build a :class:`~pcapkit.foundation.reassembly.data.dtls.Datagram` + from a message's buffered :class:`~pcapkit.foundation.reassembly.data.dtls.Fragment`. + + Arguments: + frag: message-under-construction to report + bufid: buffer identifier it was held under + message_seq: its message sequence number + completion: how completely it was reassembled + + Returns: + The reassembled (or partially reassembled) message. + + """ + if completion: + payload = bytes(frag.raw) # type: 'bytes | tuple[bytes, ...]' + elif self._flag_s: + payload = self._received_runs(frag) + else: + payload = bytes(frag.raw) + + return Datagram( + completed=completion, + id=DatagramID( + src=bufid[0], + sport=bufid[1], + dst=bufid[2], + dport=bufid[3], + epoch=bufid[4], + ), + index=tuple(frag.ind), + message_seq=message_seq, + msg_type=frag.msg_type, + length=frag.length, + body=payload, + ) + + @staticmethod + def _received_runs(frag: 'Fragment') -> 'tuple[bytes, ...]': + """Extract the octet runs actually received, for ``strict=True``'s + report of an incomplete message. + + Arguments: + frag: message-under-construction to extract runs from + + Returns: + Received octet runs, in offset order; empty runs (a hole butting + straight up against the start, the end, or another hole) are left + out. + + """ + runs = [] # type: list[bytes] + pos = 0 + for hole in sorted(frag.hdl, key=lambda hole: hole.first): + if hole.first > pos: + runs.append(bytes(frag.raw[pos:hole.first])) + pos = max(pos, hole.last + 1) + if pos < frag.length: + runs.append(bytes(frag.raw[pos:frag.length])) + return tuple(run for run in runs if run) + + @staticmethod + def _update_hole_descriptors(hdl: 'list[HoleDescriptor]', first: 'int', last: 'int') -> 'None': + """Punch a received byte range out of a message's hole list. + + Offsets are message-relative and inclusive. They never wrap and need + no open upper bound, since every fragment of a message declares the + same total ``length`` (:rfc:`6347#section-4.2.3`). + + Arguments: + hdl: the message's hole descriptor list, mutated in place + first: offset of the first octet the arriving fragment actually + held + last: offset of the last octet the arriving fragment actually + held, inclusive + + """ + new_hdl = [] # type: list[HoleDescriptor] + for hole in hdl: # step one + if first > hole.last or last < hole.first: # steps two and three + new_hdl.append(hole) + continue + # step four: this hole is (partly) filled and replaced by what of + # it survives, on either side of the arriving range + if first > hole.first: # step five + new_hdl.append(HoleDescriptor(first=hole.first, last=first - 1)) + if last < hole.last: # step six + new_hdl.append(HoleDescriptor(first=last + 1, last=hole.last)) + # step seven: go on to the next hole, since one fragment may span + # several of them + hdl[:] = new_hdl diff --git a/pcapkit/protocols/application/dtls.py b/pcapkit/protocols/application/dtls.py index 984008b7ca..458155bd7e 100644 --- a/pcapkit/protocols/application/dtls.py +++ b/pcapkit/protocols/application/dtls.py @@ -16,11 +16,13 @@ Known Limitations ----------------- -**A handshake message split across DTLS records is not reassembled** -(:rfc:`6347#section-4.2.3`, :issue:`2440`). Such a message is flagged -:attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.is_fragmented` -and its :attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.body` -kept as raw octets. +**A handshake message split across DTLS records decodes only with +reassembly** (:rfc:`6347#section-4.2.3`, :issue:`2440`). Otherwise its body +stays raw octets, flagged +:attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.is_fragmented`. +Pass a :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine as +``reassembly`` to decode the completing fragment. The caller supplies the UDP +4-tuple as ``flow``. Only the first record of each datagram is reassembled. **A unified-header record with a connection ID is not decoded** (:rfc:`9147#section-4.1`). Its length is negotiated, not carried in the @@ -33,13 +35,21 @@ .. versionadded:: 1.6.0 +.. versionchanged:: 1.6.0 + + :meth:`read` reassembles a fragmented handshake message and decodes it, + given a :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine via its + new ``reassembly`` argument (:issue:`2440`). + """ from typing import TYPE_CHECKING, cast from pcapkit.const.tls.content_type import ContentType as Enum_ContentType +from pcapkit.const.tls.handshake_type import HandshakeType as Enum_HandshakeType from pcapkit.protocols.application.ssl import DTLS13_VERSION, SSL from pcapkit.protocols.data.application.dtls import DTLS as Data_DTLS from pcapkit.protocols.data.application.dtls import DTLSCiphertext as Data_DTLSCiphertext +from pcapkit.protocols.data.application.dtls import DTLSHandshake as Data_DTLSHandshake from pcapkit.protocols.schema.application.dtls import DTLS as Schema_DTLS from pcapkit.protocols.schema.application.dtls import DTLSCiphertext as Schema_DTLSCiphertext from pcapkit.utilities.exceptions import ProtocolError @@ -51,6 +61,7 @@ from aenum import IntEnum as AenumEnum from typing_extensions import Literal + from pcapkit.foundation.reassembly.dtls import DTLS as DTLS_Reassembly from pcapkit.protocols.protocol import ProtocolBase from pcapkit.protocols.schema.schema import Schema @@ -161,11 +172,25 @@ def id(cls) -> 'tuple[Literal["DTLS"]]': # type: ignore[override] """ return ('DTLS',) - def read(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLS': + def read(self, length: 'Optional[int]' = None, *, reassembly: 'Optional[DTLS_Reassembly[Any]]' = None, + flow: 'Optional[tuple[Any, int, Any, int]]' = None, frame_num: 'int' = 0, + frame_timestamp: 'float' = 0.0, **kwargs: 'Any') -> 'Data_DTLS': """Read Datagram Transport Layer Security (DTLS). Args: length: Length of packet data. + reassembly: A :class:`~pcapkit.foundation.reassembly.dtls.DTLS` + engine to feed fragmented handshake messages to; the fragment + that completes a message is decoded in place. :data:`None` + leaves fragments undecoded. + flow: ``(src, sport, dst, dport)`` of the UDP association, which + with the record's epoch keys the reassembly buffer. Required + when ``reassembly`` is given, so that interleaved associations + do not share a buffer. + frame_num: Packet number, recorded on the reassembled message's + :attr:`~pcapkit.foundation.reassembly.data.dtls.Datagram.index`. + frame_timestamp: Capture timestamp in seconds, recorded on the + fragment; reassembly does not expire buffers by time. **kwargs: Arbitrary keyword arguments. Returns: @@ -173,10 +198,12 @@ def read(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLS': Raises: ProtocolError: If the record cannot be a genuine DTLS record -- - see :meth:`_read_dtls_record`. + see :meth:`_read_dtls_record` -- or if ``reassembly`` is + given without ``flow``. """ - return self._read_dtls_record(length, **kwargs) + return self._read_dtls_record(length, reassembly=reassembly, flow=flow, frame_num=frame_num, + frame_timestamp=frame_timestamp, **kwargs) def make(self, content_type: 'Enum_ContentType | StdlibEnum | AenumEnum | str | int' = Enum_ContentType.application_data, # pylint: disable=line-too-long @@ -297,6 +324,118 @@ def __length_hint__(self) -> 'Literal[13]': # Utilities. ########################################################################## + def _reassemble_dtls_handshake(self, content: 'tuple[Any, ...]', reassembly: 'DTLS_Reassembly[Any]', # pylint: disable=too-many-arguments + flow: 'Optional[tuple[Any, int, Any, int]]', epoch: 'int', + frame_num: 'int', frame_timestamp: 'float') -> 'tuple[Any, ...]': + """Feed a ``handshake`` record's fragmented messages to ``reassembly`` + and decode whichever one it just completed. + + Args: + content: This record's decoded ``handshake`` content, as + :meth:`SSL._read_tls_handshake_messages + ` + built it -- a tuple of + :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`. + reassembly: Engine to reassemble a fragmented message with. + flow: Buffer key, as documented on :meth:`read`, which + enforces this unconditionally before this method is ever + reached -- the check below is only for a direct call that + bypasses it. Required: :data:`None` would default every + caller that omits it onto the same buffer ID, corrupting two + interleaved associations' reassembly together. + epoch: This record's epoch, appended to ``flow`` to form the + reassembly buffer ID (:rfc:`6347#section-4.1`). + frame_num: This record's packet number. + frame_timestamp: This record's capture timestamp. + + Returns: + ``content``, with a message that :meth:`reassembly` completes + decoded as a ``ClientHello`` or ``ServerHello`` in place of its raw + octets; any other message is unchanged. A completed message that + does not decode keeps its raw reassembled bytes as ``body`` + instead of raising. + + Raises: + ProtocolError: If ``flow`` is :data:`None`. + + """ + # NOTE: imported here, as pcapkit.foundation.reassembly imports pcapkit.protocols, + # whose package import reaches this module (a cycle at import time). + from pcapkit.foundation.reassembly.data.dtls import \ + Packet as _ReasmPacket # pylint: disable=import-outside-toplevel + + if flow is None: + raise ProtocolError(f'{self.alias}: flow is required when reassembly is given -- ' + 'defaulting it would collide two concurrent DTLS associations ' + 'on the same buffer ID') + bufid = (*flow, epoch) + + messages = [] + for message in content: + if not isinstance(message, Data_DTLSHandshake) or not message.is_fragmented: + messages.append(message) + continue + + # A fragmented message's body is always raw bytes -- + # :meth:`SSL._read_tls_handshake_messages + # ` + # only decodes one when it is *not* fragmented -- so this is a type + # narrowing for mypy's benefit, not a runtime check. + body = cast('bytes', message.body) + reassembly(_ReasmPacket( + bufid=bufid, + num=frame_num, + msg_type=message.msg_type, + length=message.length, + message_seq=message.message_seq, + fragment_offset=message.fragment_offset, + fragment_length=message.fragment_length, + payload=body, + timestamp=frame_timestamp, + )) + + datagram = reassembly.last + if datagram is None or not datagram.completed: + messages.append(message) + continue + + # A malformed record may have ``fragment_offset + fragment_length`` + # past ``length`` (:rfc:`6347#section-4.2.3`). Then + # :meth:`SSL._make_tls_handshake_messages` cannot reliably slice its + # share back out of the reassembled body, so keep the raw bytes. + if message.fragment_offset + message.fragment_length > message.length: + messages.append(message) + continue + + reassembled = datagram.body + if isinstance(reassembled, tuple): # pragma: no cover -- COMPLETE never reports runs + reassembled = b''.join(reassembled) + + try: + if message.msg_type == Enum_HandshakeType.client_hello: + decoded = self._read_tls_client_hello(reassembled, dtls=True) # type: Any + elif message.msg_type == Enum_HandshakeType.server_hello: + decoded = self._read_tls_server_hello(reassembled) + else: + decoded = reassembled + except ProtocolError: + # The reassembled bytes do not decode as a valid message of + # this ``msg_type`` -- keep them raw rather than failing the + # whole record; without ``reassembly`` the same bytes would + # have stayed undecoded too; see :meth:`read`'s docstring. + decoded = reassembled + + messages.append(Data_DTLSHandshake( + msg_type=message.msg_type, + length=message.length, + body=decoded, + message_seq=message.message_seq, + fragment_offset=message.fragment_offset, + fragment_length=message.fragment_length, + is_fragmented=message.is_fragmented, + )) + return tuple(messages) + @classmethod def _make_data(cls, data: 'Data_DTLS | Data_DTLSCiphertext') -> 'dict[str, Any]': # type: ignore[override] """Create key-value pairs from ``data`` for protocol construction. @@ -327,7 +466,10 @@ def _make_data(cls, data: 'Data_DTLS | Data_DTLSCiphertext') -> 'dict[str, Any]' 'payload': cls._make_payload(data), } - def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLS | Data_DTLSCiphertext': # pylint: disable=unused-argument + def _read_dtls_record(self, length: 'Optional[int]' = None, *, # pylint: disable=unused-argument + reassembly: 'Optional[DTLS_Reassembly[Any]]' = None, + flow: 'Optional[tuple[Any, int, Any, int]]' = None, frame_num: 'int' = 0, + frame_timestamp: 'float' = 0.0, **kwargs: 'Any') -> 'Data_DTLS | Data_DTLSCiphertext': """Read a DTLS record, identifying its concrete tier as it goes. A unified-header record is handed to :meth:`_read_dtls_ciphertext`. @@ -353,6 +495,10 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> Args: length: Length of packet data. + reassembly: Reassembly engine, as documented on :meth:`read`. + flow: Reassembly buffer key, as documented on :meth:`read`. + frame_num: Packet number, as documented on :meth:`read`. + frame_timestamp: Capture timestamp, as documented on :meth:`read`. **kwargs: Arbitrary keyword arguments. Returns: @@ -363,7 +509,8 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> record -- an unrecognised content type, a legacy version outside DTLS's range, or a declared ``length`` that runs past the end of the buffer -- or if the record fragment is - malformed for its content type. + malformed for its content type; or if ``reassembly`` is given + without ``flow``. """ if isinstance(self.__header__, Schema_DTLSCiphertext): @@ -373,6 +520,13 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> available = length if length is not None else len(self) self._validate_tls_record(schema, available=available, dtls=True) + # NOTE: checked for every content type, not only handshake records that reach + # _reassemble_dtls_handshake, so interleaved associations never share a buffer. + if reassembly is not None and flow is None: + raise ProtocolError(f'{self.alias}: flow is required when reassembly is given -- ' + 'defaulting it would collide two concurrent DTLS associations ' + 'on the same buffer ID') + # NOTE: DTLS 1.3 is checked first, at 0xFEFF too: an initial ClientHello may carry # {254, 255} (RFC 9147, section 4). Unknown versions keep the generic alias. if schema.version in _DTLS13_LEGACY_VERSIONS and self._tls_negotiates(schema, DTLS13_VERSION, dtls=True): @@ -386,6 +540,12 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> content = self._read_tls_content(schema.content_type, schema.fragment, dtls=True, epoch=schema.epoch) + # NOTE: a handshake record past epoch 0 is ciphertext, so ``content`` is raw + # bytes rather than a tuple of messages, and is passed through unchanged. + if (reassembly is not None and schema.content_type == Enum_ContentType.handshake + and isinstance(content, tuple)): + content = self._reassemble_dtls_handshake(content, reassembly, flow, schema.epoch, + frame_num, frame_timestamp) dtls = Data_DTLS( content_type=schema.content_type, version=schema.version, diff --git a/pcapkit/protocols/application/ssl.py b/pcapkit/protocols/application/ssl.py index 17bb980ab8..ebdf70d140 100644 --- a/pcapkit/protocols/application/ssl.py +++ b/pcapkit/protocols/application/ssl.py @@ -394,9 +394,12 @@ def _read_tls_handshake_messages(self, fragment: 'bytes', *, ``fragment``. Notes: - Only an unfragmented ``ClientHello``/``ServerHello`` body is - decoded; a fragmented DTLS message is not reassembled - (:issue:`2440`), and other bodies stay undecoded (:issue:`2437`). + Only a whole ``ClientHello``/``ServerHello`` body is decoded: an + unfragmented one, or a fragmented DTLS message completed by a + :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine passed as + :meth:`DTLS.read `'s + ``reassembly`` (:issue:`2440`). Other bodies stay undecoded + (:issue:`2437`). """ from pcapkit.protocols.data.application.dtls import \ @@ -446,6 +449,61 @@ def _read_tls_handshake_messages(self, fragment: 'bytes', *, messages.append(Data_Handshake(msg_type=msg_type, length=length, body=body)) return tuple(messages) + def _make_tls_handshake_messages(self, messages: 'tuple[Any, ...]', *, + dtls: 'bool') -> 'bytes': + """Encode handshake messages back into a record fragment -- the + inverse of :meth:`_read_tls_handshake_messages`. + + Args: + messages: Handshake messages, as returned by + :meth:`_read_tls_handshake_messages`, or already-encoded + :obj:`bytes` per message. + dtls: Whether to write DTLS's fragment header after each + message's ``msg_type``/``length`` header. + + Returns: + Encoded record fragment. + + """ + out = bytearray() + for message in messages: + if isinstance(message, bytes): + out += message + continue + + body = message.body + if isinstance(body, Data_ClientHello): + body_bytes = self._make_tls_client_hello(body, dtls=dtls) + elif isinstance(body, Data_ServerHello): + body_bytes = self._make_tls_server_hello(body) + else: + body_bytes = bytes(body) + + # NOTE: the message's ``length`` is trusted only for a fragmented DTLS message; + # otherwise it is recomputed from ``body_bytes``, so a rebuild stays exact. + is_fragmented = dtls and message.is_fragmented + length = message.length if is_fragmented else len(body_bytes) + + if is_fragmented: + fragment_length = message.fragment_length + # ``body_bytes`` is this fragment's own octets, unless decoding the + # completing fragment replaced it with the full reassembled message + # (see :meth:`DTLS._reassemble_dtls_handshake`). Slice back to this + # record's own byte range so parse-then-rebuild stays byte-exact. + if len(body_bytes) != fragment_length: + start = message.fragment_offset + body_bytes = body_bytes[start:start + fragment_length] + else: + fragment_length = len(body_bytes) + + out += bytes([int(message.msg_type)]) + length.to_bytes(3, 'big') + if dtls: + out += (message.message_seq.to_bytes(2, 'big') + + message.fragment_offset.to_bytes(3, 'big') + + fragment_length.to_bytes(3, 'big')) + out += body_bytes + return bytes(out) + def _read_tls_client_hello(self, body: 'bytes', *, dtls: 'bool') -> 'Data_ClientHello': """Decode a ``ClientHello`` handshake message body. @@ -671,50 +729,6 @@ def _make_tls_content(self, content_type: 'Enum_ContentType', content: 'Any', *, return bytes([int(content.msg)]) return bytes(content) - def _make_tls_handshake_messages(self, messages: 'tuple[Any, ...]', *, - dtls: 'bool') -> 'bytes': - """Encode handshake messages back into a record fragment -- the - inverse of :meth:`_read_tls_handshake_messages`. - - Args: - messages: Handshake messages, as returned by - :meth:`_read_tls_handshake_messages`, or already-encoded - :obj:`bytes` per message. - dtls: Whether to write DTLS's fragment header after each - message's ``msg_type``/``length`` header. - - Returns: - Encoded record fragment. - - """ - out = bytearray() - for message in messages: - if isinstance(message, bytes): - out += message - continue - - body = message.body - if isinstance(body, Data_ClientHello): - body_bytes = self._make_tls_client_hello(body, dtls=dtls) - elif isinstance(body, Data_ServerHello): - body_bytes = self._make_tls_server_hello(body) - else: - body_bytes = bytes(body) - - # NOTE: the message's ``length`` is trusted only for a fragmented DTLS message; - # otherwise it is recomputed from ``body_bytes``, so a rebuild stays exact. - is_fragmented = dtls and message.is_fragmented - length = message.length if is_fragmented else len(body_bytes) - - out += bytes([int(message.msg_type)]) + length.to_bytes(3, 'big') - if dtls: - fragment_length = message.fragment_length if is_fragmented else len(body_bytes) - out += (message.message_seq.to_bytes(2, 'big') - + message.fragment_offset.to_bytes(3, 'big') - + fragment_length.to_bytes(3, 'big')) - out += body_bytes - return bytes(out) - def _make_tls_client_hello(self, data: 'Data_ClientHello', *, dtls: 'bool') -> 'bytes': """Encode a ``ClientHello`` handshake message body -- the inverse of :meth:`_read_tls_client_hello`. diff --git a/pcapkit/protocols/data/application/dtls.py b/pcapkit/protocols/data/application/dtls.py index 96e65ee8c0..6dc9459c1f 100644 --- a/pcapkit/protocols/data/application/dtls.py +++ b/pcapkit/protocols/data/application/dtls.py @@ -33,12 +33,13 @@ class DTLSHandshake(Handshake): in one datagram can be split across several. :attr:`body` decodes to a :class:`~pcapkit.protocols.data.application.tls.ClientHello` - or :class:`~pcapkit.protocols.data.application.tls.ServerHello` only when - :attr:`is_fragmented` is :data:`False`; a genuinely fragmented message is - not reassembled (:issue:`2440`; see - :class:`~pcapkit.protocols.application.dtls.DTLS` for why), so its - :attr:`body` is the single fragment's raw, undecoded octets regardless of - :attr:`msg_type`. + or :class:`~pcapkit.protocols.data.application.tls.ServerHello` when + :attr:`is_fragmented` is :data:`False`, or when a + :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine passed to + :meth:`DTLS.read ` completed + the message on this fragment (:issue:`2440`). Otherwise a fragmented + message's :attr:`body` is the fragment's raw octets. :attr:`is_fragmented` + reports whether the message was split on the wire, either way. """ diff --git a/tests/foundation/reassembly/test_dtls_handshake_reassembly_2440_unit.py b/tests/foundation/reassembly/test_dtls_handshake_reassembly_2440_unit.py new file mode 100644 index 0000000000..3473125188 --- /dev/null +++ b/tests/foundation/reassembly/test_dtls_handshake_reassembly_2440_unit.py @@ -0,0 +1,669 @@ +# -*- coding: utf-8 -*- +"""DTLS handshake-message reassembly (GitHub issue #2440). + +Covers the new :class:`~pcapkit.foundation.reassembly.dtls.DTLS` engine in +isolation, fed fragments directly, and the wiring into +:meth:`~pcapkit.protocols.application.dtls.DTLS.read`, which decodes a +fragmented ``ClientHello``/``ServerHello`` once its reassembly engine reports +the message complete. + +Every case builds its fragments in memory and reads no capture. Classes are +imported inside each test, after :func:`~tests._support.reimport_once_per_class`. + +""" + +from __future__ import annotations + +import unittest + +from tests._support import reimport_once_per_class + +#: Test tier (#1958): hand-picked round-trip cases, not a sweep. +__tier__ = 'ALL' + + +def _client_hello_body(cipher_suites: bytes = b'\x13\x01', extensions: 'bytes | None' = b'', + cookie: bytes = b'') -> bytes: + """A genuine DTLS ``ClientHello`` body (:rfc:`6347#section-4.2.1`'s + ``cookie`` included).""" + version = b'\xfe\xfd' + random = bytes(range(32)) + session_id = b'' + compression = b'\x00' + tail = b'' if extensions is None else len(extensions).to_bytes(2, 'big') + extensions + return (version + random + bytes([len(session_id)]) + session_id + + bytes([len(cookie)]) + cookie + + len(cipher_suites).to_bytes(2, 'big') + cipher_suites + + bytes([len(compression)]) + compression + + tail) + + +def _server_hello_body(extensions: 'bytes | None' = b'') -> bytes: + version = b'\xfe\xfd' + random = bytes(range(32, 64)) + session_id = b'' + cipher_suite = b'\x13\x01' + compression = b'\x00' + tail = b'' if extensions is None else len(extensions).to_bytes(2, 'big') + extensions + return (version + random + bytes([len(session_id)]) + session_id + + cipher_suite + compression + tail) + + +def _dtls_record(content_type: int, fragment: bytes, version: bytes = b'\xfe\xfd', + epoch: int = 0, seq: int = 1) -> bytes: + return (bytes([content_type]) + version + epoch.to_bytes(2, 'big') + seq.to_bytes(6, 'big') + + len(fragment).to_bytes(2, 'big') + fragment) + + +def _dtls_handshake(msg_type: int, body: bytes, *, message_seq: int, fragment_offset: int, + fragment_length: int, total_length: 'int | None' = None) -> bytes: + length = total_length if total_length is not None else len(body) + header = bytes([msg_type]) + length.to_bytes(3, 'big') + frag_header = (message_seq.to_bytes(2, 'big') + fragment_offset.to_bytes(3, 'big') + + fragment_length.to_bytes(3, 'big')) + return header + frag_header + body + + +def _split(body: bytes, bounds: 'tuple[int, ...]') -> 'list[tuple[int, int]]': + """``bounds`` offsets (not including 0 or ``len(body)``) into ``(offset, + length)`` pairs covering the whole of ``body``.""" + edges = (0, *bounds, len(body)) + return [(edges[i], edges[i + 1] - edges[i]) for i in range(len(edges) - 1)] + + +class TestDTLSHandshakeReassembly(unittest.TestCase): + """The reassembly engine, fed fragments directly.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + + @staticmethod + def _engine(strict: bool = True): + from pcapkit.foundation.reassembly.dtls import DTLS + + return DTLS(strict=strict) + + @staticmethod + def _packet(bufid, num, msg_type, length, message_seq, fragment_offset, fragment_length, + payload, timestamp=0.0): + from pcapkit.foundation.reassembly.data.dtls import Packet + + return Packet(bufid=bufid, num=num, msg_type=msg_type, length=length, + message_seq=message_seq, fragment_offset=fragment_offset, + fragment_length=fragment_length, payload=payload, timestamp=timestamp) + + def test_two_fragments_reassemble_byte_exact(self) -> None: + from pcapkit.const.tls.handshake_type import HandshakeType + + body = _client_hello_body(cookie=b'\xab\xcd\xef\x01') + half = len(body) // 2 + + for order in ([0, 1], [1, 0]): + with self.subTest(order=order): + engine = self._engine() + bufid = ('10.0.0.1', 1234, '10.0.0.2', 5684, 0) + frags = [(0, body[:half]), (half, body[half:])] + last = None + for num, idx in enumerate(order, 1): + fo, payload = frags[idx] + engine(self._packet(bufid, num, HandshakeType.client_hello, len(body), + 5, fo, len(payload), payload, float(num))) + if engine.last is not None: + last = engine.last + self.assertIsNotNone(last) + self.assertTrue(last.completed) + self.assertEqual(last.body, body) + self.assertEqual(last.message_seq, 5) + self.assertEqual(sorted(last.index), [1, 2]) + + def test_three_fragments_out_of_order_reassemble_byte_exact(self) -> None: + from pcapkit.const.tls.handshake_type import HandshakeType + + body = _server_hello_body(extensions=b'\x00\x0d\x00\x02\xab\xcd') + third = len(body) // 3 + frags = _split(body, (third, 2 * third)) + + for order in ([0, 1, 2], [2, 0, 1], [1, 2, 0]): + with self.subTest(order=order): + engine = self._engine() + bufid = ('1.1.1.1', 100, '2.2.2.2', 5684, 0) + last = None + for num, idx in enumerate(order, 1): + fo, length = frags[idx] + payload = body[fo:fo + length] + engine(self._packet(bufid, num, HandshakeType.server_hello, len(body), + 9, fo, length, payload, float(num))) + if engine.last is not None: + last = engine.last + self.assertIsNotNone(last) + self.assertTrue(last.completed) + self.assertEqual(last.body, body) + + def test_overlapping_fragments_are_accepted_and_complete_byte_exact(self) -> None: + """Overlapping fragments are accepted, not dropped + (:rfc:`6347#section-4.2.3`, :rfc:`9147#section-5.5`): a partial + overlap, a smaller retransmission inside held bytes, and a duplicate + all leave the message completing byte-exact on its last fragment. + Dropping overlapping fragments leaves a hole, so it never completes.""" + from pcapkit.const.tls.handshake_type import HandshakeType + + body = bytes(range(16)) + bufid = ('30.0.0.1', 1, '30.0.0.2', 5684, 0) + # (offset, length): a start, a partial overlap, a smaller + # retransmission inside held bytes, a duplicate, then the last one. + sequence = [(0, 8), (6, 6), (2, 4), (6, 6), (12, 4)] + + engine = self._engine() + for num, (fo, length) in enumerate(sequence, 1): + engine(self._packet(bufid, num, HandshakeType.client_hello, len(body), + 7, fo, length, body[fo:fo + length], float(num))) + if num < len(sequence): + self.assertIsNone(engine.last) + self.assertIsNotNone(engine.last) + self.assertTrue(engine.last.completed) + self.assertEqual(engine.last.body, body) + + def test_epoch_change_opens_a_fresh_buffer(self) -> None: + """An epoch change opens a fresh buffer (:rfc:`6347#section-4.1`): + a fragment at a new epoch must not be folded into the previous + epoch's -- otherwise unrelated -- buffer, even if its message_seq + and fragment offsets happen to agree with an incomplete message + there.""" + from pcapkit.const.tls.handshake_type import HandshakeType + + body = _client_hello_body() + half = len(body) // 2 + addr = ('5.5.5.5', 1, '6.6.6.6', 5684) + + engine = self._engine() + # epoch 0: only the first half ever arrives -- never completes. + engine(self._packet((*addr, 0), 1, HandshakeType.client_hello, len(body), + 0, 0, half, body[:half], 0.0)) + self.assertIsNone(engine.last) + + # epoch 1 reuses the same message_seq (0) and the same 4-tuple, but a + # *whole* message in one fragment -- it must complete independently. + engine(self._packet((*addr, 1), 2, HandshakeType.client_hello, len(body), + 0, 0, len(body), body, 0.1)) + self.assertIsNotNone(engine.last) + self.assertTrue(engine.last.completed) + self.assertEqual(engine.last.id.epoch, 1) + self.assertEqual(engine.last.body, body) + + # the epoch-0 buffer is still there, untouched and still incomplete. + remaining = engine.datagram + partial = [d for d in remaining if d.id.epoch == 0] + self.assertEqual(len(partial), 1) + self.assertEqual(partial[0].completed.name, 'PARTIAL') + + def test_incomplete_reassembly_never_completes_and_does_not_crash(self) -> None: + """A message missing a fragment forever stays unavailable -- reported + ``PARTIAL`` on fetch, rather than raising or + fabricating a result.""" + from pcapkit.const.tls.handshake_type import HandshakeType + + body = _server_hello_body() + third = len(body) // 3 + frags = _split(body, (third, 2 * third)) + bufid = ('7.7.7.7', 1, '8.8.8.8', 5684, 0) + + for strict in (True, False): + with self.subTest(strict=strict): + engine = self._engine(strict=strict) + # only fragments 0 and 2 ever arrive; fragment 1 is lost. + for num, idx in enumerate((0, 2), 1): + fo, length = frags[idx] + payload = body[fo:fo + length] + engine(self._packet(bufid, num, HandshakeType.server_hello, len(body), + 1, fo, length, payload, float(num))) + self.assertIsNone(engine.last) + + datagram, = engine.datagram + self.assertEqual(datagram.completed.name, 'PARTIAL') + if strict: + self.assertIsInstance(datagram.body, tuple) + self.assertEqual(b''.join(datagram.body), + body[:frags[0][1]] + body[frags[2][0]:]) + else: + self.assertIsInstance(datagram.body, bytes) + self.assertEqual(len(datagram.body), len(body)) + self.assertEqual(datagram.body[:frags[0][1]], body[:frags[0][1]]) + self.assertEqual(datagram.body[frags[2][0]:], body[frags[2][0]:]) + + def test_several_messages_in_flight_under_one_buffer_id(self) -> None: + """A ``Certificate`` and a ``ServerHelloDone`` fragmenting + independently in the same flight share a buffer ID but must not + share -- or complete into -- each other's message.""" + from pcapkit.const.tls.handshake_type import HandshakeType + + cert_body = b'C' * 40 + shd_body = b'D' * 20 + bufid = ('9.9.9.9', 1, '10.10.10.10', 5684, 0) + + engine = self._engine() + engine(self._packet(bufid, 1, HandshakeType.certificate, len(cert_body), + 1, 0, 20, cert_body[:20], 0.0)) + self.assertIsNone(engine.last) + engine(self._packet(bufid, 2, HandshakeType.server_hello_done, len(shd_body), + 2, 0, 20, shd_body, 0.1)) + self.assertIsNotNone(engine.last) + self.assertEqual(engine.last.message_seq, 2) + self.assertEqual(engine.last.body, shd_body) + + engine(self._packet(bufid, 3, HandshakeType.certificate, len(cert_body), + 1, 20, 20, cert_body[20:], 0.2)) + self.assertIsNotNone(engine.last) + self.assertEqual(engine.last.message_seq, 1) + self.assertEqual(engine.last.body, cert_body) + + def test_disagreeing_fragment_restarts_the_message_instead_of_corrupting_it(self) -> None: + """A fragment that disagrees with the message buffered under + its ``message_seq`` -- a different ``length`` or ``msg_type`` -- must + not be merged into it: a stale, abandoned buffer from an earlier + handshake sharing the same :data:`BufferID` and restarted + ``message_seq`` (:rfc:`6347#section-4.2.2`) would otherwise combine + with the new handshake's fragment into one corrupt message.""" + from pcapkit.const.tls.handshake_type import HandshakeType + + old_body = _client_hello_body(cookie=b'\x01' * 4) + new_body = _client_hello_body(cookie=b'\x02' * 8) + bufid = ('1.2.3.4', 1, '5.6.7.8', 5684, 0) + + engine = self._engine() + # the old handshake's message_seq=0 fragment never completes. + half = len(old_body) // 2 + engine(self._packet(bufid, 1, HandshakeType.client_hello, len(old_body), + 0, 0, half, old_body[:half], 0.0)) + self.assertIsNone(engine.last) + + # a new handshake reuses the same BufferID and message_seq, but + # disagrees on length -- the new fragment must start a fresh + # message, not fill the old one's remaining hole. + engine(self._packet(bufid, 2, HandshakeType.client_hello, len(new_body), + 0, 0, len(new_body), new_body, 0.1)) + self.assertIsNotNone(engine.last) + self.assertTrue(engine.last.completed) + self.assertEqual(engine.last.body, new_body) + + def test_restart_on_msg_type_disagreement_alone(self) -> None: + """A fragment disagreeing only on ``msg_type`` -- same ``length`` -- + must restart the message too, not just one disagreeing on + ``length``.""" + from pcapkit.const.tls.handshake_type import HandshakeType + + length = 20 + old_partial = b'A' * 10 + new_whole = b'B' * length + bufid = ('15.0.0.1', 1, '15.0.0.2', 5684, 0) + + engine = self._engine() + # the old handshake's message_seq=0 fragment (a Certificate) never + # completes -- same length as the new message below, different + # msg_type. + engine(self._packet(bufid, 1, HandshakeType.certificate, length, + 0, 0, 10, old_partial, 0.0)) + self.assertIsNone(engine.last) + + # a new handshake reuses the same BufferID and message_seq, agrees + # on length, but disagrees on msg_type -- it must still start a + # fresh message rather than filling the old Certificate's hole. + engine(self._packet(bufid, 2, HandshakeType.server_hello_done, length, + 0, 0, length, new_whole, 0.1)) + self.assertIsNotNone(engine.last) + self.assertTrue(engine.last.completed) + self.assertEqual(engine.last.msg_type, HandshakeType.server_hello_done) + self.assertEqual(engine.last.body, new_whole) + + def test_displaced_partial_is_reported_not_silently_dropped(self) -> None: + """The partial message a disagreeing fragment displaces is reported + as its own, PARTIAL result rather than vanishing with no trace.""" + from pcapkit.const.tls.handshake_type import HandshakeType + + old_body = _client_hello_body(cookie=b'\x01' * 4) + new_body = _client_hello_body(cookie=b'\x02' * 8) + bufid = ('16.0.0.1', 1, '16.0.0.2', 5684, 0) + + engine = self._engine() + half = len(old_body) // 2 + engine(self._packet(bufid, 1, HandshakeType.client_hello, len(old_body), + 0, 0, half, old_body[:half], 0.0)) + self.assertIsNone(engine.last) + + engine(self._packet(bufid, 2, HandshakeType.client_hello, len(new_body), + 0, 0, len(new_body), new_body, 0.1)) + self.assertIsNotNone(engine.last) + self.assertTrue(engine.last.completed) + + results = engine.fetch() + self.assertEqual(len(results), 2) + completed = [d for d in results if d.completed.name == 'COMPLETE'] + partial = [d for d in results if d.completed.name == 'PARTIAL'] + self.assertEqual(len(completed), 1) + self.assertEqual(completed[0].body, new_body) + self.assertEqual(len(partial), 1) + self.assertEqual(partial[0].index, (1,)) + + def test_stale_buffer_is_held_not_expired_by_time(self) -> None: + """A buffer whose first fragment is arbitrarily old is never abandoned + as :attr:`~pcapkit.foundation.reassembly.data.data.Completion.TIMEOUT`: + DTLS handshake reassembly has no time-based deadline + (:rfc:`6347#section-4.2.3`, :rfc:`9147#section-5.5`). Its late + fragments still complete it.""" + from pcapkit.const.tls.handshake_type import HandshakeType + + body = _server_hello_body() + third = len(body) // 3 + frags = _split(body, (third, 2 * third)) + bufid = ('17.0.0.1', 1, '17.0.0.2', 5684, 0) + other_bufid = ('18.0.0.1', 2, '18.0.0.2', 5684, 0) + + engine = self._engine() + fo, length = frags[0] + engine(self._packet(bufid, 1, HandshakeType.server_hello, len(body), + 1, fo, length, body[fo:fo + length], 0.0)) + self.assertIsNone(engine.last) + + # A fragment under an UNRELATED buffer, a million seconds later, must + # not abandon the first buffer. + fo2, length2 = frags[0] + engine(self._packet(other_bufid, 2, HandshakeType.server_hello, len(body), + 1, fo2, length2, body[fo2:fo2 + length2], 1e6)) + + # The late fragments complete the stale message as COMPLETE. Reading + # ``engine.datagram`` flushes held buffers, so it is read only after. + for num, idx in ((3, 1), (4, 2)): + fo3, length3 = frags[idx] + engine(self._packet(bufid, num, HandshakeType.server_hello, len(body), + 1, fo3, length3, body[fo3:fo3 + length3], 1e6 + num)) + self.assertIsNotNone(engine.last) + self.assertEqual(engine.last.completed.name, 'COMPLETE') + self.assertEqual(engine.last.body, body) + stale = [d for d in engine.datagram if d.id.src == '17.0.0.1'] + self.assertEqual([d.completed.name for d in stale], ['COMPLETE']) + + def test_delayed_fragments_complete_with_no_deadline(self) -> None: + """Fragments arriving minutes apart still complete their message: with + no time-based deadline, a delay between fragments abandons nothing.""" + from pcapkit.const.tls.handshake_type import HandshakeType + + body = _client_hello_body(cookie=b'\xaa\xbb\xcc\xdd') + third = len(body) // 3 + frags = _split(body, (third, 2 * third)) + bufid = ('19.0.0.1', 1, '19.0.0.2', 5684, 0) + + # Fragments at t=0, t=100, t=200 seconds of capture time. + engine = self._engine() + for i, (fo, length) in enumerate(frags): + payload = body[fo:fo + length] + timestamp = float(i * 100) + engine(self._packet(bufid, i + 1, HandshakeType.client_hello, len(body), + 0, fo, length, payload, timestamp)) + if engine.last is not None: + # Message completed on this fragment. + break + + self.assertIsNotNone(engine.last) + self.assertTrue(engine.last.completed) + self.assertEqual(engine.last.body, body) + + +class TestDTLSReadReassemblyWiring(unittest.TestCase): + """:meth:`DTLS.read`'s ``reassembly``/``flow`` arguments, exercised + through the real protocol class rather than the engine alone.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + + def test_fragmented_client_hello_decodes_once_reassembled(self) -> None: + from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + from pcapkit.protocols.application.dtls import DTLS + + body = _client_hello_body(cookie=b'\xab\xcd\xef\x01') + half = len(body) // 2 + wire1 = _dtls_record(22, _dtls_handshake(1, body[:half], message_seq=5, fragment_offset=0, + fragment_length=half, total_length=len(body)), seq=2) + wire2 = _dtls_record(22, _dtls_handshake(1, body[half:], message_seq=5, fragment_offset=half, + fragment_length=len(body) - half, total_length=len(body)), seq=3) + + engine = DTLSReassembly() + flow = ('10.0.0.1', 1234, '10.0.0.2', 5684) + + parsed1 = DTLS(wire1, len(wire1), reassembly=engine, flow=flow, frame_num=1, frame_timestamp=0.0) + msg1 = parsed1.info.content[0] + self.assertTrue(msg1.is_fragmented) + self.assertIsInstance(msg1.body, bytes) + + parsed2 = DTLS(wire2, len(wire2), reassembly=engine, flow=flow, frame_num=2, frame_timestamp=0.1) + msg2 = parsed2.info.content[0] + self.assertTrue(msg2.is_fragmented) + self.assertNotIsInstance(msg2.body, bytes) + self.assertEqual(msg2.body.cookie, b'\xab\xcd\xef\x01') + + def test_fragmented_server_hello_decodes_once_reassembled(self) -> None: + from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + from pcapkit.protocols.application.dtls import DTLS + + body = _server_hello_body(extensions=b'\x00\x0d\x00\x02\xab\xcd') + third = len(body) // 3 + frags = _split(body, (third, 2 * third)) + bufid_flow = ('1.1.1.1', 100, '2.2.2.2', 5684) + + engine = DTLSReassembly() + decoded = None + # deliver out of order: [2, 0, 1] + for num, idx in enumerate((2, 0, 1), 1): + fo, length = frags[idx] + wire = _dtls_record(22, _dtls_handshake(2, body[fo:fo + length], message_seq=9, + fragment_offset=fo, fragment_length=length, total_length=len(body)), + seq=10 + num) + parsed = DTLS(wire, len(wire), reassembly=engine, flow=bufid_flow, + frame_num=num, frame_timestamp=float(num)) + msg = parsed.info.content[0] + if not isinstance(msg.body, bytes): + decoded = msg.body + self.assertIsNotNone(decoded) + self.assertEqual(decoded.random, bytes(range(32, 64))) + + def test_reassembly_without_flow_raises_rather_than_corrupting(self) -> None: + """Omitting ``flow`` while passing ``reassembly`` must not silently + default onto a shared buffer key -- two interleaved associations + would otherwise corrupt each other's reassembly.""" + from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + from pcapkit.protocols.application.dtls import DTLS + from pcapkit.utilities.exceptions import ProtocolError + + body = _client_hello_body() + half = len(body) // 2 + wire = _dtls_record(22, _dtls_handshake(1, body[:half], message_seq=5, fragment_offset=0, + fragment_length=half, total_length=len(body)), seq=2) + + engine = DTLSReassembly() + with self.assertRaises(ProtocolError): + DTLS(wire, len(wire), reassembly=engine, frame_num=1, frame_timestamp=0.0) + + def test_encrypted_handshake_record_skips_reassembly_unharmed(self) -> None: + """A ``handshake``-typed record at epoch >= 1 is ciphertext -- + :meth:`SSL._read_tls_content ` + returns it as raw :obj:`bytes` rather than parsed messages. Every + real DTLS handshake ends with exactly such a record (the encrypted + ``Finished``), so a ``reassembly`` engine must not choke on it.""" + from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + from pcapkit.protocols.application.dtls import DTLS + + ciphertext = b'\xa3\x5d\x12\xee' * 12 + wire = _dtls_record(22, ciphertext, epoch=1, seq=7) + + engine = DTLSReassembly() + flow = ('10.0.0.1', 1234, '10.0.0.2', 5684) + + parsed = DTLS(wire, len(wire), reassembly=engine, flow=flow, + frame_num=1, frame_timestamp=0.0) + self.assertEqual(parsed.info.content, ciphertext) + + def test_completing_record_rebuilds_byte_exact(self) -> None: + """The record that completes a reassembled message must still + round-trip exactly through ``from_data``/``make``: + the fragment header has to keep describing *this* record's own byte + range even though ``body`` holds the full decoded message, or + re-encoding it drifts out of sync with the bytes written after it.""" + from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + from pcapkit.protocols.application.dtls import DTLS + + body = _client_hello_body(cookie=b'\xab\xcd\xef\x01') + half = len(body) // 2 + wire1 = _dtls_record(22, _dtls_handshake(1, body[:half], message_seq=5, fragment_offset=0, + fragment_length=half, total_length=len(body)), seq=2) + wire2 = _dtls_record(22, _dtls_handshake(1, body[half:], message_seq=5, fragment_offset=half, + fragment_length=len(body) - half, total_length=len(body)), seq=3) + + engine = DTLSReassembly() + flow = ('10.0.0.1', 1234, '10.0.0.2', 5684) + + DTLS(wire1, len(wire1), reassembly=engine, flow=flow, frame_num=1, frame_timestamp=0.0) + parsed2 = DTLS(wire2, len(wire2), reassembly=engine, flow=flow, frame_num=2, frame_timestamp=0.1) + + msg2 = parsed2.info.content[0] + # the completing fragment: decoded in place, not raw bytes. + self.assertNotIsInstance(msg2.body, bytes) + + rebuilt = bytes(DTLS.from_data(parsed2.info)) + self.assertEqual(rebuilt, wire2) + + def test_overrunning_completing_fragment_keeps_raw_bytes_and_round_trips(self) -> None: + """A fragment whose ``fragment_offset + fragment_length`` overruns the + declared ``length`` (5 + 10 > 10) completes its message, but its record + must stay as it arrived. Replacing its body with the clipped message + corrupts the round trip, and + :meth:`SSL._make_tls_handshake_messages + `'s + exactness check cannot be relied on to catch it.""" + from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + from pcapkit.protocols.application.dtls import DTLS + + length = 10 + first = b'A' * 5 + second = b'B' * 10 # this record's own 10 octets, but offset 5 + 10 > length 10 + wire1 = _dtls_handshake(11, first, message_seq=3, fragment_offset=0, + fragment_length=5, total_length=length) + wire2 = _dtls_handshake(11, second, message_seq=3, fragment_offset=5, + fragment_length=10, total_length=length) + record1 = _dtls_record(22, wire1, seq=2) + record2 = _dtls_record(22, wire2, seq=3) + + engine = DTLSReassembly() + flow = ('19.0.0.1', 1, '19.0.0.2', 5684) + + DTLS(record1, len(record1), reassembly=engine, flow=flow, frame_num=1, frame_timestamp=0.0) + parsed2 = DTLS(record2, len(record2), reassembly=engine, flow=flow, frame_num=2, + frame_timestamp=0.1) + + # the engine itself did complete the message (clipping the overrun) + # but this record is left exactly as it arrived. + self.assertIsNotNone(engine.last) + self.assertTrue(engine.last.completed) + + msg2 = parsed2.info.content[0] + self.assertIsInstance(msg2.body, bytes) + self.assertEqual(msg2.body, second) + self.assertEqual(msg2.fragment_offset, 5) + self.assertEqual(msg2.fragment_length, 10) + + rebuilt = bytes(DTLS.from_data(parsed2.info)) + self.assertEqual(rebuilt, record2) + + def test_undecodable_reassembled_body_falls_back_to_raw_bytes(self) -> None: + """A completed message whose reassembled bytes do not decode as a + valid message of its ``msg_type`` keeps those raw bytes instead of + raising (dtls.py's ``except ProtocolError`` fallback).""" + from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + from pcapkit.protocols.application.dtls import DTLS + + # far too short to be a genuine ClientHello (needs >= 34 octets + # before even the session_id length is reached), but a perfectly + # in-bounds pair of fragments for the reassembler itself. + body = b'\x01\x02\x03\x04\x05' + wire1 = _dtls_record(22, _dtls_handshake(1, body[:3], message_seq=1, fragment_offset=0, + fragment_length=3, total_length=len(body)), seq=2) + wire2 = _dtls_record(22, _dtls_handshake(1, body[3:], message_seq=1, fragment_offset=3, + fragment_length=len(body) - 3, total_length=len(body)), seq=3) + + engine = DTLSReassembly() + flow = ('20.0.0.1', 1, '20.0.0.2', 5684) + + DTLS(wire1, len(wire1), reassembly=engine, flow=flow, frame_num=1, frame_timestamp=0.0) + parsed2 = DTLS(wire2, len(wire2), reassembly=engine, flow=flow, frame_num=2, frame_timestamp=0.1) + + self.assertIsNotNone(engine.last) + self.assertTrue(engine.last.completed) + + msg2 = parsed2.info.content[0] + # decode was attempted (the fragment is complete and in-bounds) and + # failed -- raw reassembled bytes kept, not raised. + self.assertIsInstance(msg2.body, bytes) + self.assertEqual(msg2.body, body) + + def test_flow_required_check_fires_for_non_handshake_records_too(self) -> None: + """``flow`` is required the moment ``reassembly`` is given, + regardless of this record's content type, not only for a + cleartext ``handshake`` record.""" + from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + from pcapkit.protocols.application.dtls import DTLS + from pcapkit.utilities.exceptions import ProtocolError + + engine = DTLSReassembly() + + # an ``alert`` record -- never reaches ``_reassemble_dtls_handshake`` + # at all, so a handshake-only check would never see it. + alert_wire = _dtls_record(21, bytes([1, 0]), seq=2) + with self.assertRaises(ProtocolError): + DTLS(alert_wire, len(alert_wire), reassembly=engine, frame_num=1, frame_timestamp=0.0) + + # an encrypted (epoch >= 1) ``handshake`` record -- ``content`` is + # raw ciphertext :obj:`bytes` here, not a :obj:`tuple`, so a check + # keyed on the tuple would skip it too. + encrypted_wire = _dtls_record(22, b'\xaa' * 16, epoch=1, seq=3) + with self.assertRaises(ProtocolError): + DTLS(encrypted_wire, len(encrypted_wire), reassembly=engine, frame_num=2, + frame_timestamp=0.0) + + def test_without_reassembly_argument_body_stays_raw(self) -> None: + """With no engine, the body stays raw whatever the fragment.""" + from pcapkit.protocols.application.dtls import DTLS + + body = _client_hello_body() + half = len(body) // 2 + wire = _dtls_record(22, _dtls_handshake(1, body[half:], message_seq=5, fragment_offset=half, + fragment_length=len(body) - half, total_length=len(body)), seq=2) + + parsed = DTLS(wire, len(wire)) + msg = parsed.info.content[0] + self.assertTrue(msg.is_fragmented) + self.assertIsInstance(msg.body, bytes) + + def test_incomplete_reassembly_through_read_stays_raw_and_does_not_crash(self) -> None: + from pcapkit.foundation.reassembly.dtls import DTLS as DTLSReassembly + from pcapkit.protocols.application.dtls import DTLS + + body = _server_hello_body() + third = len(body) // 3 + frags = _split(body, (third, 2 * third)) + flow = ('3.3.3.3', 200, '4.4.4.4', 5684) + + engine = DTLSReassembly() + for num, idx in enumerate((0, 2), 1): # fragment 1 of 3 never arrives + fo, length = frags[idx] + wire = _dtls_record(22, _dtls_handshake(2, body[fo:fo + length], message_seq=1, + fragment_offset=fo, fragment_length=length, total_length=len(body)), + seq=20 + num) + parsed = DTLS(wire, len(wire), reassembly=engine, flow=flow, + frame_num=num, frame_timestamp=float(num)) + msg = parsed.info.content[0] + self.assertTrue(msg.is_fragmented) + self.assertIsInstance(msg.body, bytes) + + datagram, = engine.fetch() + self.assertEqual(datagram.completed.name, 'PARTIAL') + + +if __name__ == '__main__': + unittest.main()