refactor(foundation): route DTLS reassembly via toolkit and analysis (#2496)

docs/source/contributing/conventions/reassembly-integration.rst

@@ -4,19 +4,25 @@ Reassembly Integration
 -----------------------
 
 Where a new reassembly engine under :mod:`pcapkit.foundation.reassembly` plugs into
-the rest of the package. The owner's ruling, on this session's DTLS reassembly work.
+the rest of the package. The owner's ruling, given on the DTLS reassembly work of
+:issue:`2440`.
 
 The Rule
 ~~~~~~~~
 
 #. **A protocol class does not handle or integrate reassembly itself.** The protocol
    dissector stays a parser; it does not reach into
-   :mod:`pcapkit.foundation.reassembly` directly.
+   :mod:`pcapkit.foundation.reassembly` directly. A public method that parses a
+   reassembled unit from its own octets is still a parser, so a protocol class may
+   offer one (:issue:`2496`).
 #. **A new engine needs a new** :mod:`pcapkit.toolkit` **helper** that declares which
    output engines (|dpkt|_, |pyshark|_, ``scapy``, the default) support it.
 #. **A new engine's reassembled outcome needs a new**
    :mod:`pcapkit.foundation.analysis` **method** to parse that outcome into a usable
-   result, rather than leaving the caller to interpret raw reassembled bytes.
+   result, rather than leaving the caller to interpret raw reassembled bytes. The
+   package mirrors :mod:`pcapkit.foundation.reassembly`: a module per engine, whose
+   public names the package, :mod:`pcapkit.foundation` and :mod:`pcapkit.all`
+   export (:issue:`2496`).
 #. **Reassembly always follows the RFC's own definition**, never an algorithm this
    package invents. Where the RFC states no required algorithm -- buffer the whole
    message, accept overlaps -- the engine does the same and adds no timeout, cap or
@@ -25,33 +31,35 @@ The Rule
 Compliance, Measured
 ~~~~~~~~~~~~~~~~~~~~~
 
-**Point 1 is an existing, enforced precedent.** Neither
-:class:`~pcapkit.protocols.internet.ipv4.IPv4` nor
-:class:`~pcapkit.protocols.internet.ipv6.IPv6` nor
-:class:`~pcapkit.protocols.transport.tcp.TCP` imports or calls into
+**Point 1 holds for every engine.** None of
+:class:`~pcapkit.protocols.internet.ipv4.IPv4`,
+:class:`~pcapkit.protocols.internet.ipv6.IPv6`,
+:class:`~pcapkit.protocols.transport.tcp.TCP` and
+:class:`~pcapkit.protocols.application.dtls.DTLS` imports or calls into
 :mod:`pcapkit.foundation.reassembly` at all. The real path is
 :mod:`pcapkit.foundation.engines` calling a ``pcapkit.toolkit.<engine>`` extraction
 function on an already-parsed frame, feeding the result to the reassembly instance the
-:class:`~pcapkit.foundation.extraction.Extractor` holds.
-
-**Points 2 and 3 describe infrastructure that does not exist yet, for any engine.**
-No single :mod:`pcapkit.toolkit` helper enumerates which output engines support a given
-reassembly engine -- support is established ad hoc, per engine backend, in three
-different shapes (define the extraction functions; define them raising
-:exc:`~pcapkit.utilities.exceptions.UnsupportedCall`; or disable reassembly at the
-engine level, as :mod:`pcapkit.foundation.engines.pyshark` does). No
-:mod:`pcapkit.foundation.analysis` module exists; what parses a reassembled outcome
-today is the pre-existing, general-purpose
+:class:`~pcapkit.foundation.extraction.Extractor` holds; for DTLS that function is
+:func:`pcapkit.toolkit.pcap.dtls_reassembly` (:issue:`2496`).
+:meth:`DTLS.read_handshake <pcapkit.protocols.application.dtls.DTLS.read_handshake>`
+is the parser method the rule allows.
+
+**No engine meets point 2.** No single :mod:`pcapkit.toolkit` helper enumerates which
+output engines support a given reassembly engine. Support is established ad hoc, per
+engine backend, by defining the extraction functions, by defining them raising
+:exc:`~pcapkit.utilities.exceptions.UnsupportedCall`, or by disabling reassembly at the
+engine level, as :mod:`pcapkit.foundation.engines.pyshark` does. DTLS defines them for
+the default engine and is disabled at the engine level on every other.
+
+**Point 3 holds for DTLS alone.**
+:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` parses a reassembled DTLS
+handshake message (:issue:`2496`). IP and TCP have no such method: what parses their
+outcome is the general-purpose
 :meth:`Protocol.analyze <pcapkit.protocols.protocol.Protocol.analyze>` dispatcher,
 reused via :class:`~pcapkit.foundation.reassembly.data.data.Deferred`, not a method
-written for reassembly specifically. So points 2 and 3 are a forward-looking ask, not
-yet a pattern any engine follows.
-
-**The DTLS reassembly work on** :pr:`2451` **violates point 1.**
-:class:`~pcapkit.protocols.application.dtls.DTLS` imports
-:class:`~pcapkit.foundation.reassembly.dtls.DTLS` directly and calls it from
-:meth:`DTLS.read`, which no other protocol class does. Its own module docstring notes
-the further divergence from the IP/TCP shape: it is "not toolkit-fed", reports its
-reassembled body eagerly instead of through a :class:`Deferred
-<pcapkit.foundation.reassembly.data.data.Deferred>`, and sets no
-``__protocol_type__``.
+written for reassembly specifically.
+
+DTLS differs from the IP/TCP shape where the rule is silent: its reassembled body is
+reported eagerly, with no deferred ``packet``, and
+:class:`~pcapkit.foundation.reassembly.dtls.DTLS` sets no ``__protocol_type__``
+(:issue:`2531`).

docs/source/contributing/pep.rst

@@ -247,10 +247,12 @@ without keys.
 
 **Done: reassembling a handshake message DTLS has fragmented across several
 records** (:rfc:`6347#section-4.2.3`, :issue:`2440`), via
-:class:`~pcapkit.foundation.reassembly.dtls.DTLS`, keyed on the UDP 4-tuple.
-Pass ``reassembly=`` and ``flow=`` to
-:class:`~pcapkit.protocols.application.dtls.DTLS` to opt in; the completing
-fragment's body decodes to the full reassembled message, while every other
+:class:`~pcapkit.foundation.reassembly.dtls.DTLS`, keyed on the UDP 4-tuple
+and the epoch. ``Extractor(reassembly=True, dtls=True)`` opts in on the
+default engine, which feeds it through
+:func:`pcapkit.toolkit.pcap.dtls_reassembly`, and
+:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes each
+reassembled message, of any length (:issue:`2496`). In the frame itself every
 fragment's body stays that one fragment's raw octets -- see
 :attr:`DTLSHandshake.is_fragmented <pcapkit.protocols.data.application.dtls.DTLSHandshake.is_fragmented>`.
 An unfragmented message is unaffected either way.

docs/source/pcapkit/foundation/analysis/dtls.rst

@@ -0,0 +1,12 @@
+===============================
+DTLS Handshake-Message Analysis
+===============================
+
+.. module:: pcapkit.foundation.analysis.dtls
+
+:mod:`pcapkit.foundation.analysis.dtls` contains
+:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` only, which decodes a
+DTLS handshake message that :class:`~pcapkit.foundation.reassembly.dtls.DTLS`
+reassembled (:rfc:`6347#section-4.2.3`).
+
+.. autofunction:: pcapkit.foundation.analysis.dtls.dtls_handshake

docs/source/pcapkit/foundation/analysis/index.rst

@@ -0,0 +1,17 @@
+============================
+Reassembled Payload Analysis
+============================
+
+.. module:: pcapkit.foundation.analysis
+
+:mod:`pcapkit.foundation.analysis` parses what a
+:mod:`pcapkit.foundation.reassembly` engine reassembled into a usable result,
+so that a caller does not interpret the raw reassembled octets itself
+(:ref:`reassembly-integration`).
+
+.. versionadded:: 1.6.0
+
+.. toctree::
+   :maxdepth: 2
+
+   dtls

docs/source/pcapkit/foundation/engines/index.rst

@@ -115,6 +115,11 @@ their row says otherwise:
 |                                                                 | Scapy binds it to on the host it runs on, and no TCP or       |
 |                                                                 | Fragment header behind an AH, Mobility, HIP or Shim6 header   |
 +-----------------------------------------------------------------+---------------------------------------------------------------+
+| :class:`~pcapkit.foundation.engines.dpkt.DPKT`,                 | no DTLS reassembly, since                                     |
+| :class:`~pcapkit.foundation.engines.scapy.Scapy` and            | :func:`~pcapkit.toolkit.pcap.dtls_reassembly` reads           |
+| :class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile`      | :mod:`pcapkit`'s own DTLS layer -- disabled with an           |
+|                                                                 | :class:`~pcapkit.utilities.warnings.AttributeWarning`         |
++-----------------------------------------------------------------+---------------------------------------------------------------+
 | :class:`~pcapkit.foundation.engines.dpkt.DPKT`,                 | no PCAP output, since each frame reaches the writer as a      |
 | :class:`~pcapkit.foundation.engines.scapy.Scapy`,               | plain :obj:`dict` -- ``json`` is written instead, with a      |
 | :class:`~pcapkit.foundation.engines.pyshark.PyShark`,           | :class:`~pcapkit.utilities.warnings.FormatWarning`            |

docs/source/pcapkit/foundation/index.rst

@@ -6,8 +6,9 @@ Library Foundation
 :mod:`pcapkit.foundation` is a collection of foundations for
 :mod:`pcapkit`, including PCAP file extraction tool
 :class:`~pcapkit.foundation.extraction.Extractor` and the
-:mod:`~pcapkit.foundation.engines` it reads through, TCP/IP reassembly
-implementations, TCP flow tracer
+:mod:`~pcapkit.foundation.engines` it reads through, TCP/IP and DTLS
+reassembly implementations and the :mod:`~pcapkit.foundation.analysis` of what
+they reassemble, TCP flow tracer
 :class:`~pcapkit.foundation.traceflow.traceflow.TraceFlow`, and registry
 management APIs for :mod:`pcapkit`.
 
@@ -17,5 +18,6 @@ management APIs for :mod:`pcapkit`.
    extraction
    engines/index
    reassembly/index
+   analysis/index
    traceflow/index
    registry

docs/source/pcapkit/foundation/reassembly/dtls.rst

@@ -54,8 +54,9 @@ Terminology
        is :class:`~pcapkit.foundation.reassembly.data.dtls.Packet` -- one
        fragment's own ``bufid``, ``message_seq``, ``fragment_offset``,
        ``fragment_length`` and raw ``payload``, built by
-       :meth:`DTLS.read <pcapkit.protocols.application.dtls.DTLS.read>` from
-       a single :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`.
+       :func:`pcapkit.toolkit.pcap.dtls_reassembly` or
+       :func:`pcapkit.toolkit.pcapng.dtls_reassembly` from a single
+       :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`.
 
    reasm.dtls.datagram
        Data structure for **reassembled DTLS handshake message** (element of
@@ -64,7 +65,8 @@ Terminology
        ``completed``, the original packet identifier, the packet numbers
        that built it, and ``body``, the reassembled payload (a contiguous
        :obj:`bytes`, or -- under ``strict=True`` and an incomplete message --
-       a :obj:`tuple` of the octet runs actually received).
+       a :obj:`tuple` of the octet runs actually received), which
+       :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes.
 
    reasm.dtls.buffer
        Data structure for internal buffering

docs/source/pcapkit/protocols/application/dtls.rst

@@ -49,11 +49,13 @@ the two formats are told apart by that octet alone.
 Known Limitations
 -----------------
 
-**A handshake message split across DTLS records is not reassembled**
-(:rfc:`6347#section-4.2.3`, :issue:`2440`). Such a message is flagged
+**A handshake message split across DTLS records is not decoded here**
+(:rfc:`6347#section-4.2.3`, :issue:`2440`). Each fragment is flagged
 :attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.is_fragmented`
 and its :attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.body`
-kept as raw octets.
+kept as raw octets. ``Extractor(reassembly=True, dtls=True)`` reassembles the
+message, and :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes it
+with :meth:`~pcapkit.protocols.application.dtls.DTLS.read_handshake`.
 
 **A unified-header record with a connection ID is not decoded**
 (:rfc:`9147#section-4.1`, :issue:`2467`). Its length is negotiated, not carried
@@ -83,6 +85,7 @@ ones on the wire.
    .. automethod:: read
    .. automethod:: make
    .. automethod:: unpack
+   .. automethod:: read_handshake
 
    .. automethod:: __length_hint__
 

docs/source/pcapkit/toolkit/builtin.rst

@@ -12,14 +12,17 @@ The reassembly and flow tracing adapters return the data their
 :mod:`pcapkit.foundation` counterpart consumes, or :data:`None` if the frame
 cannot be used for it. The TCP adapters here and in
 :mod:`pcapkit.toolkit.pcapng` read the segment with
-:func:`~pcapkit.toolkit.pcap.tcp_segment`.
+:func:`~pcapkit.toolkit.pcap.tcp_segment`, and the DTLS ones the record with
+:func:`~pcapkit.toolkit.pcap._dtls_fragments`.
 
 .. autofunction:: pcapkit.toolkit.pcap.ipv4_reassembly
 
 .. autofunction:: pcapkit.toolkit.pcap.ipv6_reassembly
 
 .. autofunction:: pcapkit.toolkit.pcap.tcp_reassembly
 
+.. autofunction:: pcapkit.toolkit.pcap.dtls_reassembly
+
 .. autofunction:: pcapkit.toolkit.pcap.tcp_traceflow
 
 Auxiliary Functions
@@ -29,6 +32,8 @@ Auxiliary Functions
 
 .. autofunction:: pcapkit.toolkit.pcap._fragment
 
+.. autofunction:: pcapkit.toolkit.pcap._dtls_fragments
+
 .. autoclass:: pcapkit.toolkit.pcap.TCPSegment
    :members:
    :show-inheritance:
@@ -49,6 +54,8 @@ cannot be used for it.
 
 .. autofunction:: pcapkit.toolkit.pcapng.tcp_reassembly
 
+.. autofunction:: pcapkit.toolkit.pcapng.dtls_reassembly
+
 .. autofunction:: pcapkit.toolkit.pcapng.tcp_traceflow
 
 Auxiliary Functions

pcapkit/all.py

@@ -84,6 +84,9 @@
     'TCP_Reassembly',                                       # TCP Reassembly
     'DTLS_Reassembly',                                      # DTLS Reassembly
 
+    # pcapkit.foundation.analysis
+    'dtls_handshake',                                       # DTLS Analysis
+
     # pcapkit.foundation.registry
     'register_protocol',
     'register_protocol_code',
@@ -137,8 +140,9 @@
     'Data', 'data',                                         # Protocol Data
 
     # pcapkit.toolkit
-    'ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'tcp_traceflow',
-    'pcapng_ipv4_reassembly', 'pcapng_ipv6_reassembly', 'pcapng_tcp_reassembly', 'pcapng_tcp_traceflow',
+    'ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'dtls_reassembly', 'tcp_traceflow',
+    'pcapng_ipv4_reassembly', 'pcapng_ipv6_reassembly', 'pcapng_tcp_reassembly', 'pcapng_dtls_reassembly',
+    'pcapng_tcp_traceflow',
     'pcapng_block2frame',                                   # default engine
     'dpkt_ipv6_hdr_len', 'dpkt_packet2chain', 'dpkt_packet2dict',
     'dpkt_ipv4_reassembly', 'dpkt_ipv6_reassembly', 'dpkt_tcp_reassembly', 'dpkt_tcp_traceflow',

pcapkit/foundation/__init__.py

@@ -9,9 +9,11 @@
 :mod:`pcapkit`, including PCAP file extraction tool
 :class:`~pcapkit.foundation.extraction.Extractor`, flow tracing
 :mod:`~pcapkit.foundation.traceflow`, registry management
-APIs for :mod:`pcapkit`, and TCP/IP reassembly implementations.
+APIs for :mod:`pcapkit`, TCP/IP and DTLS reassembly implementations, and
+the :mod:`~pcapkit.foundation.analysis` of what they reassemble.
 
 """
+from pcapkit.foundation.analysis import *
 from pcapkit.foundation.extraction import Extractor
 from pcapkit.foundation.reassembly import *
 from pcapkit.foundation.registry import *
@@ -23,6 +25,8 @@
     'IPv4_Reassembly', 'IPv6_Reassembly', 'TCP_Reassembly', 'DTLS_Reassembly',
     'ReassemblyManager',
 
+    'dtls_handshake',
+
     'TCP_TraceFlow',
     'TraceFlowManager',
 

pcapkit/foundation/analysis/__init__.py

@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+# pylint: disable=unused-import
+"""Reassembled Payload Analysis
+==================================
+
+.. module:: pcapkit.foundation.analysis
+
+:mod:`pcapkit.foundation.analysis` parses what a
+:mod:`pcapkit.foundation.reassembly` engine reassembled into a usable result,
+so that a caller does not interpret the raw reassembled octets itself
+(:ref:`reassembly-integration`).
+
+.. versionadded:: 1.6.0
+
+"""
+# Analysis of DTLS handshake messages
+from pcapkit.foundation.analysis.dtls import dtls_handshake
+
+__all__ = [
+    'dtls_handshake',                       # DTLS Analysis
+]

pcapkit/foundation/analysis/dtls.py

@@ -0,0 +1,67 @@
+# -*- coding: utf-8 -*-
+"""DTLS Handshake-Message Analysis
+=====================================
+
+.. module:: pcapkit.foundation.analysis.dtls
+
+:mod:`pcapkit.foundation.analysis.dtls` contains
+:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` only, which decodes a
+DTLS handshake message that :class:`~pcapkit.foundation.reassembly.dtls.DTLS`
+reassembled (:rfc:`6347#section-4.2.3`).
+
+.. versionadded:: 1.6.0
+
+"""
+from typing import TYPE_CHECKING, cast
+
+from pcapkit.protocols.application.dtls import DTLS
+from pcapkit.protocols.data.application.dtls import DTLSHandshake as Data_DTLSHandshake
+from pcapkit.utilities.exceptions import ProtocolError
+
+if TYPE_CHECKING:
+    from typing import Optional
+
+    from pcapkit.foundation.reassembly.data.dtls import Datagram as DTLS_Datagram
+
+__all__ = ['dtls_handshake']
+
+
+def dtls_handshake(datagram: 'DTLS_Datagram') -> 'Optional[Data_DTLSHandshake]':
+    """Parse a DTLS handshake message reassembled by
+    :class:`~pcapkit.foundation.reassembly.dtls.DTLS`.
+
+    The message is read as if it had arrived unfragmented, ``fragment_offset``
+    0 and ``fragment_length`` its ``length`` (:rfc:`6347#section-4.2.3`), by
+    :meth:`DTLS.read_handshake
+    <pcapkit.protocols.application.dtls.DTLS.read_handshake>`. So a
+    ``ClientHello`` or ``ServerHello`` body decodes as an unfragmented one
+    does, whatever its length, and any other body stays raw octets.
+
+    Args:
+        datagram: Reassembled message (:term:`reasm.dtls.datagram`).
+
+    Returns:
+        The message, unfragmented. Its body stays the raw reassembled octets if
+        they do not decode as its ``msg_type``. :data:`None` if ``datagram`` is
+        incomplete, as its missing octets cannot be decoded.
+
+    """
+    if not datagram.completed:
+        return None
+    # NOTE: a complete message is always one contiguous run.
+    body = cast('bytes', datagram.body)
+    message = (bytes([int(datagram.msg_type)]) + datagram.length.to_bytes(3, 'big')
+               + datagram.message_seq.to_bytes(2, 'big') + bytes(3)  # fragment_offset 0
+               + datagram.length.to_bytes(3, 'big') + body)
+    try:
+        return DTLS.read_handshake(message)
+    except ProtocolError:
+        return Data_DTLSHandshake(
+            msg_type=datagram.msg_type,
+            length=datagram.length,
+            body=body,
+            message_seq=datagram.message_seq,
+            fragment_offset=0,
+            fragment_length=datagram.length,
+            is_fragmented=False,
+        )