refactor(analysis): give IP and TCP reassembly their own foundation.analysis modules

docs/source/contributing/conventions/reassembly-integration.rst

@@ -61,17 +61,29 @@ engine backend, by defining the extraction functions, by defining them raising
 engine level, as :mod:`pcapkit.foundation.engines.pyshark` does. DTLS defines them for
 the default engine and is disabled at the engine level on every other.
 
-**Point 3 holds for DTLS alone.**
-:func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` parses a reassembled DTLS
-handshake message (:issue:`2496`). IP and TCP have no such method: what parses their
-outcome is the general-purpose
-:meth:`Protocol.analyze <pcapkit.protocols.protocol.Protocol.analyze>` dispatcher,
-not a method written for reassembly specifically.
+**Point 3 holds for every engine.**
+:func:`~pcapkit.foundation.analysis.ip.ip_datagram` parses a reassembled IPv4 or IPv6
+datagram's payload by its payload protocol type, through
+:meth:`Protocol.analyze <pcapkit.protocols.protocol.Protocol.analyze>`;
+:func:`~pcapkit.foundation.analysis.tcp.tcp_datagram` parses a reassembled TCP
+datagram's by its ports, through
+:meth:`Transport.analyze <pcapkit.protocols.transport.transport.Transport.analyze>`;
+and :func:`~pcapkit.foundation.analysis.dtls.dtls_handshake` decodes a reassembled
+DTLS handshake message (:issue:`2496`). The owner chose on :issue:`2581` to give IP
+and TCP modules of their own, each function named for the datagram its engine
+reassembles, rather than leave their outcomes to those dispatchers with no analysis
+method, which kept the rule true for DTLS alone.
 
 **Where the rule is silent, DTLS takes the IP/TCP shape** (:issue:`2531`). IPv4, IPv6,
 TCP and DTLS reassembly each set ``__protocol_type__``, and each reports its parsed
 outcome as a datagram's ``packet``, deferred to first read by
-:class:`~pcapkit.foundation.reassembly.data.data.Deferred`. IP and TCP run
-``__protocol_type__``'s ``analyze``. DTLS runs the decoder behind ``dtls_handshake``
-instead, where point 3 places it, so its ``packet`` is what ``dtls_handshake`` returns:
-:data:`None` for an incomplete message.
+:class:`~pcapkit.foundation.reassembly.data.data.Deferred`. Each ``Deferred`` runs the
+parse behind its engine's analysis function, so a ``packet`` is what that function
+returns. IP and TCP bind that parse to ``__protocol_type__`` and call its ``analyze``,
+so a subclass naming another protocol has its ``packet`` parsed by that one
+(:issue:`2581`); DTLS decodes with
+:meth:`DTLS.read_handshake <pcapkit.protocols.application.dtls.DTLS.read_handshake>`
+whatever ``__protocol_type__`` names. The engines differ in what an incomplete outcome
+holds. IP and TCP parse a contiguous payload whether or not it is complete, as
+``strict=False`` reports one, and hold :data:`None` only for the received runs
+``strict=True`` reports; DTLS holds :data:`None` for any incomplete message.

docs/source/pcapkit/foundation/analysis/index.rst

@@ -14,4 +14,6 @@ so that a caller does not interpret the raw reassembled octets itself
 .. toctree::
    :maxdepth: 2
 
+   ip
+   tcp
    dtls

docs/source/pcapkit/foundation/analysis/ip.rst

@@ -0,0 +1,13 @@
+====================
+IP Datagram Analysis
+====================
+
+.. module:: pcapkit.foundation.analysis.ip
+
+:mod:`pcapkit.foundation.analysis.ip` contains
+:func:`~pcapkit.foundation.analysis.ip.ip_datagram` only, which parses the
+payload of an IPv4 or IPv6 datagram that
+:class:`~pcapkit.foundation.reassembly.ipv4.IPv4` or
+:class:`~pcapkit.foundation.reassembly.ipv6.IPv6` reassembled.
+
+.. autofunction:: pcapkit.foundation.analysis.ip.ip_datagram

docs/source/pcapkit/foundation/analysis/tcp.rst

@@ -0,0 +1,12 @@
+=====================
+TCP Datagram Analysis
+=====================
+
+.. module:: pcapkit.foundation.analysis.tcp
+
+:mod:`pcapkit.foundation.analysis.tcp` contains
+:func:`~pcapkit.foundation.analysis.tcp.tcp_datagram` only, which parses the
+application-layer payload of a TCP datagram that
+:class:`~pcapkit.foundation.reassembly.tcp.TCP` reassembled.
+
+.. autofunction:: pcapkit.foundation.analysis.tcp.tcp_datagram

docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst

@@ -95,9 +95,11 @@ Terminology
 
        .. note::
 
-          ``packet`` is analysed on the first read, not when the datagram is
-          submitted. A datagram is submitted for *every* frame whose **DF** flag
-          is clear -- an unfragmented one included -- and the analysis
+          ``packet`` is what :func:`~pcapkit.foundation.analysis.ip.ip_datagram`
+          parses ``payload`` to, unless ``__protocol_type__`` names another
+          protocol, and it is analysed on the first read, not when the
+          datagram is submitted. A datagram is submitted for *every* frame
+          whose **DF** flag is clear -- an unfragmented one included -- and the analysis
           is a second full parse of the payload, so running it eagerly would
           charge every caller for a result most never read. Reading the attribute, or any
           mapping view of it (``datagram['packet']``, ``to_dict()``, ``items()``,

docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst

@@ -140,10 +140,12 @@ Terminology
 
        .. note::
 
-          ``packet`` is analysed on the first read, not when the datagram is
-          submitted. The analysis is a second full parse of the payload, so
-          running it eagerly would charge every caller for a result most never
-          read. Reading the attribute, or any mapping view of it (``datagram['packet']``, ``to_dict()``, ``items()``,
+          ``packet`` is what :func:`~pcapkit.foundation.analysis.ip.ip_datagram`
+          parses ``payload`` to, unless ``__protocol_type__`` names another
+          protocol, and it is analysed on the first read, not when the
+          datagram is submitted. The analysis is a second full parse of the
+          payload, so running it eagerly would charge every caller for a result
+          most never read. Reading the attribute, or any mapping view of it (``datagram['packet']``, ``to_dict()``, ``items()``,
           ``repr()``), runs it and keeps the result; see
           :class:`~pcapkit.foundation.reassembly.data.data.Deferred`.
 

docs/source/pcapkit/foundation/reassembly/tcp.rst

@@ -259,6 +259,10 @@ Terminology
            |     |                  |--> ...
            |--> (Info) data ...
 
+       ``packet`` is what :func:`~pcapkit.foundation.analysis.tcp.tcp_datagram`
+       parses ``payload`` to, on its first read, unless ``__protocol_type__``
+       names another protocol.
+
        ``completed`` and ``conflict`` are independent signals: a datagram can
        be :attr:`~pcapkit.foundation.reassembly.data.data.Completion.COMPLETE`
        and still carry a non-empty ``conflict`` -- the resolution of a

pcapkit/all.py

@@ -85,6 +85,8 @@
     'DTLS_Reassembly',                                      # DTLS Reassembly
 
     # pcapkit.foundation.analysis
+    'ip_datagram',                                          # IP Analysis
+    'tcp_datagram',                                         # TCP Analysis
     'dtls_handshake',                                       # DTLS Analysis
 
     # pcapkit.foundation.registry

pcapkit/foundation/__init__.py

@@ -25,7 +25,7 @@
     'IPv4_Reassembly', 'IPv6_Reassembly', 'TCP_Reassembly', 'DTLS_Reassembly',
     'ReassemblyManager',
 
-    'dtls_handshake',
+    'ip_datagram', 'tcp_datagram', 'dtls_handshake',
 
     'TCP_TraceFlow',
     'TraceFlowManager',

pcapkit/foundation/analysis/__init__.py

@@ -15,7 +15,13 @@
 """
 # Analysis of DTLS handshake messages
 from pcapkit.foundation.analysis.dtls import dtls_handshake
+# Analysis of IP datagrams
+from pcapkit.foundation.analysis.ip import ip_datagram
+# Analysis of TCP datagrams
+from pcapkit.foundation.analysis.tcp import tcp_datagram
 
 __all__ = [
+    'ip_datagram',                          # IP Analysis
+    'tcp_datagram',                         # TCP Analysis
     'dtls_handshake',                       # DTLS Analysis
 ]

pcapkit/foundation/analysis/ip.py

@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+"""IP Datagram Analysis
+========================
+
+.. module:: pcapkit.foundation.analysis.ip
+
+:mod:`pcapkit.foundation.analysis.ip` contains
+:func:`~pcapkit.foundation.analysis.ip.ip_datagram` only, which parses the
+payload of an IPv4 or IPv6 datagram that
+:class:`~pcapkit.foundation.reassembly.ipv4.IPv4` or
+:class:`~pcapkit.foundation.reassembly.ipv6.IPv6` reassembled.
+
+.. versionadded:: 1.6.0
+
+"""
+from ipaddress import IPv6Address
+from typing import TYPE_CHECKING
+
+from pcapkit.protocols.internet.ipv4 import IPv4
+from pcapkit.protocols.internet.ipv6 import IPv6
+
+if TYPE_CHECKING:
+    from typing import Optional, Type
+
+    from pcapkit.const.reg.transtype import TransType
+    from pcapkit.foundation.reassembly.data.ip import Datagram as IP_Datagram
+    from pcapkit.protocols.internet.ip import IP as IP_Protocol
+    from pcapkit.protocols.protocol import ProtocolBase
+
+__all__ = ['ip_datagram']
+
+
+def ip_datagram(datagram: 'IP_Datagram') -> 'Optional[ProtocolBase]':
+    """Parse the payload of an IP datagram reassembled by
+    :class:`~pcapkit.foundation.reassembly.ipv4.IPv4` or
+    :class:`~pcapkit.foundation.reassembly.ipv6.IPv6`.
+
+    The payload is dispatched on the datagram's payload protocol type -- the
+    **Protocol** field for IPv4, the Fragment header's **Next Header** for
+    IPv6 -- by :meth:`Protocol.analyze
+    <pcapkit.protocols.protocol.Protocol.analyze>` of
+    :class:`~pcapkit.protocols.internet.ipv4.IPv4` or
+    :class:`~pcapkit.protocols.internet.ipv6.IPv6`, whichever the datagram's
+    addresses belong to.
+
+    Args:
+        datagram: Reassembled datagram (:term:`reasm.ipv4.datagram` or
+            :term:`reasm.ipv6.datagram`).
+
+    Returns:
+        The parsed payload. A payload reported as one contiguous run is parsed
+        whether or not ``datagram`` is complete, as ``strict=False`` reports an
+        incomplete one. :data:`None` if the payload is the tuple of runs
+        received, as ``strict=True`` reports an incomplete one.
+
+    Note:
+        :attr:`Datagram.packet <pcapkit.foundation.reassembly.data.ip.Datagram.packet>`
+        holds the same result, parsed on its first read, unless the
+        reassembly's ``__protocol_type__`` names another protocol, which then
+        parses it instead.
+
+    """
+    if isinstance(datagram.payload, tuple):
+        return None
+    protocol = IPv6 if isinstance(datagram.id.src, IPv6Address) else IPv4
+    return _analyze_payload(protocol, datagram.id.proto, datagram.payload)
+
+
+def _analyze_payload(protocol: 'Type[IP_Protocol]', proto: 'TransType',
+                     payload: 'bytes') -> 'ProtocolBase':
+    """Parse a reassembled payload by its payload protocol type.
+
+    :func:`ip_datagram` runs this, and so does the
+    :class:`~pcapkit.foundation.reassembly.data.data.Deferred` that
+    :meth:`IP.submit <pcapkit.foundation.reassembly.ip.IP.submit>` stores as a
+    datagram's ``packet``, with the reassembly's own protocol bound as
+    ``protocol``.
+
+    Args:
+        protocol: The IP protocol whose ``analyze`` parses ``payload``.
+        proto: The payload protocol type.
+        payload: The reassembled payload.
+
+    Returns:
+        The parsed payload.
+
+    """
+    return protocol.analyze(proto, payload)

pcapkit/foundation/analysis/tcp.py

@@ -0,0 +1,87 @@
+# -*- coding: utf-8 -*-
+"""TCP Datagram Analysis
+=========================
+
+.. module:: pcapkit.foundation.analysis.tcp
+
+:mod:`pcapkit.foundation.analysis.tcp` contains
+:func:`~pcapkit.foundation.analysis.tcp.tcp_datagram` only, which parses the
+application-layer payload of a TCP datagram that
+:class:`~pcapkit.foundation.reassembly.tcp.TCP` reassembled.
+
+.. versionadded:: 1.6.0
+
+"""
+from typing import TYPE_CHECKING
+
+from pcapkit.const.reg.apptype import TransportProtocol as Enum_TransportProtocol
+from pcapkit.protocols.transport.tcp import TCP
+
+if TYPE_CHECKING:
+    from typing import Optional, Type
+
+    from pcapkit.const.reg.apptype import TCP as AppType_TCP
+    from pcapkit.foundation.reassembly.data.tcp import Datagram as TCP_Datagram
+    from pcapkit.protocols.protocol import ProtocolBase
+
+__all__ = ['tcp_datagram']
+
+
+def tcp_datagram(datagram: 'TCP_Datagram') -> 'Optional[ProtocolBase]':
+    """Parse the payload of a TCP datagram reassembled by
+    :class:`~pcapkit.foundation.reassembly.tcp.TCP`.
+
+    A reassembled datagram carries no protocol number, so its payload is
+    dispatched on its ``(source, destination)`` ports by
+    :meth:`Transport.analyze <pcapkit.protocols.transport.transport.Transport.analyze>`
+    of :class:`~pcapkit.protocols.transport.tcp.TCP`. Each port is resolved to
+    its :class:`~pcapkit.const.reg.apptype.tcp.TCP` member by
+    :meth:`Transport._make_port
+    <pcapkit.protocols.transport.transport.Transport._make_port>`, as
+    :meth:`TCP._make_ports <pcapkit.foundation.reassembly.tcp.TCP._make_ports>`
+    resolves them for the reassembly.
+
+    Args:
+        datagram: Reassembled datagram (:term:`reasm.tcp.datagram`).
+
+    Returns:
+        The parsed payload. A payload reported as one contiguous run is parsed
+        whether or not ``datagram`` is complete, as ``strict=False`` reports an
+        incomplete one. :data:`None` if the payload is the tuple of runs
+        received, as ``strict=True`` reports an incomplete one.
+
+    Note:
+        :attr:`Datagram.packet <pcapkit.foundation.reassembly.data.tcp.Datagram.packet>`
+        holds the same result, parsed on its first read, unless the
+        reassembly's ``__protocol_type__`` names another protocol, which then
+        parses it instead.
+
+    """
+    if isinstance(datagram.payload, tuple):
+        return None
+    transport = Enum_TransportProtocol.tcp
+    ports = (TCP._make_port(datagram.id.src[1], transport),  # pylint: disable=protected-access
+             TCP._make_port(datagram.id.dst[1], transport))  # pylint: disable=protected-access
+    return _analyze_payload(TCP, ports, datagram.payload)
+
+
+def _analyze_payload(protocol: 'Type[TCP]', ports: 'tuple[AppType_TCP, AppType_TCP]',
+                     payload: 'bytes') -> 'ProtocolBase':
+    """Parse a reassembled payload by its datagram's ports.
+
+    :func:`tcp_datagram` runs this, and so does the
+    :class:`~pcapkit.foundation.reassembly.data.data.Deferred` that
+    :meth:`TCP.submit <pcapkit.foundation.reassembly.tcp.TCP.submit>` stores as
+    a datagram's ``packet``, with the reassembly's own protocol bound as
+    ``protocol``.
+
+    Args:
+        protocol: The TCP protocol whose ``analyze`` parses ``payload``.
+        ports: The datagram's ``(source, destination)`` ports.
+        payload: The reassembled payload.
+
+    Returns:
+        The parsed payload.
+
+    """
+    return protocol.analyze(ports, payload)

pcapkit/foundation/reassembly/data/data.py

@@ -160,12 +160,15 @@ class Deferred(Generic[_KT]):
     * :meth:`IP.submit <pcapkit.foundation.reassembly.ip.IP.submit>` builds a
       ``Deferred[TransType]``. It passes the payload protocol type, a
       :class:`~pcapkit.const.reg.transtype.TransType` read from the reassembled
-      datagram's **Protocol** / **Next Header** field, to
+      datagram's **Protocol** / **Next Header** field, to the parse
+      :func:`~pcapkit.foundation.analysis.ip.ip_datagram` runs:
       :meth:`Protocol.analyze <pcapkit.protocols.protocol.Protocol.analyze>`,
       which looks the next layer up by protocol registry number.
     * :meth:`TCP.submit <pcapkit.foundation.reassembly.tcp.TCP.submit>` builds a
       ``Deferred[tuple[AppType_TCP, AppType_TCP]]``. It passes a
-      ``(source, destination)`` port pair, because :meth:`Transport.analyze
+      ``(source, destination)`` port pair to the parse
+      :func:`~pcapkit.foundation.analysis.tcp.tcp_datagram` runs, because
+      :meth:`Transport.analyze
       <pcapkit.protocols.transport.transport.Transport.analyze>` overrides that
       parameter to one: a reassembled transport-layer payload carries no
       protocol number, so its application protocol is inferred from the ports.
@@ -209,10 +212,10 @@ class Deferred(Generic[_KT]):
     either way.
 
     Args:
-        analyze: The analyser to call:
-            :meth:`Protocol.analyze <pcapkit.protocols.protocol.Protocol.analyze>`
-            bound to the reassembly object's protocol for IP and TCP, the
-            handshake decoder for DTLS.
+        analyze: The analyser to call: the parse
+            :mod:`pcapkit.foundation.analysis` runs for the engine, bound to
+            the reassembly object's protocol for IP and TCP, whose ``analyze``
+            it calls, and the handshake decoder for DTLS.
         proto: What the analyser dispatches the payload on -- the payload
             protocol type for IP, the port pair for TCP, the handshake header
             for DTLS, as above.
@@ -224,7 +227,10 @@ class Deferred(Generic[_KT]):
 
        Also constrained to the DTLS handshake header, and the analyser may
        return a :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`
-       (:issue:`2531`).
+       (:issue:`2531`). IP's and TCP's analyser is the parse their
+       :mod:`pcapkit.foundation.analysis` module runs, bound to the
+       reassembly's protocol, rather than that protocol's bound ``analyze``
+       (:issue:`2581`).
 
     """
 

pcapkit/foundation/reassembly/data/ip.py

@@ -113,9 +113,12 @@ class Datagram(DeferredPacket, Info, Generic[_AT]):
     header: 'bytes'
     #: Reassembled IP payload.
     payload: 'bytes | tuple[bytes, ...]'
-    #: Parsed IP payload. Analysed on first read, not at construction time; a
-    #: :class:`~pcapkit.foundation.reassembly.data.data.Deferred` may be passed in its
-    #: place, and reading this attribute then runs it and keeps the result.
+    #: Parsed IP payload, as :func:`~pcapkit.foundation.analysis.ip.ip_datagram`
+    #: returns it unless the reassembly's ``__protocol_type__`` names another
+    #: protocol, which then parses it. Analysed on first read, not at
+    #: construction time; a :class:`~pcapkit.foundation.reassembly.data.data.Deferred`
+    #: may be passed in its place, and reading this attribute then runs it and
+    #: keeps the result.
     packet: 'Optional[ProtocolBase]'
     #: Octet ranges, absolute into the reassembled payload and both
     #: **inclusive** -- the same convention as :attr:`Packet.fo` combined with