docs(foundation): seventh whole-repo pass over foundation, utilities, interface, toolkit and dumpkit (#1103)

docs/source/pcapkit/dumpkit/index.rst

@@ -53,6 +53,7 @@ hierarchy:
 
    The ``tree`` output (also selected as ``text`` and ``txt``) is a lossy,
    human-readable view, and nothing reads it back. It renders ``''``, ``b''``,
-   :data:`None` and ``{}`` all as ``NIL``, and writes a newline inside a string
-   as is, breaking the layout. Use ``json`` or ``plist`` (also selected as
+   :data:`None` and ``{}`` all as ``NIL``, writes a newline inside a short
+   string as is, breaking the layout, and rewraps a longer one, turning its
+   newlines into spaces. Use ``json`` or ``plist`` (also selected as
    ``xml``) where the report has to round-trip.

docs/source/pcapkit/foundation/engines/3rdparty.rst

@@ -323,7 +323,7 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
    interface rather than 1.3.0.
 
    Every attribute this engine touches -- the ``pcap.pcap(name=..., promisc=...)``
-   constructor, ``pcap.pcap.datalink``, ``snaplen``, iteration yielding
+   constructor, ``pcap.pcap.datalink``, iteration yielding
    ``(timestamp, bytes)``, and ``pcap.pcap.close`` -- is present on both and
    behaves identically, byte for byte and timestamp for timestamp, on
    ``pcap-ct`` 1.3.0b3 against ``pypcap`` 1.3.0 -- a statement about those

docs/source/pcapkit/foundation/engines/index.rst

@@ -75,7 +75,8 @@ their row says otherwise:
 +-----------------------------------------------------------------+---------------------------------------------------------------+
 | Engine                                                          | Gap, and how it is surfaced                                   |
 +=================================================================+===============================================================+
-| :class:`~pcapkit.foundation.engines.pyshark.PyShark`            | no reassembly -- disabled with an                             |
+| :class:`~pcapkit.foundation.engines.pyshark.PyShark`            | no reassembly, and no ``trace_analyse``, since it reports     |
+|                                                                 | dissected fields rather than octets -- each disabled with an  |
 |                                                                 | :class:`~pcapkit.utilities.warnings.AttributeWarning`         |
 +-----------------------------------------------------------------+---------------------------------------------------------------+
 | :class:`~pcapkit.foundation.engines.pypcap.PyPCAP`              | no protocol dissection at all, hence no reassembly and no     |

docs/source/pcapkit/foundation/extraction.rst

@@ -33,6 +33,7 @@ and extracts parameters from a PCAP file.
    .. autoproperty:: reassembly
    .. autoproperty:: trace
    .. autoproperty:: engine
+   .. autoproperty:: magic_number
 
    .. automethod:: register_dumper
    .. automethod:: register_engine

docs/source/pcapkit/foundation/reassembly/ip/index.rst

@@ -52,20 +52,21 @@ Algorithm
       IF (FO = 0 AND MF = 0) {
          IF (buffer with BUFID is allocated) {
             flush all reassembly for this BUFID;
-            Submit datagram to next step;
-            DONE.
          }
+         Submit datagram to next step;
+         DONE.
       }
 
       IF (no buffer with BUFID is allocated) {
          allocate reassembly resources with BUFID;
          TIMER <- TLB;
          TDL <- 0;
-         put data from fragment into data buffer with BUFID
-            [from octet FO*8 to octet (TL-(IHL*4))+FO*8];
-         set RCVBT bits [from FO to FO+((TL-(IHL*4)+7)/8)];
       }
 
+      put data from fragment into data buffer with BUFID
+         [from octet FO*8 to octet (TL-(IHL*4))+FO*8];
+      set RCVBT bits [from FO to FO+((TL-(IHL*4)+7)/8)];
+
       IF (MF = 0) {
          TDL <- TL-(IHL*4)+(FO*8)
       }
@@ -100,6 +101,10 @@ Departures from the procedure above:
   timeout after its first-arriving fragment, on the capture's clock -- 60
   seconds by default, per :rfc:`1122#section-3.3.2` for IPv4 and
   :rfc:`8200#section-4.5` for IPv6.
+* A buffer flushed by an unfragmented datagram reusing its ``BUFID`` is
+  reported as an incomplete datagram rather than discarded.
+* ``TDL`` starts at ``-1`` rather than ``0``, so that a datagram carrying no
+  data can complete (:issue:`1359`).
 * Only the final fragment's partial last ``RCVBT`` block is set. A non-final
   fragment carries a multiple of 8 octets, so one ending mid-block was cut or
   is malformed, and setting its last block would complete the datagram with

docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst

@@ -174,10 +174,10 @@ Terminology
            |     |--> ipv6_frag.id   |
            |     |--> ipv6_frag.next |
            |                         |--> 'TDL' : (int) total data length
-           |                         |--> RCVBT : (bytearray) fragment received bit table
-           |                         |             |--> (bytes) b'\\x00' -> not received
-           |                         |             |--> (bytes) b'\\x01' -> received
-           |                         |             |--> (bytes) ...
+           |                         |--> 'RCVBT' : (bytearray) fragment received bit table
+           |                         |               |--> (bytes) b'\\x00' -> not received
+           |                         |               |--> (bytes) b'\\x01' -> received
+           |                         |               |--> (bytes) ...
            |                         |--> 'index' : (list) list of reassembled packets
            |                         |               |--> (int) packet range number
            |                         |--> 'header' : (bytes) header buffer

docs/source/pcapkit/foundation/reassembly/tcp.rst

@@ -349,9 +349,10 @@ Terminology
        its own ``isn`` -- ``raw[n]`` holds the octet with sequence number
        ``isn + n``, and ``isn`` is revised downwards whenever a segment turns
        up below the data already buffered, so it is not necessarily the
-       connection's own initial sequence number.
-       :meth:`TCP.submit <pcapkit.foundation.reassembly.tcp.TCP.submit>` is the
-       one place that converts between the two.
+       connection's own initial sequence number. Each ACK's ``gap`` and
+       ``conflict`` lists are kept in sequence numbers too;
+       :meth:`TCP.submit <pcapkit.foundation.reassembly.tcp.TCP.submit>` turns
+       the ``gap`` entries into offsets into ``raw`` when it reports the buffer.
 
 Data Models
 ===========

docs/source/pcapkit/foundation/traceflow/tcp.rst

@@ -40,7 +40,7 @@ Terminology
 
        .. code-block:: python
 
-          tract_dict = dict(
+          trace_dict = dict(
               protocol=data_link,                     # data link type from global header
               index=frame.info.number,                # frame number
               frame=frame.info,                       # extracted frame info
@@ -56,7 +56,8 @@ Terminology
               dst=ip.dst,                             # destination IP
               srcport=tcp.srcport,                    # TCP source port
               dstport=tcp.dstport,                    # TCP destination port
-              timestamp=frame.info.time_epoch,        # frame timestamp
+              timestamp=float(
+                  frame.info.time_epoch),             # frame timestamp
           )
 
        .. seealso:: :class:`pcapkit.foundation.traceflow.data.tcp.Packet`

docs/source/pcapkit/index.rst

@@ -137,24 +137,26 @@ Environment Variables
 
 .. envvar:: PCAPKIT_DEVMODE
 
-   If set to ``1``, :mod:`pcapkit` will run in development mode.
+   If set to ``1`` (or ``yes``, ``true`` or ``on``, in any case), :mod:`pcapkit`
+   will run in development mode.
 
    .. seealso::
 
       :data:`pcapkit.utilities.logging.DEVMODE`
 
 .. envvar:: PCAPKIT_VERBOSE
 
-   If set to ``1``, :mod:`pcapkit` will run with verbose output.
+   If set to ``1`` (or ``yes``, ``true`` or ``on``, in any case), :mod:`pcapkit`
+   will run with verbose output.
 
    .. seealso::
 
       :data:`pcapkit.utilities.logging.VERBOSE`
 
 .. envvar:: PCAPKIT_SPHINX
 
-   If set to ``1``, :mod:`pcapkit` will run with Sphinx additional
-   typing hints and docstrings support.
+   If set to ``1`` (or ``yes``, ``true`` or ``on``, in any case), :mod:`pcapkit`
+   will run with Sphinx additional typing hints and docstrings support.
 
    .. seealso::
 

docs/source/pcapkit/utilities/index.rst

@@ -62,8 +62,8 @@ following objects and functions:
 
 It also exports :mod:`pathlib`, :exc:`ModuleNotFoundError`,
 :class:`collections.abc.Collection` and :func:`functools.cached_property` as the
-standard library has them. Their fallbacks were for interpreters below Python 3.8,
-which 1.5.0 no longer installs on (:issue:`2051`).
+standard library has them, with no fallback, since 1.5.0 requires Python 3.8 or
+later (:issue:`2051`).
 
 .. function:: final(f)
 
@@ -100,5 +100,5 @@ which 1.5.0 no longer installs on (:issue:`2051`).
 
    .. versionchanged:: 1.5.0
 
-      No longer a backport below Python 3.8, which 1.5.0 no longer installs on
+      No longer a backport below Python 3.8, which 1.5.0 does not install on
       (:issue:`2051`).

docs/source/pcapkit/utilities/warnings.rst

@@ -29,9 +29,10 @@ channel**:
    category on the command line, noted below).
 
 The counts are the same in development mode as outside it:
-:envvar:`PCAPKIT_DEVMODE` and :envvar:`PCAPKIT_VERBOSE` change how much detail
-the log record carries, never how many records there are. Constructing a warning
-class is not an act of reporting one -- it emits nothing and has no side effects.
+:envvar:`PCAPKIT_DEVMODE` changes where the log records go, and
+:envvar:`PCAPKIT_VERBOSE` how much detail they carry, never how many records
+there are. Constructing a warning class is not an act of reporting one -- it
+emits nothing and has no side effects.
 
 A next layer captured shorter than its length hint is parsed on trial, and
 replaced with :class:`~pcapkit.protocols.misc.raw.Raw` when its header was not

pcapkit/__init__.py

@@ -30,7 +30,7 @@
 
 - Protocols (:mod:`pcapkit.protocols`)
 
-  Collection of all protocol family, with detailed
+  Collection of all protocol families, with detailed
   implementation and methods.
 
 - Utilities (:mod:`pcapkit.utilities`)

pcapkit/dumpkit/common.py

@@ -220,7 +220,7 @@ def render_enum(o: 'enum.Enum | aenum.Enum') -> 'str':
         plausible member name, so a consumer splitting the rendering on ``::``
         cannot tell it from a member genuinely so named -- and ``NONE`` *is* a
         declared name elsewhere in the library. And the defect is not confined to
-        zero: ``Flags(1)``, ``Flags(8)`` and ``Flags(65536)`` are every bit as
+        zero: ``Flags(1)``, ``Flags(8)`` and ``Flags(15)`` are every bit as
         nameless, so a guard written against ``value == 0`` would fix one case
         and leave the rest.
 

pcapkit/dumpkit/pcap.py

@@ -246,8 +246,8 @@ def _append_value(self, value: 'Data_Frame', file: 'IO[bytes]', name: 'str') ->
         # NOTE: The payload is read before the metadata so that a caller passing a
         # mapping rather than a dissected frame -- which the flow-tracing adapters
         # of several engines do -- still fails naming ``packet``.
-        # :mod:`pcapkit.foundation.extraction` substitutes a
-        # dict-capable trace format on the strength of that error.
+        # :mod:`pcapkit.foundation.extraction` cites that error when it
+        # substitutes a dict-capable trace format for those engines up front.
         packet = value.packet
         frame_info = value.frame_info
         ts_sec, ts_frac = self._make_timestamp(frame_info, value.time_epoch)

pcapkit/foundation/engines/engine.py

@@ -104,8 +104,9 @@ def name(self) -> 'str':
         """Engine name.
 
         Note:
-            This property is not available as a class
-            attribute.
+            This instance property is not what a class attribute lookup
+            reaches: on the class, the metaclass property of the same name
+            answers instead, with the same value.
 
         """
         if hasattr(self, '__engine_name__'):
@@ -117,8 +118,9 @@ def module(self) -> 'str':
         """Engine module name.
 
         Note:
-            This property is not available as a class
-            attribute.
+            This instance property is not what a class attribute lookup
+            reaches: on the class, the metaclass property of the same name
+            answers instead, with the same value.
 
         """
         if hasattr(self, '__engine_module__'):

pcapkit/foundation/extraction.py

@@ -835,11 +835,9 @@ def run(self) -> 'None':  # pylint: disable=inconsistent-return-statements
         * pcap-ct driver: :class:`pcapkit.foundation.engines.pcap_ct.PCAP_CT`
         * PyPCAPFile driver: :class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile`
 
-        Warns:
-            pcapkit.utilities.warnings.EngineWarning: If the extraction engine is not
-                available: its dependency is not installed, its
-                :meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason`
-                rules it out, or the supplied engine is unknown.
+        Note:
+            An unavailable engine is not reported here: :meth:`__init__` settles
+            the engine, and warns about it, before calling this method.
 
         Raises:
             FormatError: If the selected engine cannot read the input's format --
@@ -1263,9 +1261,9 @@ def __init__(self,
                 :class:`~pcapkit.corekit.context.ContextRegistry`, a single
                 :class:`~pcapkit.corekit.context.ProtocolContext`, a mapping,
                 or any iterable of contexts. The channel is honoured by the
-                ``default``, ``pcap`` and ``pcapng`` engines, which parse with
-                :mod:`pcapkit`'s own protocol implementations; the third party
-                engines ignore it.
+                :term:`default engine`, whose PCAP and PCAP-NG readers parse
+                with :mod:`pcapkit`'s own protocol implementations; the third
+                party engines ignore it.
 
         Warns:
             pcapkit.utilities.warnings.FormatWarning: Warns under the following circumstances:
@@ -1281,7 +1279,9 @@ def __init__(self,
                 engine first, which supports PCAP output.
 
             pcapkit.utilities.warnings.EngineWarning: If the requested extraction
-                engine is not available, cf. :meth:`run`.
+                engine is not available: its dependency is not installed, its
+                :meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason`
+                rules it out, or the supplied engine is unknown.
             pcapkit.utilities.warnings.AttributeWarning: If ``trace_analyse`` is
                 requested while the extraction engine is PyShark.
 

pcapkit/foundation/reassembly/data/data.py

@@ -51,7 +51,7 @@ class Completion(EnumLookup, StrEnum):
     :class:`pcapkit.const.pcapng.tls_key_label.TLSKeyLabel` is the same kind of
     string enumeration, but it is generated like its :mod:`pcapkit.const.pcapng`
     siblings (:issue:`886`), so it derives from |aenum|_'s own
-    ``StrEnum`` (via :class:`~pcapkit.corekit.enum.EnumRegistry`). The
+    ``StrEnum``, alongside :class:`~pcapkit.corekit.enum.EnumRegistry`. The
     properties above hold for it, as ``aenum.StrEnum`` is a :class:`str`
     subclass, but ``isinstance``/``issubclass`` against
     ``pcapkit.utilities.compat.StrEnum`` does not on Python 3.11 and later,

pcapkit/foundation/reassembly/data/ip.py

@@ -67,7 +67,7 @@ class DatagramID(Info, Generic[_AT]):
     src: '_AT'
     #: Destination address.
     dst: '_AT'
-    #: IP protocol identifier.
+    #: IP identification.
     id: 'int'
     #: Payload protocol type.
     proto: 'TransType'
@@ -150,10 +150,9 @@ class Datagram(DeferredPacket, Info, Generic[_AT]):
         # ``bytes`` payload and a parsed ``packet``, and an incomplete one with a
         # tuple of fragments and ``packet=None``, but that correlation does not hold:
         # under ``strict=False`` an *incomplete* datagram is reported as one
-        # contiguous ``bytes`` with its holes zero-filled, and analysed, because
-        # that is the payload buffer as it stands -- which is what
-        # :func:`~pcapkit.interface.misc.follow_tcp_stream` reconstructs a stream
-        # from. Overloads keyed on a literal cannot be selected from a ``completed``
+        # contiguous ``bytes`` -- its holes zero-filled, or only the prefix up to
+        # the first hole while the total length is unknown -- and analysed.
+        # Overloads keyed on a literal cannot be selected from a ``completed``
         # computed at runtime anyway, so they would make the reassemblers' own calls
         # untypeable while promising a correlation the code does not keep.
         def __init__(self, completed: 'Completion', id: 'DatagramID[_AT]', index: 'tuple[int, ...]', header: 'bytes', payload: 'bytes | tuple[bytes, ...]', packet: 'Optional[ProtocolBase | Deferred]', conflict: 'tuple[tuple[int, int], ...]') -> 'None': ...  # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long,redefined-builtin

pcapkit/foundation/reassembly/reassembly.py

@@ -177,8 +177,8 @@ def name(self) -> 'str':
         """Protocol name of current reassembly object.
 
         Note:
-            This property is not available as a class
-            attribute.
+            The same value is also available as a class attribute,
+            from the property of the same name on :class:`ReassemblyMeta`.
 
         """
         if hasattr(self, '__protocol_name__'):
@@ -190,8 +190,8 @@ def protocol(self) -> 'Type[ProtocolBase]':
         """Protocol of current reassembly object.
 
         Note:
-            This property is not available as a class
-            attribute.
+            The same value is also available as a class attribute,
+            from the property of the same name on :class:`ReassemblyMeta`.
 
         """
         if hasattr(self, '__protocol_type__'):
@@ -287,10 +287,10 @@ def expire(self, timestamp: 'float') -> 'list[_DT]':
         Note:
             The clock only advances when *this* reassembly object is fed, since
             a packet handed to it is the only evidence an offline parser has that
-            capture time has moved on. For IPv4 and TCP that is nearly every
-            frame of the relevant protocol; for IPv6 it is only the fragments,
-            so an IPv6 buffer that stalls and is followed by no further IPv6
-            fragment is reported as
+            capture time has moved on. For TCP that is every segment, for IPv4
+            every frame whose **DF** flag is clear, and for IPv6 only the
+            fragments, so an IPv6 buffer that stalls and is followed by no
+            further IPv6 fragment is reported as
             :attr:`Completion.PARTIAL <pcapkit.foundation.reassembly.data.data.Completion.PARTIAL>`
             at the end of the capture. That is the honest answer: the capture
             never shows that the deadline passed. A caller with an outside

pcapkit/foundation/reassembly/tcp.py

@@ -116,7 +116,7 @@ class TCP(ReassemblyBase[Packet, Datagram, BufferID, Buffer]):
     #: fragmentation one, and the numbers that look like candidates are not
     #: reassembly timeouts: the Maximum Segment Lifetime of
     #: :rfc:`9293#section-3.4.1` bounds how long a *segment* may linger in the
-    #: network, and the user timeout of :rfc:`9293#section-3.8.3` aborts a
+    #: network, and the user timeout of :rfc:`9293#section-3.10.8` aborts a
     #: connection whose data goes unacknowledged. Picking either as a default
     #: would silently discard buffered stream data on captures that are merely
     #: idle -- a long-lived connection with a two-minute lull is ordinary, while