diff --git a/docs/source/pcapkit/dumpkit/index.rst b/docs/source/pcapkit/dumpkit/index.rst index 2c8c7a6d13..8d083c1cfb 100644 --- a/docs/source/pcapkit/dumpkit/index.rst +++ b/docs/source/pcapkit/dumpkit/index.rst @@ -53,6 +53,7 @@ hierarchy: The ``tree`` output (also selected as ``text`` and ``txt``) is a lossy, human-readable view, and nothing reads it back. It renders ``''``, ``b''``, - :data:`None` and ``{}`` all as ``NIL``, and writes a newline inside a string - as is, breaking the layout. Use ``json`` or ``plist`` (also selected as + :data:`None` and ``{}`` all as ``NIL``, writes a newline inside a short + string as is, breaking the layout, and rewraps a longer one, turning its + newlines into spaces. Use ``json`` or ``plist`` (also selected as ``xml``) where the report has to round-trip. diff --git a/docs/source/pcapkit/foundation/engines/3rdparty.rst b/docs/source/pcapkit/foundation/engines/3rdparty.rst index fb6dbcdc78..fb29287adc 100644 --- a/docs/source/pcapkit/foundation/engines/3rdparty.rst +++ b/docs/source/pcapkit/foundation/engines/3rdparty.rst @@ -323,7 +323,7 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`. interface rather than 1.3.0. Every attribute this engine touches -- the ``pcap.pcap(name=..., promisc=...)`` - constructor, ``pcap.pcap.datalink``, ``snaplen``, iteration yielding + constructor, ``pcap.pcap.datalink``, iteration yielding ``(timestamp, bytes)``, and ``pcap.pcap.close`` -- is present on both and behaves identically, byte for byte and timestamp for timestamp, on ``pcap-ct`` 1.3.0b3 against ``pypcap`` 1.3.0 -- a statement about those diff --git a/docs/source/pcapkit/foundation/engines/index.rst b/docs/source/pcapkit/foundation/engines/index.rst index 019595784c..a4ec00a68c 100644 --- a/docs/source/pcapkit/foundation/engines/index.rst +++ b/docs/source/pcapkit/foundation/engines/index.rst @@ -75,7 +75,8 @@ their row says otherwise: +-----------------------------------------------------------------+---------------------------------------------------------------+ | Engine | Gap, and how it is surfaced | +=================================================================+===============================================================+ -| :class:`~pcapkit.foundation.engines.pyshark.PyShark` | no reassembly -- disabled with an | +| :class:`~pcapkit.foundation.engines.pyshark.PyShark` | no reassembly, and no ``trace_analyse``, since it reports | +| | dissected fields rather than octets -- each disabled with an | | | :class:`~pcapkit.utilities.warnings.AttributeWarning` | +-----------------------------------------------------------------+---------------------------------------------------------------+ | :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` | no protocol dissection at all, hence no reassembly and no | diff --git a/docs/source/pcapkit/foundation/extraction.rst b/docs/source/pcapkit/foundation/extraction.rst index 42ff82b12b..b4b9102c2d 100644 --- a/docs/source/pcapkit/foundation/extraction.rst +++ b/docs/source/pcapkit/foundation/extraction.rst @@ -33,6 +33,7 @@ and extracts parameters from a PCAP file. .. autoproperty:: reassembly .. autoproperty:: trace .. autoproperty:: engine + .. autoproperty:: magic_number .. automethod:: register_dumper .. automethod:: register_engine diff --git a/docs/source/pcapkit/foundation/reassembly/ip/index.rst b/docs/source/pcapkit/foundation/reassembly/ip/index.rst index 989e854734..7d62173812 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/index.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/index.rst @@ -52,20 +52,21 @@ Algorithm IF (FO = 0 AND MF = 0) { IF (buffer with BUFID is allocated) { flush all reassembly for this BUFID; - Submit datagram to next step; - DONE. } + Submit datagram to next step; + DONE. } IF (no buffer with BUFID is allocated) { allocate reassembly resources with BUFID; TIMER <- TLB; TDL <- 0; - put data from fragment into data buffer with BUFID - [from octet FO*8 to octet (TL-(IHL*4))+FO*8]; - set RCVBT bits [from FO to FO+((TL-(IHL*4)+7)/8)]; } + put data from fragment into data buffer with BUFID + [from octet FO*8 to octet (TL-(IHL*4))+FO*8]; + set RCVBT bits [from FO to FO+((TL-(IHL*4)+7)/8)]; + IF (MF = 0) { TDL <- TL-(IHL*4)+(FO*8) } @@ -100,6 +101,10 @@ Departures from the procedure above: timeout after its first-arriving fragment, on the capture's clock -- 60 seconds by default, per :rfc:`1122#section-3.3.2` for IPv4 and :rfc:`8200#section-4.5` for IPv6. +* A buffer flushed by an unfragmented datagram reusing its ``BUFID`` is + reported as an incomplete datagram rather than discarded. +* ``TDL`` starts at ``-1`` rather than ``0``, so that a datagram carrying no + data can complete (:issue:`1359`). * Only the final fragment's partial last ``RCVBT`` block is set. A non-final fragment carries a multiple of 8 octets, so one ending mid-block was cut or is malformed, and setting its last block would complete the datagram with diff --git a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst index c105fd3168..be390d5631 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst @@ -174,10 +174,10 @@ Terminology | |--> ipv6_frag.id | | |--> ipv6_frag.next | | |--> 'TDL' : (int) total data length - | |--> RCVBT : (bytearray) fragment received bit table - | | |--> (bytes) b'\\x00' -> not received - | | |--> (bytes) b'\\x01' -> received - | | |--> (bytes) ... + | |--> 'RCVBT' : (bytearray) fragment received bit table + | | |--> (bytes) b'\\x00' -> not received + | | |--> (bytes) b'\\x01' -> received + | | |--> (bytes) ... | |--> 'index' : (list) list of reassembled packets | | |--> (int) packet range number | |--> 'header' : (bytes) header buffer diff --git a/docs/source/pcapkit/foundation/reassembly/tcp.rst b/docs/source/pcapkit/foundation/reassembly/tcp.rst index d1a0875221..875045cd60 100644 --- a/docs/source/pcapkit/foundation/reassembly/tcp.rst +++ b/docs/source/pcapkit/foundation/reassembly/tcp.rst @@ -349,9 +349,10 @@ Terminology its own ``isn`` -- ``raw[n]`` holds the octet with sequence number ``isn + n``, and ``isn`` is revised downwards whenever a segment turns up below the data already buffered, so it is not necessarily the - connection's own initial sequence number. - :meth:`TCP.submit ` is the - one place that converts between the two. + connection's own initial sequence number. Each ACK's ``gap`` and + ``conflict`` lists are kept in sequence numbers too; + :meth:`TCP.submit ` turns + the ``gap`` entries into offsets into ``raw`` when it reports the buffer. Data Models =========== diff --git a/docs/source/pcapkit/foundation/traceflow/tcp.rst b/docs/source/pcapkit/foundation/traceflow/tcp.rst index dba740148d..e5c0043e56 100644 --- a/docs/source/pcapkit/foundation/traceflow/tcp.rst +++ b/docs/source/pcapkit/foundation/traceflow/tcp.rst @@ -40,7 +40,7 @@ Terminology .. code-block:: python - tract_dict = dict( + trace_dict = dict( protocol=data_link, # data link type from global header index=frame.info.number, # frame number frame=frame.info, # extracted frame info @@ -56,7 +56,8 @@ Terminology dst=ip.dst, # destination IP srcport=tcp.srcport, # TCP source port dstport=tcp.dstport, # TCP destination port - timestamp=frame.info.time_epoch, # frame timestamp + timestamp=float( + frame.info.time_epoch), # frame timestamp ) .. seealso:: :class:`pcapkit.foundation.traceflow.data.tcp.Packet` diff --git a/docs/source/pcapkit/index.rst b/docs/source/pcapkit/index.rst index 3ece229f80..5c1448b4ed 100644 --- a/docs/source/pcapkit/index.rst +++ b/docs/source/pcapkit/index.rst @@ -137,7 +137,8 @@ Environment Variables .. envvar:: PCAPKIT_DEVMODE - If set to ``1``, :mod:`pcapkit` will run in development mode. + If set to ``1`` (or ``yes``, ``true`` or ``on``, in any case), :mod:`pcapkit` + will run in development mode. .. seealso:: @@ -145,7 +146,8 @@ Environment Variables .. envvar:: PCAPKIT_VERBOSE - If set to ``1``, :mod:`pcapkit` will run with verbose output. + If set to ``1`` (or ``yes``, ``true`` or ``on``, in any case), :mod:`pcapkit` + will run with verbose output. .. seealso:: @@ -153,8 +155,8 @@ Environment Variables .. envvar:: PCAPKIT_SPHINX - If set to ``1``, :mod:`pcapkit` will run with Sphinx additional - typing hints and docstrings support. + If set to ``1`` (or ``yes``, ``true`` or ``on``, in any case), :mod:`pcapkit` + will run with Sphinx additional typing hints and docstrings support. .. seealso:: diff --git a/docs/source/pcapkit/utilities/index.rst b/docs/source/pcapkit/utilities/index.rst index 085ed3c8de..a893ab2bbc 100644 --- a/docs/source/pcapkit/utilities/index.rst +++ b/docs/source/pcapkit/utilities/index.rst @@ -62,8 +62,8 @@ following objects and functions: It also exports :mod:`pathlib`, :exc:`ModuleNotFoundError`, :class:`collections.abc.Collection` and :func:`functools.cached_property` as the -standard library has them. Their fallbacks were for interpreters below Python 3.8, -which 1.5.0 no longer installs on (:issue:`2051`). +standard library has them, with no fallback, since 1.5.0 requires Python 3.8 or +later (:issue:`2051`). .. function:: final(f) @@ -100,5 +100,5 @@ which 1.5.0 no longer installs on (:issue:`2051`). .. versionchanged:: 1.5.0 - No longer a backport below Python 3.8, which 1.5.0 no longer installs on + No longer a backport below Python 3.8, which 1.5.0 does not install on (:issue:`2051`). diff --git a/docs/source/pcapkit/utilities/warnings.rst b/docs/source/pcapkit/utilities/warnings.rst index ebca95270b..de072d1d36 100644 --- a/docs/source/pcapkit/utilities/warnings.rst +++ b/docs/source/pcapkit/utilities/warnings.rst @@ -29,9 +29,10 @@ channel**: category on the command line, noted below). The counts are the same in development mode as outside it: -:envvar:`PCAPKIT_DEVMODE` and :envvar:`PCAPKIT_VERBOSE` change how much detail -the log record carries, never how many records there are. Constructing a warning -class is not an act of reporting one -- it emits nothing and has no side effects. +:envvar:`PCAPKIT_DEVMODE` changes where the log records go, and +:envvar:`PCAPKIT_VERBOSE` how much detail they carry, never how many records +there are. Constructing a warning class is not an act of reporting one -- it +emits nothing and has no side effects. A next layer captured shorter than its length hint is parsed on trial, and replaced with :class:`~pcapkit.protocols.misc.raw.Raw` when its header was not diff --git a/pcapkit/__init__.py b/pcapkit/__init__.py index 856b8e2e1a..7f17c232c6 100644 --- a/pcapkit/__init__.py +++ b/pcapkit/__init__.py @@ -30,7 +30,7 @@ - Protocols (:mod:`pcapkit.protocols`) - Collection of all protocol family, with detailed + Collection of all protocol families, with detailed implementation and methods. - Utilities (:mod:`pcapkit.utilities`) diff --git a/pcapkit/dumpkit/common.py b/pcapkit/dumpkit/common.py index 4e14b44093..93e44bb9f9 100644 --- a/pcapkit/dumpkit/common.py +++ b/pcapkit/dumpkit/common.py @@ -220,7 +220,7 @@ def render_enum(o: 'enum.Enum | aenum.Enum') -> 'str': plausible member name, so a consumer splitting the rendering on ``::`` cannot tell it from a member genuinely so named -- and ``NONE`` *is* a declared name elsewhere in the library. And the defect is not confined to - zero: ``Flags(1)``, ``Flags(8)`` and ``Flags(65536)`` are every bit as + zero: ``Flags(1)``, ``Flags(8)`` and ``Flags(15)`` are every bit as nameless, so a guard written against ``value == 0`` would fix one case and leave the rest. diff --git a/pcapkit/dumpkit/pcap.py b/pcapkit/dumpkit/pcap.py index 745b1be447..56acfa3185 100644 --- a/pcapkit/dumpkit/pcap.py +++ b/pcapkit/dumpkit/pcap.py @@ -246,8 +246,8 @@ def _append_value(self, value: 'Data_Frame', file: 'IO[bytes]', name: 'str') -> # NOTE: The payload is read before the metadata so that a caller passing a # mapping rather than a dissected frame -- which the flow-tracing adapters # of several engines do -- still fails naming ``packet``. - # :mod:`pcapkit.foundation.extraction` substitutes a - # dict-capable trace format on the strength of that error. + # :mod:`pcapkit.foundation.extraction` cites that error when it + # substitutes a dict-capable trace format for those engines up front. packet = value.packet frame_info = value.frame_info ts_sec, ts_frac = self._make_timestamp(frame_info, value.time_epoch) diff --git a/pcapkit/foundation/engines/engine.py b/pcapkit/foundation/engines/engine.py index 473236dc43..b495250c3d 100644 --- a/pcapkit/foundation/engines/engine.py +++ b/pcapkit/foundation/engines/engine.py @@ -104,8 +104,9 @@ def name(self) -> 'str': """Engine name. Note: - This property is not available as a class - attribute. + This instance property is not what a class attribute lookup + reaches: on the class, the metaclass property of the same name + answers instead, with the same value. """ if hasattr(self, '__engine_name__'): @@ -117,8 +118,9 @@ def module(self) -> 'str': """Engine module name. Note: - This property is not available as a class - attribute. + This instance property is not what a class attribute lookup + reaches: on the class, the metaclass property of the same name + answers instead, with the same value. """ if hasattr(self, '__engine_module__'): diff --git a/pcapkit/foundation/extraction.py b/pcapkit/foundation/extraction.py index 8e1922c477..b94e95a1c0 100644 --- a/pcapkit/foundation/extraction.py +++ b/pcapkit/foundation/extraction.py @@ -835,11 +835,9 @@ def run(self) -> 'None': # pylint: disable=inconsistent-return-statements * pcap-ct driver: :class:`pcapkit.foundation.engines.pcap_ct.PCAP_CT` * PyPCAPFile driver: :class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile` - Warns: - pcapkit.utilities.warnings.EngineWarning: If the extraction engine is not - available: its dependency is not installed, its - :meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason` - rules it out, or the supplied engine is unknown. + Note: + An unavailable engine is not reported here: :meth:`__init__` settles + the engine, and warns about it, before calling this method. Raises: FormatError: If the selected engine cannot read the input's format -- @@ -1263,9 +1261,9 @@ def __init__(self, :class:`~pcapkit.corekit.context.ContextRegistry`, a single :class:`~pcapkit.corekit.context.ProtocolContext`, a mapping, or any iterable of contexts. The channel is honoured by the - ``default``, ``pcap`` and ``pcapng`` engines, which parse with - :mod:`pcapkit`'s own protocol implementations; the third party - engines ignore it. + :term:`default engine`, whose PCAP and PCAP-NG readers parse + with :mod:`pcapkit`'s own protocol implementations; the third + party engines ignore it. Warns: pcapkit.utilities.warnings.FormatWarning: Warns under the following circumstances: @@ -1281,7 +1279,9 @@ def __init__(self, engine first, which supports PCAP output. pcapkit.utilities.warnings.EngineWarning: If the requested extraction - engine is not available, cf. :meth:`run`. + engine is not available: its dependency is not installed, its + :meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason` + rules it out, or the supplied engine is unknown. pcapkit.utilities.warnings.AttributeWarning: If ``trace_analyse`` is requested while the extraction engine is PyShark. diff --git a/pcapkit/foundation/reassembly/data/data.py b/pcapkit/foundation/reassembly/data/data.py index b2c7020dd0..1504e1ecab 100644 --- a/pcapkit/foundation/reassembly/data/data.py +++ b/pcapkit/foundation/reassembly/data/data.py @@ -51,7 +51,7 @@ class Completion(EnumLookup, StrEnum): :class:`pcapkit.const.pcapng.tls_key_label.TLSKeyLabel` is the same kind of string enumeration, but it is generated like its :mod:`pcapkit.const.pcapng` siblings (:issue:`886`), so it derives from |aenum|_'s own - ``StrEnum`` (via :class:`~pcapkit.corekit.enum.EnumRegistry`). The + ``StrEnum``, alongside :class:`~pcapkit.corekit.enum.EnumRegistry`. The properties above hold for it, as ``aenum.StrEnum`` is a :class:`str` subclass, but ``isinstance``/``issubclass`` against ``pcapkit.utilities.compat.StrEnum`` does not on Python 3.11 and later, diff --git a/pcapkit/foundation/reassembly/data/ip.py b/pcapkit/foundation/reassembly/data/ip.py index e349bf8a17..811bd91984 100644 --- a/pcapkit/foundation/reassembly/data/ip.py +++ b/pcapkit/foundation/reassembly/data/ip.py @@ -67,7 +67,7 @@ class DatagramID(Info, Generic[_AT]): src: '_AT' #: Destination address. dst: '_AT' - #: IP protocol identifier. + #: IP identification. id: 'int' #: Payload protocol type. proto: 'TransType' @@ -150,10 +150,9 @@ class Datagram(DeferredPacket, Info, Generic[_AT]): # ``bytes`` payload and a parsed ``packet``, and an incomplete one with a # tuple of fragments and ``packet=None``, but that correlation does not hold: # under ``strict=False`` an *incomplete* datagram is reported as one - # contiguous ``bytes`` with its holes zero-filled, and analysed, because - # that is the payload buffer as it stands -- which is what - # :func:`~pcapkit.interface.misc.follow_tcp_stream` reconstructs a stream - # from. Overloads keyed on a literal cannot be selected from a ``completed`` + # contiguous ``bytes`` -- its holes zero-filled, or only the prefix up to + # the first hole while the total length is unknown -- and analysed. + # Overloads keyed on a literal cannot be selected from a ``completed`` # computed at runtime anyway, so they would make the reassemblers' own calls # untypeable while promising a correlation the code does not keep. def __init__(self, completed: 'Completion', id: 'DatagramID[_AT]', index: 'tuple[int, ...]', header: 'bytes', payload: 'bytes | tuple[bytes, ...]', packet: 'Optional[ProtocolBase | Deferred]', conflict: 'tuple[tuple[int, int], ...]') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long,redefined-builtin diff --git a/pcapkit/foundation/reassembly/reassembly.py b/pcapkit/foundation/reassembly/reassembly.py index 1a7d49d107..3a89563849 100644 --- a/pcapkit/foundation/reassembly/reassembly.py +++ b/pcapkit/foundation/reassembly/reassembly.py @@ -177,8 +177,8 @@ def name(self) -> 'str': """Protocol name of current reassembly object. Note: - This property is not available as a class - attribute. + The same value is also available as a class attribute, + from the property of the same name on :class:`ReassemblyMeta`. """ if hasattr(self, '__protocol_name__'): @@ -190,8 +190,8 @@ def protocol(self) -> 'Type[ProtocolBase]': """Protocol of current reassembly object. Note: - This property is not available as a class - attribute. + The same value is also available as a class attribute, + from the property of the same name on :class:`ReassemblyMeta`. """ if hasattr(self, '__protocol_type__'): @@ -287,10 +287,10 @@ def expire(self, timestamp: 'float') -> 'list[_DT]': Note: The clock only advances when *this* reassembly object is fed, since a packet handed to it is the only evidence an offline parser has that - capture time has moved on. For IPv4 and TCP that is nearly every - frame of the relevant protocol; for IPv6 it is only the fragments, - so an IPv6 buffer that stalls and is followed by no further IPv6 - fragment is reported as + capture time has moved on. For TCP that is every segment, for IPv4 + every frame whose **DF** flag is clear, and for IPv6 only the + fragments, so an IPv6 buffer that stalls and is followed by no + further IPv6 fragment is reported as :attr:`Completion.PARTIAL ` at the end of the capture. That is the honest answer: the capture never shows that the deadline passed. A caller with an outside diff --git a/pcapkit/foundation/reassembly/tcp.py b/pcapkit/foundation/reassembly/tcp.py index 97cbb11f97..f8eba1a4b3 100644 --- a/pcapkit/foundation/reassembly/tcp.py +++ b/pcapkit/foundation/reassembly/tcp.py @@ -116,7 +116,7 @@ class TCP(ReassemblyBase[Packet, Datagram, BufferID, Buffer]): #: fragmentation one, and the numbers that look like candidates are not #: reassembly timeouts: the Maximum Segment Lifetime of #: :rfc:`9293#section-3.4.1` bounds how long a *segment* may linger in the - #: network, and the user timeout of :rfc:`9293#section-3.8.3` aborts a + #: network, and the user timeout of :rfc:`9293#section-3.10.8` aborts a #: connection whose data goes unacknowledged. Picking either as a default #: would silently discard buffered stream data on captures that are merely #: idle -- a long-lived connection with a two-minute lull is ordinary, while diff --git a/pcapkit/foundation/registry/protocols.py b/pcapkit/foundation/registry/protocols.py index 16b686a742..bff9d12e28 100644 --- a/pcapkit/foundation/registry/protocols.py +++ b/pcapkit/foundation/registry/protocols.py @@ -1158,8 +1158,9 @@ def register_apptype(code: 'int | Enum_AppType', module: 'str | ModuleDescriptor ``name``. A composite such as ``tcp | udp``, or its string form ``'tcp|udp'``, names two and is refused for the same reason: one call registers under one transport protocol. - pcapkit.utilities.exceptions.ProtocolError: Raised lazily, from + pcapkit.utilities.exceptions.ProtocolError: Raised from :attr:`ModuleDescriptor.klass `, + which the port-keyed registry resolves before storing the entry, when ``module`` is a :class:`str` and either ``class_`` names no attribute of it, or ``class_`` was never given at all -- the latter distinguished from the former rather than reported as a @@ -1503,7 +1504,7 @@ def register_sctp(code: 'int | SCTP_PayloadProtocolIdentifier', module: 'str | M ############################################################################### -# NOTE: pcapkit.protocols.application.httpv2.HTTPv2.__frame__ +# NOTE: pcapkit.protocols.application.httpv2.HTTP.__frame__ def register_http_frame(code: 'HTTP_Frame', meth: 'str | tuple[HTTP_FrameParser, HTTP_FrameConstructor]', *, schema: 'Optional[Type[Schema_HTTP_FrameType]]' = None) -> 'None': """Register a frame parser. diff --git a/pcapkit/foundation/traceflow/data/data.py b/pcapkit/foundation/traceflow/data/data.py index d36545722e..2549500cbf 100644 --- a/pcapkit/foundation/traceflow/data/data.py +++ b/pcapkit/foundation/traceflow/data/data.py @@ -34,10 +34,11 @@ class Deferred: :class:`pcapkit.foundation.reassembly.data.data.Deferred`, and not shared with it. That one postpones a single ``analyze()`` call over bytes already in hand; this postpones a *submit* over a reassembler's buffers. The - reassembly and flow tracing subpackages are siblings and neither should - depend on the other, so the few lines are written in each rather than one - importing the other -- the same reason their ``data/data.py`` modules - mirror each other instead of merging. + reassembly and flow tracing subpackages are siblings, and the flow tracer + reaches the reassembler only lazily, when ``analyse`` asks for one, so the + few lines are written in each rather than one importing the other -- the + same reason their ``data/data.py`` modules mirror each other instead of + merging. Args: reassembly: The flow's own diff --git a/pcapkit/foundation/traceflow/data/tcp.py b/pcapkit/foundation/traceflow/data/tcp.py index a04afd52ba..e9221bc374 100644 --- a/pcapkit/foundation/traceflow/data/tcp.py +++ b/pcapkit/foundation/traceflow/data/tcp.py @@ -61,7 +61,7 @@ class Packet(Info, Generic[_AT]): #: TCP finish (FIN) flag. fin: 'bool' #: TCP reset (RST) flag. A connection can end abruptly as well as politely - #: (:rfc:`9293#section-3.5.2`), and the tracer cannot notice that unless the + #: (:rfc:`9293#section-3.5.3`), and the tracer cannot notice that unless the #: flag reaches it, otherwise a reset connection looks merely idle and a later #: connection reusing the same endpoints merges into it. rst: 'bool' @@ -130,7 +130,7 @@ class Buffer(Info, Generic[_AT]): #: and the final acknowledgement out of the flow. fin: 'set[tuple[_AT, int]]' #: Whether a TCP **RST** has been seen on this flow. A reset ends the - #: connection at once (:rfc:`9293#section-3.5.2`), where a polite close needs + #: connection at once (:rfc:`9293#section-3.5.3`), where a polite close needs #: a FIN from each side, so it is tracked as a flag rather than per endpoint. reset: 'bool' #: The flow's own :class:`~pcapkit.foundation.reassembly.tcp.TCP` reassembler, diff --git a/pcapkit/foundation/traceflow/tcp.py b/pcapkit/foundation/traceflow/tcp.py index 9bf1a280a4..50e366ea52 100644 --- a/pcapkit/foundation/traceflow/tcp.py +++ b/pcapkit/foundation/traceflow/tcp.py @@ -228,6 +228,9 @@ def trace(self, packet: 'Packet[_AT]', *, output: 'bool' = False) -> 'Dumper | s f'{packet.src}_{packet.srcport}-{packet.dst}_{packet.dstport}-{packet.timestamp}' + On an IPv6 flow every ``:`` in it is replaced by ``.``, since a + Windows file name cannot carry one. + It is built from the packet that **opened** the flow, not from the canonical buffer ID, so the label names the initiator first. The reverse half of a bidirectional @@ -272,8 +275,8 @@ def trace(self, packet: 'Packet[_AT]', *, output: 'bool' = False) -> 'Dumper | s if BUFID not in self._buffer: label = f'{packet.src}_{packet.srcport}-{packet.dst}_{packet.dstport}-{packet.timestamp}' if packet.src.version != 4: - # ``:`` is a path separator on Windows and a drive separator - # elsewhere in the tooling, so an IPv6 label cannot carry it + # ``:`` cannot appear in a Windows file name, where it separates + # the drive letter, so an IPv6 label cannot carry it label = label.replace(':', '.') label = self._unique_label(label) logger.debug('new TCP flow %s', label) @@ -465,7 +468,7 @@ def _ended(buffer: 'Buffer[_AT]') -> 'bool': Returns: Whether a teardown was observed -- a **FIN from both endpoints**, the polite close of :rfc:`9293#section-3.6`, or a **RST** from either, the - abrupt one of :rfc:`9293#section-3.5.2`. + abrupt one of :rfc:`9293#section-3.5.3`. This is not the same as "nothing more will arrive on these endpoints": the acknowledgement that completes a four-way close, and any duplicate of it, diff --git a/pcapkit/foundation/traceflow/traceflow.py b/pcapkit/foundation/traceflow/traceflow.py index ed43aa60fa..8226e8ba6d 100644 --- a/pcapkit/foundation/traceflow/traceflow.py +++ b/pcapkit/foundation/traceflow/traceflow.py @@ -175,8 +175,8 @@ def name(self) -> 'str': """Protocol name of current flow tracing object. Note: - This property is not available as a class - attribute. + The same value is also available as a class attribute, + from the property of the same name on :class:`TraceFlowMeta`. """ if hasattr(self, '__protocol_name__'): @@ -188,8 +188,8 @@ def protocol(self) -> 'Type[ProtocolBase]': """Protocol of current flow tracing object. Note: - This property is not available as a class - attribute. + The same value is also available as a class attribute, + from the property of the same name on :class:`TraceFlowMeta`. """ if hasattr(self, '__protocol_type__'): @@ -334,7 +334,8 @@ def make_fout(cls, fout: 'str' = './tmp', fmt: 'str' = 'pcap') -> 'tuple[Type[Du raise FileExists(*error.args).with_traceback(error.__traceback__) dumper = make_dumper(output) - # NOTE: as above -- make_dumper()'s subclass is always called 'DictDumper'. + # NOTE: logs ``output``, not ``dumper`` -- make_dumper()'s subclass is + # always called 'DictDumper'. logger.debug('flow tracing output root %s, format %s via %s', fout, fmt, output.__name__) return dumper, ext diff --git a/pcapkit/interface/core.py b/pcapkit/interface/core.py index e9d569e7e0..36667a9e61 100644 --- a/pcapkit/interface/core.py +++ b/pcapkit/interface/core.py @@ -54,13 +54,14 @@ # extraction engines # -# NOTE: each value is the key the engine is registered under in +# NOTE: each third-party engine's value is the key it is registered under in # :attr:`Extractor.__engine__ `, # not its display name -- ``Extractor`` looks the requested engine up in that # mapping, and an unknown key only warns and falls back to the default engine, so a -# constant whose value does not match the key would silently do nothing. The -# identifiers, by contrast, follow each engine's ``__engine_name__``, which is why -# the casing of the two sides differs. +# constant whose value does not match the key would silently do nothing. +# ``PCAPKit`` is ``'default'``, which ``Extractor`` resolves to its built-in engine +# without a lookup. The third-party identifiers, by contrast, follow each engine's +# ``__engine_name__``, which is why the casing of the two sides differs. DPKT = 'dpkt' Scapy = 'scapy' PCAPKit = 'default' @@ -88,7 +89,7 @@ def extract(fin: 'Optional[str | IO[bytes]]' = None, fout: 'Optional[str]' = Non Arguments: fin: file name to be read or a binary IO object; - if file not exist, raise :exc:`~pcapkit.utilities.exceptions.FileNotFound` + if the file does not exist, raise :exc:`~pcapkit.utilities.exceptions.FileNotFound` fout: file name to be written format: file format of output @@ -98,7 +99,7 @@ def extract(fin: 'Optional[str | IO[bytes]]' = None, fout: 'Optional[str]' = Non files: if split each frame into different files nofile: if no output file is to be dumped - verbose: a :obj:`bool` value or a function takes the + verbose: a :obj:`bool` value or a function that takes the :class:`~pcapkit.foundation.extraction.Extractor` instance and current parsed frame (depends on engine selected) as parameters to print verbose output information diff --git a/pcapkit/interface/misc.py b/pcapkit/interface/misc.py index ef23f37ecc..f9e7980d1e 100644 --- a/pcapkit/interface/misc.py +++ b/pcapkit/interface/misc.py @@ -75,7 +75,7 @@ class Stream(Info): def __init__(self, filename: 'Optional[str]', packets: 'tuple[Packet, ...]', conversations: 'tuple[bytes | tuple[bytes, ...], ...]') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long -def follow_tcp_stream(fin: 'Optional[str]' = None, verbose: 'bool' = False, # Extrator options +def follow_tcp_stream(fin: 'Optional[str]' = None, verbose: 'bool' = False, # Extractor options extension: 'bool' = True, engine: 'Optional[Engines]' = None, fout: 'Optional[str]' = None, format: 'Optional[Formats]' = None, # TraceFlow options # pylint: disable=redefined-builtin byteorder: 'ByteOrder' = sys.byteorder, nanosecond: 'bool' = False, @@ -84,7 +84,7 @@ def follow_tcp_stream(fin: 'Optional[str]' = None, verbose: 'bool' = False, """Follow TCP streams. Arguments: - fin: file name to be read; if file not exist, raise + fin: file name to be read; if the file does not exist, raise :exc:`~pcapkit.utilities.exceptions.FileNotFound` extension: if check and append extensions to output file verbose: if print verbose output information diff --git a/pcapkit/toolkit/dpkt.py b/pcapkit/toolkit/dpkt.py index 1a4a4e24cf..3a14808f37 100644 --- a/pcapkit/toolkit/dpkt.py +++ b/pcapkit/toolkit/dpkt.py @@ -561,11 +561,11 @@ def ipv4_reassembly(packet: 'Packet', timestamp: 'float', *, :class:`~pcapkit.foundation.engines.dpkt.DPKT` attached to the frame, so they are the captured ones. A packet built by hand carries no such buffer, and its octets are then |dpkt|_'s re-serialisation of it, in - which a zeroed checksum or length comes back recomputed -- - :func:`attach_buffer` is what makes them the wire's again. They are that - re-serialisation too where |dpkt|_'s own lengths misplace the layer - in the record, as behind some stacks of VLAN tags (:issue:`1646`). The - packet itself is left untouched either way. + which a zeroed checksum comes back recomputed, along with the Total + Length -- :func:`attach_buffer` is what makes them the wire's again. + They are that re-serialisation too where |dpkt|_'s own lengths + misplace the layer in the record, as behind some stacks of VLAN tags + (:issue:`1646`). The packet itself is left untouched either way. The header is the IHL's octets, and the payload ends at the Total Length, or at the end of the frame for a zero one. A Total Length @@ -648,7 +648,7 @@ def ipv6_reassembly(packet: 'Packet', timestamp: 'float', *, :class:`~pcapkit.foundation.engines.dpkt.DPKT` attached to the frame, so they are the captured ones. A packet built by hand carries no such buffer, and its octets are then |dpkt|_'s re-serialisation of it, in - which a zeroed checksum or length comes back recomputed -- + which a zeroed TCP, UDP or ICMPv6 checksum comes back recomputed -- :func:`attach_buffer` is what makes them the wire's again. They are that re-serialisation too where |dpkt|_'s own lengths misplace the layer in the record, as behind some stacks of VLAN tags (:issue:`1646`). The @@ -757,15 +757,15 @@ def tcp_reassembly(packet: 'Packet', timestamp: 'float', *, reassembled; otherwise :data:`None`. Note: - The header and payload are sliced out of the record octets + The header is sliced out of the record octets :class:`~pcapkit.foundation.engines.dpkt.DPKT` attached to the frame, so - they are the captured ones. A packet built by hand carries no such - buffer, and its octets are then |dpkt|_'s re-serialisation of it, in - which a zeroed checksum or length comes back recomputed -- - :func:`attach_buffer` is what makes them the wire's again. They are that - re-serialisation too where |dpkt|_'s own lengths misplace the layer - in the record, as behind some stacks of VLAN tags (:issue:`1646`). The - packet itself is left untouched either way. + it is the captured one, and the payload is the segment data |dpkt|_ + itself read from the frame. A packet built by hand carries no such + buffer, and its header is then |dpkt|_'s serialisation of the TCP + layer, which recomputes nothing. It is that serialisation too where + |dpkt|_'s own lengths misplace the layer in the record, as behind some + stacks of VLAN tags (:issue:`1646`). The packet itself is left untouched + either way. See Also: :class:`pcapkit.foundation.reassembly.tcp.TCP` @@ -836,15 +836,15 @@ def tcp_traceflow(packet: 'Packet', timestamp: 'float', *, traced; otherwise :data:`None`. Note: - The header and payload are sliced out of the record octets + The header is sliced out of the record octets :class:`~pcapkit.foundation.engines.dpkt.DPKT` attached to the frame, so - they are the captured ones. A packet built by hand carries no such - buffer, and its octets are then |dpkt|_'s re-serialisation of it, in - which a zeroed checksum or length comes back recomputed -- - :func:`attach_buffer` is what makes them the wire's again. They are that - re-serialisation too where |dpkt|_'s own lengths misplace the layer - in the record, as behind some stacks of VLAN tags (:issue:`1646`). The - packet itself is left untouched either way. + it is the captured one, and the payload is the segment data |dpkt|_ + itself read from the frame. A packet built by hand carries no such + buffer, and its header is then |dpkt|_'s serialisation of the TCP + layer, which recomputes nothing. It is that serialisation too where + |dpkt|_'s own lengths misplace the layer in the record, as behind some + stacks of VLAN tags (:issue:`1646`). The packet itself is left untouched + either way. See Also: :class:`pcapkit.foundation.traceflow.tcp.TCP` diff --git a/pcapkit/utilities/config.py b/pcapkit/utilities/config.py index 3b7af9dc73..4b8ddf3d43 100644 --- a/pcapkit/utilities/config.py +++ b/pcapkit/utilities/config.py @@ -212,8 +212,6 @@ def read_flag(name: 'str', default: 'bool' = False) -> 'bool': #: warn exactly once each. #: #: .. versionadded:: 1.5.0 -#: -#: Moved from ``pcapkit.toolkit.pypcapfile.IPV6_TCP_WARNED_LIMIT`` (:issue:`1726`). IPV6_TCP_WARNED_LIMIT = read_limit('IPV6_TCP_WARNED_LIMIT', 32, risk=( 'each capture remembered pins one of its packets in memory')) @@ -325,10 +323,10 @@ def read_flag(name: 'str', default: 'bool' = False) -> 'bool': #: been parsed before it. A guard whose answer moves with history is not a guard. #: #: 65,536 is what removes that. It is the whole span of a 16-bit wire length -#: field -- which is how an IP header, an IPv6 payload, a TCP or IPv4 option and -#: a PCAP-NG option all declare their size -- so no shortfall that one of those -#: can produce is subject to the budget at all, and every one of them is padded -#: unconditionally. Measured: the largest legitimate single +#: field -- which is how an IPv4 datagram, an IPv6 payload and a PCAP-NG option +#: all declare their size, and a TCP or IPv4 option's 8-bit length declares less +#: -- so no shortfall that one of those can produce is subject to the budget at +#: all, and every one of them is padded unconditionally. Measured: the largest legitimate single #: shortfall found anywhere was 65,495 octets, from a snapshot-truncated #: offload-sized frame (100 frames at ``incl_len`` 54 declaring an IPv4 total #: length of 65,535, padding 6,549,500 octets from 7,024 read), and 64,750 from a diff --git a/pcapkit/utilities/exceptions.py b/pcapkit/utilities/exceptions.py index 5d4f99c5b5..c19edf674a 100644 --- a/pcapkit/utilities/exceptions.py +++ b/pcapkit/utilities/exceptions.py @@ -285,8 +285,8 @@ class BaseError(Exception): def __init__(self, *args: 'Any', quiet: 'bool' = False, **kwargs: 'Any') -> 'None': """Log the error, unless it is quiet, and initialise the exception. - A loud error logs at ``CRITICAL`` and, unless ``PCAPKIT_DEVMODE`` is set, - installs the terse exception hooks; the class docstring says when an error is + A loud error logs at ``CRITICAL`` and, unless ``PCAPKIT_DEVMODE`` turns + development mode on, installs the terse exception hooks; the class docstring says when an error is quiet. Args: diff --git a/pcapkit/utilities/warnings.py b/pcapkit/utilities/warnings.py index 1ce2333b1a..0727deb090 100644 --- a/pcapkit/utilities/warnings.py +++ b/pcapkit/utilities/warnings.py @@ -19,11 +19,11 @@ exactly as they govern any other library's (see below for the one wrinkle in naming a category on the command line). -The counts are the same in development mode as outside it; -:data:`~pcapkit.utilities.logging.DEVMODE` and -:data:`~pcapkit.utilities.logging.VERBOSE` change how much detail the log record -carries, never how many records there are. Constructing a warning class is not -an act of reporting one, and has no side effects at all. +The counts are the same in development mode as outside it: +:data:`~pcapkit.utilities.logging.DEVMODE` changes where the log records go, and +:data:`~pcapkit.utilities.logging.VERBOSE` how much detail they carry, never how +many records there are. Constructing a warning class is not an act of reporting +one, and has no side effects at all. To silence :mod:`pcapkit` warnings, filter them like any other category:: @@ -424,9 +424,10 @@ class ExtractionWarning(BaseWarning, RuntimeWarning): class ConfigWarning(BaseWarning, RuntimeWarning): """Configuration warning. - Reported by :mod:`pcapkit.utilities.config` for an environment variable that - is invalid, and so falls back to its default, or that lifts a limit past its - default. + Reported for an environment variable that is invalid, and so falls back to + its default, or that lifts a limit past its default: by the readers of + :mod:`pcapkit.utilities.config`, and by this module for the flags + :mod:`pcapkit.utilities.logging` reads. .. versionadded:: 1.5.0