diff --git a/docs/source/contributing/pep.rst b/docs/source/contributing/pep.rst index d429a6f885..e746fe64a3 100644 --- a/docs/source/contributing/pep.rst +++ b/docs/source/contributing/pep.rst @@ -56,8 +56,7 @@ with DTLS: :class:`~pcapkit.protocols.application.ssl.SSL` is the shared record-layer base of :class:`~pcapkit.protocols.application.tls.TLS` and :class:`~pcapkit.protocols.application.dtls.DTLS` -- see the `DTLS`_ section below. Neighbor Discovery (:rfc:`4861`, the ``NDP`` above) and Multicast -Listener Discovery (:rfc:`9777`), both carried in ICMPv6, are not implemented, -nor is ICMP/ICMPv6 checksum computation or verification (:issue:`2470`). +Listener Discovery (:rfc:`9777`), both carried in ICMPv6, are not implemented. :class:`~pcapkit.protocols.link.vlan.VLAN` is the closer precedent for a pair whose *layout* is identical -- it holds the whole of the tag, and :class:`~pcapkit.protocols.link.c_tag.C_Tag` and @@ -819,11 +818,13 @@ These protocols parse a checksum or CRC field — :class:`~pcapkit.protocols.application.ospf.OSPF`, :class:`~pcapkit.protocols.transport.sctp.SCTP`, :class:`~pcapkit.protocols.transport.tcp.TCP` and -:class:`~pcapkit.protocols.transport.udp.UDP` — and of them only SCTP checks -whether the value is *right*, through +:class:`~pcapkit.protocols.transport.udp.UDP` — and SCTP was the first to +check whether the value is *right*, through :attr:`SCTP.checksum_valid `. -For the others the field is recorded and never questioned, so a corrupted -capture parses as cleanly as an intact one. +IPv4, OSPF, TCP and UDP check it too, as do ICMP and ICMPv6, as the +one's-complement half below describes. For HIP, HOPOPT, IPv6-Opts, IPX and MH the +field is recorded and never questioned, so a corrupted capture parses as cleanly +as an intact one. **These are different problems and they want separating**, because only one of them is cryptographic and the distinction decides what is actually hard. @@ -862,8 +863,10 @@ owner, and each lands on a precedent the library already has: large capture, and `Maybe Even Faster?`_ above is a standing concern, so it is a flag rather than unconditional behaviour. * **A wrong checksum is a warning, not a hard failure, and the outcome is - recorded on the protocol class.** The warning belongs in - :class:`~pcapkit.utilities.warnings.ProtocolWarning`, and the recording has an + recorded on the protocol class.** The warning is a + :class:`~pcapkit.utilities.warnings.ChecksumWarning`, a + :class:`~pcapkit.utilities.warnings.ProtocolWarning` of its own so a filter can + silence it alone (:issue:`2470`), and the recording has an exact precedent in :class:`~pcapkit.protocols.internet.esp.ESPStatus`: an enumeration of outcomes carried on the parsed result, so a caller can ask after the fact rather than having to have been capturing warnings at the @@ -879,8 +882,8 @@ owner, and each lands on a precedent the library already has: what the construction side needs in order to *emit* a correct checksum, and the two paths then share one definition instead of agreeing by coincidence. -One case still wants a decision at implementation time, since it is about -wording rather than design: **checksum offload**. A capture taken on the sending +One case was left to implementation time, since it is about wording rather +than design: **checksum offload**. A capture taken on the sending host routinely contains checksums the NIC had not computed yet, so the field is zero or garbage on the wire and "invalid" is the wrong word for it. Whatever status enumeration this grows should be able to say *not computed* distinctly @@ -888,10 +891,40 @@ from *wrong*, or it will cry wolf on the commonest capture there is — :class:`~pcapkit.protocols.internet.esp.ESPStatus`'s ``NO_SA`` member is the model for how to spell "expected, not an error". -Sequenced for **wave 2 or 3**. The cryptographic half could be done sooner since -ESP has already laid the groundwork. The checksum half builds on the IP -pseudo-header class, :class:`~pcapkit.protocols.misc.checksum.ip_pseudo.IP_Pseudo` -(:issue:`2471`). +**The one's-complement half is done** (:issue:`2470`, :issue:`2472` to +:issue:`2475`), on :class:`~pcapkit.protocols.misc.checksum.ip_pseudo.IP_Pseudo` +(:issue:`2471`) and :mod:`pcapkit.protocols.misc.checksum.checksum`, settled as +below: the toggles by the owner on :issue:`2471`, the rest at implementation, +as the offload question above asked; the owner accepted the toggles and the +offload heuristic on :issue:`2470`. + +* **Every checksum path is togglable, and verifying and computing default apart.** Parsing + verifies only under :envvar:`PCAPKIT_CHECKSUM_VERIFY`, off by default; + construction computes a checksum the caller leaves out unless + :envvar:`PCAPKIT_CHECKSUM_COMPUTE` turns it off. +* **The toggle is an environment variable**, with a per-call override only where + a caller needs one. A checksum given to ``make()`` is written as given, which is + what keeps a rebuild byte for byte; there is no + :class:`~pcapkit.foundation.extraction.Extractor` or per-instance parameter. +* **The IP layer owns the pseudo-header.** :class:`~pcapkit.protocols.internet.ipv4.IPv4` + and :class:`~pcapkit.protocols.internet.ipv6.IPv6` verify their upper layer's + checksum once parsed, and compute it for an upper layer constructed as their + payload; ``make()`` of TCP, UDP and ICMPv6 also takes the ``pseudo`` itself. +* **Offload is** :attr:`~pcapkit.protocols.misc.checksum.checksum.ChecksumStatus.NOT_COMPUTED`, + modelled on ``NO_SA``: a zero IPv4 or TCP checksum, or a TCP or UDP one holding + the pseudo-header's sum alone, as Linux offload leaves it, never warns. So does + a zero UDP checksum over IPv4 (:rfc:`768`), but not over IPv6, where it is + mandatory (:rfc:`8200#section-8.1`). +* **A fragment is** + :attr:`~pcapkit.protocols.misc.checksum.checksum.ChecksumStatus.NO_PSEUDO_HEADER`, + atomic ones aside: TCP, UDP, ICMP and ICMPv6 checksums cover all of the + datagram's data, which it carries part of. OSPF's Packet Length says what its + checksum covers, so it is verified when the fragment holds that much, and + :attr:`~pcapkit.protocols.misc.checksum.checksum.ChecksumStatus.TRUNCATED` + otherwise. + +The cryptographic half is sequenced for **wave 2 or 3**, and could be done sooner +since ESP has already laid the groundwork. Deterministic Construction Arguments ------------------------------------ @@ -1097,8 +1130,7 @@ is the abstract base :class:`~pcapkit.protocols.internet.icmpv4.ICMPv4` and :class:`~pcapkit.protocols.internet.icmpv6.ICMPv6` share for selector composition (:issue:`2436`); each implements its own message-type parsing independently rather than through a shared method body. The two are distinguished by whether they carry pseudo-header -checksums (ICMPv6 does; ICMPv4 does not) and parse without checksum computation or verification -(:issue:`2470`). +checksums (ICMPv6 does; ICMPv4 does not), both verified and computed (:issue:`2470`). **IGMP — done within wave 2** (:issue:`2439`, :issue:`2448`). The three concrete classes :class:`IGMPv1 `, @@ -1271,14 +1303,13 @@ files what it merely suspects transfers the work rather than doing it, and this project has already had to correct a finding whose count and whose diagnosis were both wrong when re-derived. -**Queued for wave 2 or 3 — checksum and integrity verification.** See `Checksum -and Integrity Verification`_ above. It splits in two, and the halves are not -equally blocked: the cryptographic half (HIP's ``HIP_MAC``/``RVS_HMAC``/ -``RELAY_HMAC`` and its signature parameters) could start whenever, since +**Queued for wave 2 or 3 — cryptographic integrity verification.** See `Checksum +and Integrity Verification`_ above. HIP's ``HIP_MAC``/``RVS_HMAC``/``RELAY_HMAC`` +and its signature parameters could start whenever, since :class:`~pcapkit.protocols.internet.esp.ESP` has already established both the keying-context channel and the report-rather-than-raise policy. The -one's-complement half (IPv4, TCP, UDP, ICMP/ICMPv6, OSPF) is :issue:`2470` and -:issue:`2472` to :issue:`2475`. +one's-complement half (IPv4, TCP, UDP, ICMP/ICMPv6, OSPF) is **done** +(:issue:`2470`, :issue:`2472` to :issue:`2475`). **IP pseudo-header — done** (:issue:`2471`). :class:`~pcapkit.protocols.misc.checksum.ip_pseudo.IP_Pseudo` is the diff --git a/docs/source/pcapkit/protocols/internet/icmpv6.rst b/docs/source/pcapkit/protocols/internet/icmpv6.rst index bfb09ba422..09aced2e6f 100644 --- a/docs/source/pcapkit/protocols/internet/icmpv6.rst +++ b/docs/source/pcapkit/protocols/internet/icmpv6.rst @@ -33,6 +33,7 @@ Octets Bits Name Description .. automethod:: make .. automethod:: _make_data + .. automethod:: _verify_pseudo_checksum .. automethod:: _read_data_type_none .. automethod:: _read_data_type_destination_unreachable diff --git a/docs/source/pcapkit/protocols/internet/ipv4.rst b/docs/source/pcapkit/protocols/internet/ipv4.rst index 7dc4885adc..76d1dc2f7d 100644 --- a/docs/source/pcapkit/protocols/internet/ipv4.rst +++ b/docs/source/pcapkit/protocols/internet/ipv4.rst @@ -89,6 +89,7 @@ Octets Bits Name Description :no-value: .. automethod:: __index__ + .. automethod:: __post_init__ Header Schemas -------------- diff --git a/docs/source/pcapkit/protocols/internet/ipv6.rst b/docs/source/pcapkit/protocols/internet/ipv6.rst index f810cf6162..b48f2e44e0 100644 --- a/docs/source/pcapkit/protocols/internet/ipv6.rst +++ b/docs/source/pcapkit/protocols/internet/ipv6.rst @@ -44,6 +44,7 @@ Octets Bits Name Description .. automethod:: _import_next_layer .. automethod:: __index__ + .. automethod:: __post_init__ .. automethod:: _read_ip_hextet diff --git a/docs/source/pcapkit/protocols/misc/checksum.rst b/docs/source/pcapkit/protocols/misc/checksum.rst index d580a05175..6f52a000d8 100644 --- a/docs/source/pcapkit/protocols/misc/checksum.rst +++ b/docs/source/pcapkit/protocols/misc/checksum.rst @@ -83,3 +83,40 @@ Data Models .. autoclass:: pcapkit.protocols.data.misc.checksum.ip_pseudo.IP_Pseudo :members: :show-inheritance: + +Internet Checksum +----------------- + +.. module:: pcapkit.protocols.misc.checksum.checksum + +:mod:`pcapkit.protocols.misc.checksum.checksum` contains what the +one's-complement Internet checksum (:rfc:`1071`) of the IPv4 header, TCP, UDP, +ICMP, ICMPv6 and OSPF shares. Parsing verifies a checksum only if +:data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` is on, and records the +outcome as ``checksum_status``; construction computes one left out unless +:data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE` is off (:issue:`2471`). +A mismatch warns with a :exc:`~pcapkit.utilities.warnings.ChecksumWarning`, +which a filter can silence alone (:issue:`2470`). + +A checksum over the IP pseudo-header is verified and computed by the enclosing +:class:`~pcapkit.protocols.internet.ipv4.IPv4` or +:class:`~pcapkit.protocols.internet.ipv6.IPv6` layer, which builds the +:class:`~pcapkit.protocols.misc.checksum.ip_pseudo.IP_Pseudo`, rather than by +the upper layer reaching back to its parent. So an upper layer constructed +alone, or behind another header such as an IPv6 extension header or an +Authentication Header, has its checksum computed only if given the ``pseudo`` +to compute it over, and one parsed alone records no outcome. ICMP's checksum, +over the message alone, is verified as it is parsed, unless the IP layer it is +parsed from is a fragment. + +.. autoclass:: pcapkit.protocols.misc.checksum.checksum.ChecksumStatus + :members: + :show-inheritance: + +.. autofunction:: pcapkit.protocols.misc.checksum.checksum.internet_checksum +.. autofunction:: pcapkit.protocols.misc.checksum.checksum.compute_checksum +.. autofunction:: pcapkit.protocols.misc.checksum.checksum.partial_checksum +.. autofunction:: pcapkit.protocols.misc.checksum.checksum.verify_checksum +.. autofunction:: pcapkit.protocols.misc.checksum.checksum.pseudo_header +.. autofunction:: pcapkit.protocols.misc.checksum.checksum.verify_upper +.. autofunction:: pcapkit.protocols.misc.checksum.checksum.fragment_payload diff --git a/docs/source/pcapkit/protocols/transport/tcp.rst b/docs/source/pcapkit/protocols/transport/tcp.rst index 08694b8c2b..28f6cb112f 100644 --- a/docs/source/pcapkit/protocols/transport/tcp.rst +++ b/docs/source/pcapkit/protocols/transport/tcp.rst @@ -50,6 +50,7 @@ Octets Bits Name Description .. automethod:: make .. automethod:: _make_data + .. automethod:: _verify_pseudo_checksum .. automethod:: _read_tcp_options .. automethod:: _read_mode_donone diff --git a/docs/source/pcapkit/protocols/transport/udp.rst b/docs/source/pcapkit/protocols/transport/udp.rst index 2f7777ff8f..e14c7208df 100644 --- a/docs/source/pcapkit/protocols/transport/udp.rst +++ b/docs/source/pcapkit/protocols/transport/udp.rst @@ -30,6 +30,7 @@ Octets Bits Name Description .. automethod:: make .. automethod:: _make_data + .. automethod:: _verify_pseudo_checksum .. autoattribute:: __proto__ :no-value: diff --git a/docs/source/pcapkit/utilities/config.rst b/docs/source/pcapkit/utilities/config.rst index 7bdbd780d8..bd17f9df56 100644 --- a/docs/source/pcapkit/utilities/config.rst +++ b/docs/source/pcapkit/utilities/config.rst @@ -4,8 +4,8 @@ Configuration .. module:: pcapkit.utilities.config :mod:`pcapkit.utilities.config` holds every tunable limit of :mod:`pcapkit` -(:issue:`1726`). Each limit is read from a ``PCAPKIT_*`` environment variable -once, when the module is imported: +(:issue:`1726`), and the checksum `flags`_. Each is read from a ``PCAPKIT_*`` +environment variable once, when the module is imported. For a limit: * A value that is not an integer, or is below the variable's minimum, warns with a :exc:`~pcapkit.utilities.warnings.ConfigWarning` and falls back to the @@ -134,13 +134,43 @@ Limits .. autodata:: pcapkit.utilities.config.MAX_ZERO_PAD_SHORTFALL .. autodata:: pcapkit.utilities.config.ZERO_PAD_BUDGET_RATIO +Flags +----- + +The checksum flags (:issue:`2471`) take the values :func:`read_flag` accepts, +and a bad value warns and falls back to the default. + +.. list-table:: + :header-rows: 1 + + * - Variable + - Default + - Effect + * - :envvar:`PCAPKIT_CHECKSUM_VERIFY` + - off + - parsing verifies each checksum and records the outcome + * - :envvar:`PCAPKIT_CHECKSUM_COMPUTE` + - on + - construction computes a checksum the caller leaves out + +.. envvar:: PCAPKIT_CHECKSUM_VERIFY + + Sets :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`. + +.. envvar:: PCAPKIT_CHECKSUM_COMPUTE + + Sets :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE`. + +.. autodata:: pcapkit.utilities.config.CHECKSUM_VERIFY +.. autodata:: pcapkit.utilities.config.CHECKSUM_COMPUTE + .. _config-readers: Readers ------- -The limits above are read with these, and a module that keeps a variable of its -own reads it with them too, so that a bad value is reported the same way. +The variables above are read with these, and a module that keeps a variable of +its own reads it with them too, so that a bad value is reported the same way. .. autofunction:: pcapkit.utilities.config.read_limit .. autofunction:: pcapkit.utilities.config.read_flag diff --git a/docs/source/pcapkit/utilities/warnings.rst b/docs/source/pcapkit/utilities/warnings.rst index de072d1d36..a6e8d8a27f 100644 --- a/docs/source/pcapkit/utilities/warnings.rst +++ b/docs/source/pcapkit/utilities/warnings.rst @@ -132,6 +132,10 @@ configured separately, through the :mod:`logging` module:: :no-members: :show-inheritance: +.. autoexception:: pcapkit.utilities.warnings.ChecksumWarning + :no-members: + :show-inheritance: + .. autoexception:: pcapkit.utilities.warnings.AttributeWarning :no-members: :show-inheritance: diff --git a/pcapkit/protocols/application/ospf.py b/pcapkit/protocols/application/ospf.py index ef89a4ee6c..2e5c8bf64e 100644 --- a/pcapkit/protocols/application/ospf.py +++ b/pcapkit/protocols/application/ospf.py @@ -58,6 +58,13 @@ length from the payload and dropped those octets; it takes them as ``packet_length`` and ``trailer`` (:issue:`1249`, :issue:`1455`). +.. versionchanged:: 1.6.0 + + The OSPFv2 checksum is computed when + :meth:`~pcapkit.protocols.application.ospf.OSPF.make` is given none, and + verified on read under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` + (:issue:`2475`). + """ import ipaddress from typing import TYPE_CHECKING, cast @@ -74,6 +81,7 @@ from pcapkit.protocols.schema.application.ospf import OSPF as Schema_OSPF from pcapkit.protocols.schema.application.ospf import \ CryptographicAuthentication as Schema_CryptographicAuthentication +from pcapkit.utilities import config from pcapkit.utilities.exceptions import ProtocolError if TYPE_CHECKING: @@ -214,6 +222,21 @@ def read(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_OSPF': ospf.__update__([ ('trailer', schema.trailer), ]) + + # NOTE: The checksum leaves out the 8-octet Authentication field at + # octet 16, and is unused under cryptographic authentication (RFC 2328). + if config.CHECKSUM_VERIFY and ospf.version == 2: + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus, verify_checksum # isort: skip # pylint: disable=import-outside-toplevel + if ospf.autype == Enum_Authentication.Cryptographic_authentication: + status = ChecksumStatus.NOT_COMPUTED + else: + end = max(ospf.len, self.length) + status = verify_checksum(self.alias, self._data[:16] + self._data[self.length:end], + ospf.chksum, length=end - 8) + ospf.__update__([ + ('checksum_status', status), + ]) + # OSPF carries no next-protocol field -- the body is LSAs and packet-type # specific fields, which pcapkit does not dissect -- so dispatch on the # -1 sentinel, as ARP does, rather than on a code read off the wire; @@ -229,7 +252,7 @@ def make(self, packet_length: 'Optional[int]' = None, router_id: 'IPv4Address | str | bytes | bytearray' = '0.0.0.0', # nosec: B104 area_id: 'IPv4Address | str | bytes | bytearray' = '0.0.0.0', # nosec: B104 - checksum: 'bytes' = b'\x00\x00', + checksum: 'Optional[bytes]' = None, auth_type: 'Enum_Authentication | StdlibEnum | AenumEnum | str | int' = Enum_Authentication.No_Authentication, auth_type_default: 'Optional[int]' = None, auth_type_namespace: 'Optional[dict[str, int] | dict[int, str] | Type[StdlibEnum] | Type[AenumEnum]]' = None, # pylint: disable=line-too-long @@ -252,7 +275,11 @@ def make(self, packet, or pass those as ``trailer`` instead. router_id: Router ID. area_id: Area ID. - checksum: Checksum. + checksum: Checksum; if :data:`None` and + :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE` is on, computed + for OSPFv2 over the packet but its authentication field + (:rfc:`2328#appendix-A.3.1`); zero otherwise, as cryptographic + authentication leaves it (:rfc:`2328#appendix-D.4.3`). auth_type: Authentication type. auth_type_default: Default value for ``auth_type`` if not specified. auth_type_namespace: Namespace for ``auth_type``. @@ -303,19 +330,27 @@ def make(self, or not 0 <= packet_length <= 0xFFFF): raise ProtocolError(f'OSPF: invalid packet length: {packet_length!r}') - return Schema_OSPF( + schema = Schema_OSPF( version=version, type=type_, # type: ignore[arg-type] length=packet_length, router_id=router_id, area_id=area_id, - checksum=checksum, + checksum=b'\x00\x00' if checksum is None else checksum, auth_type=auth_type_, # type: ignore[arg-type] auth_data=data, payload=payload, trailer=trailer, ) + if (checksum is None and config.CHECKSUM_COMPUTE and version == 2 + and auth_type_ != Enum_Authentication.Cryptographic_authentication): + from pcapkit.protocols.misc.checksum.checksum import compute_checksum # isort: skip # pylint: disable=import-outside-toplevel + # NOTE: Over the packet but the Authentication field, as read() checks it (#2475). + octets = schema.pack() + schema.checksum = compute_checksum(octets[:16] + octets[self.length:max(packet_length, self.length)]) + return schema + ########################################################################## # Data models. ########################################################################## diff --git a/pcapkit/protocols/data/application/ospf.py b/pcapkit/protocols/data/application/ospf.py index cc9b60624a..9b2af28480 100644 --- a/pcapkit/protocols/data/application/ospf.py +++ b/pcapkit/protocols/data/application/ospf.py @@ -12,6 +12,7 @@ from pcapkit.const.ospf.authentication import Authentication from pcapkit.const.ospf.packet import Packet + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus __all__ = ['OSPF', 'CryptographicAuthentication'] @@ -71,6 +72,9 @@ class OSPF(Protocol): #: Octets captured past the Packet Length, present only when there are #: any. trailer: 'bytes' + #: Outcome of verifying :attr:`chksum`, present only for OSPFv2 when + #: :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` is on. + checksum_status: 'ChecksumStatus' def __init__(self, version: 'int', type: 'Packet', len: 'int', router_id: 'IPv4Address', area_id: 'IPv4Address', chksum: 'bytes', autype: 'Authentication') -> 'None': ... # pylint: disable=super-init-not-called,unused-argument,line-too-long,multiple-statements,redefined-builtin diff --git a/pcapkit/protocols/data/internet/icmpv4.py b/pcapkit/protocols/data/internet/icmpv4.py index 8ce62c2a0e..4117fc3209 100644 --- a/pcapkit/protocols/data/internet/icmpv4.py +++ b/pcapkit/protocols/data/internet/icmpv4.py @@ -8,6 +8,7 @@ if TYPE_CHECKING: from pcapkit.const.icmp.type import Type as Enum_Type + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus __all__ = [ 'ICMPv4', @@ -32,7 +33,7 @@ class ICMPv4(Protocol): #: is entirely :attr:`type`-dependent; each subclass below documents #: which registry (if any) its own ``code`` values are drawn from. code: 'int' - #: Checksum, kept as on the wire; it is neither verified nor computed. + #: Checksum. checksum: 'bytes' #: Message length, in octets, header included. ICMP messages carry no #: length field of their own -- this is simply how many octets @@ -41,6 +42,11 @@ class ICMPv4(Protocol): #: truncated capture. length: 'int' + if TYPE_CHECKING: + #: Outcome of verifying :attr:`checksum`, present only when + #: :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` is on. + checksum_status: 'ChecksumStatus' + @info_final class Unknown(ICMPv4): diff --git a/pcapkit/protocols/data/internet/icmpv6.py b/pcapkit/protocols/data/internet/icmpv6.py index b3e48c49f2..ddeddeb7dd 100644 --- a/pcapkit/protocols/data/internet/icmpv6.py +++ b/pcapkit/protocols/data/internet/icmpv6.py @@ -8,6 +8,7 @@ if TYPE_CHECKING: from pcapkit.const.icmpv6.type import Type as Enum_Type + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus __all__ = [ 'ICMPv6', @@ -34,7 +35,7 @@ class ICMPv6(Protocol): #: documents which registry (if any) its own ``code`` values are drawn #: from. code: 'int' - #: Checksum, kept as on the wire; it is neither verified nor computed. + #: Checksum. checksum: 'bytes' #: Message length, in octets, header included. ICMPv6 messages carry no #: length field of their own -- this is simply how many octets @@ -43,6 +44,12 @@ class ICMPv6(Protocol): #: truncated capture. length: 'int' + if TYPE_CHECKING: + #: Outcome of verifying :attr:`checksum`, present only when + #: the enclosing IPv6 layer verified it + #: under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`. + checksum_status: 'ChecksumStatus' + @info_final class Unknown(ICMPv6): diff --git a/pcapkit/protocols/data/internet/ipv4.py b/pcapkit/protocols/data/internet/ipv4.py index c7cf2f64f2..5a717de337 100644 --- a/pcapkit/protocols/data/internet/ipv4.py +++ b/pcapkit/protocols/data/internet/ipv4.py @@ -29,6 +29,7 @@ from pcapkit.const.reg.transtype import TransType from pcapkit.corekit.infoclass import OrderedMultiInfo from pcapkit.corekit.multidict import OrderedMultiDict + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus __all__ = [ 'IPv4', @@ -142,6 +143,9 @@ class IPv4(Protocol): padding: 'bytes' #: Octets captured past the Total Length, present only when there are any. trailer: 'bytes' + #: Outcome of verifying :attr:`checksum`, present only when + #: :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` is on. + checksum_status: 'ChecksumStatus' def __init__(self, version: 'Literal[4]', hdr_len: 'int', tos: 'ToSField', len: 'int', id: 'int', flags: 'Flags', offset: 'int', ttl: 'timedelta', protocol: 'TransType', checksum: 'bytes', src: 'IPv4Address', dst: 'IPv4Address') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long,redefined-builtin diff --git a/pcapkit/protocols/data/transport/tcp.py b/pcapkit/protocols/data/transport/tcp.py index 5baf4266a2..7d512b2056 100644 --- a/pcapkit/protocols/data/transport/tcp.py +++ b/pcapkit/protocols/data/transport/tcp.py @@ -18,6 +18,7 @@ from pcapkit.const.tcp.mp_tcp_option import MPTCPOption from pcapkit.const.tcp.option import Option as OptionNumber from pcapkit.corekit.infoclass import OrderedMultiInfo + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus IPAddress = Union[IPv4Address, IPv6Address] @@ -104,6 +105,10 @@ class TCP(Protocol): padding: 'bytes' #: Connection control flags. connection: 'TCP_Flags' + #: Outcome of verifying :attr:`checksum`, present only when the + #: enclosing IP layer verified it under + #: :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`. + checksum_status: 'ChecksumStatus' def __init__(self, srcport: 'AppType', dstport: 'AppType', seq: 'int', ack: 'int', hdr_len: 'int', flags: 'Flags', window_size: 'int', checksum: 'bytes', urgent_pointer: 'int') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long,redefined-builtin diff --git a/pcapkit/protocols/data/transport/udp.py b/pcapkit/protocols/data/transport/udp.py index 85cfd4f1eb..9b3e42e39f 100644 --- a/pcapkit/protocols/data/transport/udp.py +++ b/pcapkit/protocols/data/transport/udp.py @@ -8,6 +8,7 @@ if TYPE_CHECKING: from pcapkit.const.reg.apptype import AppType + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus __all__ = ['UDP'] @@ -29,5 +30,9 @@ class UDP(Protocol): #: Octets captured past the Length field, present only when there are #: any. trailer: 'bytes' + #: Outcome of verifying :attr:`checksum`, present only when the + #: enclosing IP layer verified it under + #: :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`. + checksum_status: 'ChecksumStatus' def __init__(self, srcport: 'AppType', dstport: 'AppType', len: 'int', checksum: 'bytes') -> 'None': ... # pylint: disable=unused-argument,super-init-not-called,multiple-statements,redefined-builtin diff --git a/pcapkit/protocols/internet/icmpv4.py b/pcapkit/protocols/internet/icmpv4.py index 3889f99440..f222d873e7 100644 --- a/pcapkit/protocols/internet/icmpv4.py +++ b/pcapkit/protocols/internet/icmpv4.py @@ -45,6 +45,7 @@ from pcapkit.protocols.schema.internet.icmpv4 import ParameterProblem as Schema_ParameterProblem from pcapkit.protocols.schema.internet.icmpv4 import TimeExceeded as Schema_TimeExceeded from pcapkit.protocols.schema.internet.icmpv4 import Unknown as Schema_Unknown +from pcapkit.utilities import config from pcapkit.utilities.exceptions import ProtocolError from pcapkit.utilities.warnings import RegistryWarning, warn @@ -94,6 +95,12 @@ class ICMPv4(ICMP[Data_ICMPv4, Schema_ICMPv4], Any other type is kept as its raw message body, see :meth:`_read_data_type_none` and :meth:`_make_data_type_none`. + .. versionchanged:: 1.6.0 + + The checksum is computed when :meth:`make` is given none, and verified + on read under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`, + unless the enclosing IP layer is a fragment (:issue:`2470`). + """ ########################################################################## @@ -194,6 +201,19 @@ def read(self, length: 'Optional[int]' = None, *, meth = name[0] icmp = meth(schema.data, header=schema, length=length) + # NOTE: The checksum covers the whole message, with no pseudo-header (RFC 792); + # an IP fragment carries only part of one. + if config.CHECKSUM_VERIFY: + from pcapkit.protocols.misc.checksum.checksum import ( # isort: skip # pylint: disable=import-outside-toplevel + ChecksumStatus, _in_fragment, verify_checksum) + if _in_fragment.get(): + status = ChecksumStatus.NO_PSEUDO_HEADER + else: + status = verify_checksum(self.alias, self._data, schema.checksum, length=length) + icmp.__update__([ + ('checksum_status', status), + ]) + # NOTE: The quoted original datagram stays raw in the message body, # so there is no next layer; the schema has no ``payload`` field. self._next = NoPayload() # pylint: disable=attribute-defined-outside-init @@ -218,8 +238,9 @@ def make(self, type_namespace: Namespace of ICMP type. type_reversed: If the namespace of ICMP type is reversed. code: ICMP code; meaning is entirely ``type``-dependent. - checksum: Checksum, stored as given; :data:`None` leaves it - ``b'\\x00\\x00'``, uncomputed. + checksum: Checksum; if :data:`None`, computed over the message + (:rfc:`792`) when :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE` + is on, zero when it is off. body: Message body. :obj:`bytes` is parsed against ``type``'s own schema; a :obj:`dict` or a :class:`~pcapkit.protocols.data.internet.icmpv4.ICMPv4` data model is handed to that type's own constructor method @@ -266,16 +287,18 @@ def make(self, else: raise ProtocolError(f'{self.alias}: invalid message body type: {body.__class__}') - if checksum is None: - checksum = b'\x00\x00' - - return Schema_ICMPv4( + schema = Schema_ICMPv4( type=type_val, code=code, - checksum=checksum, + checksum=b'\x00\x00' if checksum is None else checksum, data=data_val, ) + if checksum is None and config.CHECKSUM_COMPUTE: + from pcapkit.protocols.misc.checksum.checksum import compute_checksum # isort: skip # pylint: disable=import-outside-toplevel + schema.checksum = compute_checksum(schema.pack()) + return schema + @classmethod def register_type(cls, code: 'Enum_Type', meth: 'str | tuple[TypeParser, TypeConstructor]') -> 'None': """Register a message body parser. diff --git a/pcapkit/protocols/internet/icmpv6.py b/pcapkit/protocols/internet/icmpv6.py index 5dca22af10..6813342493 100644 --- a/pcapkit/protocols/internet/icmpv6.py +++ b/pcapkit/protocols/internet/icmpv6.py @@ -47,6 +47,7 @@ from pcapkit.protocols.schema.internet.icmpv6 import ParameterProblem as Schema_ParameterProblem from pcapkit.protocols.schema.internet.icmpv6 import TimeExceeded as Schema_TimeExceeded from pcapkit.protocols.schema.internet.icmpv6 import Unknown as Schema_Unknown +from pcapkit.utilities import config from pcapkit.utilities.exceptions import ProtocolError from pcapkit.utilities.warnings import RegistryWarning, warn @@ -58,6 +59,8 @@ from mypy_extensions import DefaultArg, KwArg, NamedArg from typing_extensions import Literal + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo + TypeParser = Callable[[Schema_ICMPv6Type, NamedArg(Schema_ICMPv6, 'header'), NamedArg(int, 'length')], Data_ICMPv6] TypeConstructor = Callable[[Enum_Type, DefaultArg(Optional[Data_ICMPv6]), KwArg(Any)], Schema_ICMPv6Type] @@ -100,6 +103,13 @@ class ICMPv6(ICMP[Data_ICMPv6, Schema_ICMPv6], Listener Discovery (:rfc:`9777`), is kept as its raw message body, see :meth:`_read_data_type_none` and :meth:`_make_data_type_none`. + .. versionchanged:: 1.6.0 + + The checksum is computed when :meth:`make` is given none, over the + ``pseudo`` it is given or by the enclosing IPv6 layer, and verified by + that layer under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` + (:issue:`2470`). + """ ########################################################################## @@ -123,6 +133,10 @@ class ICMPv6(ICMP[Data_ICMPv6, Schema_ICMPv6], }, ) # type: DefaultDict[Enum_Type | int, str | tuple[TypeParser, TypeConstructor]] + #: Whether :meth:`make` left the checksum for the enclosing IP layer to + #: compute, see :mod:`pcapkit.protocols.misc.checksum.checksum`. + _checksum_pending = False + ########################################################################## # Properties. ########################################################################## @@ -210,6 +224,7 @@ def make(self, checksum: 'Optional[bytes]' = None, # Echo identifier and sequence number, the shortest body. body: 'bytes | Data_ICMPv6 | Schema_ICMPv6Type | dict[str, Any]' = b'\x00\x00\x00\x00', + pseudo: 'Optional[IP_Pseudo]' = None, **kwargs: 'Any') -> 'Schema_ICMPv6': """Make (construct) packet data. @@ -219,8 +234,10 @@ def make(self, type_namespace: Namespace of ICMPv6 type. type_reversed: If the namespace of ICMPv6 type is reversed. code: ICMPv6 code; meaning is entirely ``type``-dependent. - checksum: Checksum, stored as given; :data:`None` leaves it - ``b'\\x00\\x00'``, uncomputed. + checksum: Checksum; if :data:`None` and + :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE` is on, + computed over ``pseudo`` (:rfc:`4443#section-2.3`), or left to + an IPv6 layer this message is the payload of; zero otherwise. body: Message body. :obj:`bytes` is parsed against ``type``'s own schema; a :obj:`dict` or a :class:`~pcapkit.protocols.data.internet.icmpv6.ICMPv6` data @@ -228,6 +245,8 @@ def make(self, :attr:`__type__`); any other :class:`~pcapkit.protocols.schema.internet.icmpv6.ICMPv6Type` is used as is. + pseudo: Pseudo-header of the enclosing IPv6 layer; its length is + replaced by this message's own. **kwargs: Arbitrary keyword arguments. Returns: @@ -268,16 +287,23 @@ def make(self, else: raise ProtocolError(f'{self.alias}: invalid message body type: {body.__class__}') - if checksum is None: - checksum = b'\x00\x00' - - return Schema_ICMPv6( + schema = Schema_ICMPv6( type=type_val, code=code, - checksum=checksum, + checksum=b'\x00\x00' if checksum is None else checksum, data=data_val, ) + self._checksum_pending = False + if checksum is None and config.CHECKSUM_COMPUTE: + if pseudo is None: + self._checksum_pending = True + else: + from pcapkit.protocols.misc.checksum.checksum import compute_checksum, pseudo_header # isort: skip # pylint: disable=import-outside-toplevel + octets = schema.pack() + schema.checksum = compute_checksum(octets, prefix=pseudo_header(pseudo, len(octets))) + return schema + @classmethod def register_type(cls, code: 'Enum_Type', meth: 'str | tuple[TypeParser, TypeConstructor]') -> 'None': """Register a message body parser. @@ -342,6 +368,25 @@ def _make_data(cls, data: 'Data_ICMPv6') -> 'dict[str, Any]': # type: ignore[ov 'body': data, } + def _verify_pseudo_checksum(self, pseudo: 'Optional[IP_Pseudo]') -> 'None': + """Verify the checksum over the pseudo-header of the enclosing IP layer. + + That layer calls this once parsed, see + :func:`~pcapkit.protocols.misc.checksum.checksum.verify_upper`. + + Args: + pseudo: Pseudo-header, or :data:`None` if the IP layer gives none. + + """ + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus, verify_checksum # isort: skip # pylint: disable=import-outside-toplevel + + if pseudo is None: + status = ChecksumStatus.NO_PSEUDO_HEADER + else: + status = verify_checksum(self.alias, self._data, self._info.checksum, + length=pseudo.info.len, prefix=pseudo.data) + self._info.__update__(checksum_status=status) + def _read_data_type_none(self, schema: 'Schema_Unknown', *, header: 'Schema_ICMPv6', length: 'int') -> 'Data_Unknown': """Read ICMPv6 message of unknown type. diff --git a/pcapkit/protocols/internet/ipv4.py b/pcapkit/protocols/internet/ipv4.py index fcc88bfe9b..b85bedc56c 100644 --- a/pcapkit/protocols/internet/ipv4.py +++ b/pcapkit/protocols/internet/ipv4.py @@ -82,6 +82,7 @@ from pcapkit.protocols.data.internet.ipv4 import TSOption as Data_TSOption from pcapkit.protocols.data.internet.ipv4 import UnassignedOption as Data_UnassignedOption from pcapkit.protocols.internet.ip import IP +from pcapkit.protocols.protocol import _rebuilding from pcapkit.protocols.schema.internet.ipv4 import IPV4_HEADER_LEN from pcapkit.protocols.schema.internet.ipv4 import EOOLOption as Schema_EOOLOption from pcapkit.protocols.schema.internet.ipv4 import ESECOption as Schema_ESECOption @@ -104,6 +105,7 @@ from pcapkit.protocols.schema.internet.ipv4 import TSOption as Schema_TSOption from pcapkit.protocols.schema.internet.ipv4 import UnassignedOption as Schema_UnassignedOption from pcapkit.protocols.schema.schema import Schema +from pcapkit.utilities import config from pcapkit.utilities.exceptions import ProtocolError from pcapkit.utilities.warnings import ProtocolWarning, RegistryWarning, warn @@ -111,7 +113,7 @@ from datetime import timedelta from enum import IntEnum as StdlibEnum from ipaddress import IPv4Address - from typing import Any, Callable, DefaultDict, Optional, Type + from typing import IO, Any, Callable, DefaultDict, Optional, Type from aenum import IntEnum as AenumEnum from mypy_extensions import DefaultArg, KwArg, NamedArg @@ -208,6 +210,12 @@ class IPv4(IP[Data_IPv4, Schema_IPv4], * The TS option's maker takes an address as a string as well (:issue:`1402`). + .. versionchanged:: 1.6.0 + + The header checksum is computed when :meth:`make` is given none, and + verified on read under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` + (:issue:`2472`). + """ ########################################################################## @@ -360,6 +368,15 @@ def read(self, length: 'Optional[int]' = None, *, # pylint: disable=unused-argu ('padding', self._data[IPV4_HEADER_LEN + _used:ipv4.hdr_len]), ]) + # NOTE: The checksum covers the header alone (RFC 791 section 3.1); a zero + # one is what header checksum offload leaves on the sending host. + if config.CHECKSUM_VERIFY: + from pcapkit.protocols.misc.checksum.checksum import verify_checksum # isort: skip # pylint: disable=import-outside-toplevel + ipv4.__update__([ + ('checksum_status', verify_checksum(self.alias, self._data[:ipv4.hdr_len], schema.chksum, + offload=(b'\x00\x00',))), + ]) + # NOTE: Octets captured past the Total Length belong to this layer, # since its own length field is what leaves them out of the payload: an # Ethernet carrier hands the whole rest of its frame here, padding @@ -407,8 +424,14 @@ def read(self, length: 'Optional[int]' = None, *, # pylint: disable=unused-argu # as, and ``'UDP' in frame`` does not hold of a slice of UDP payload # (#1553). Reassembly takes ``packet.payload``, the octets as # captured, which this leaves as they were. - return self._decode_next_layer(ipv4, None if ipv4.offset else ipv4.protocol, _plen, - packet=__packet__) + proto = None if ipv4.offset else ipv4.protocol + if config.CHECKSUM_VERIFY: + # NOTE: An upper layer checksummed alone, as ICMP is, is not verified + # in a fragment; see pcapkit.protocols.misc.checksum.checksum (#2470). + from pcapkit.protocols.misc.checksum.checksum import fragment_payload # isort: skip # pylint: disable=import-outside-toplevel + with fragment_payload(bool(ipv4.offset or ipv4.flags.mf)): + return self._decode_next_layer(ipv4, proto, _plen, packet=__packet__) + return self._decode_next_layer(ipv4, proto, _plen, packet=__packet__) def make(self, tos_pre: 'Enum_ToSPrecedence | StdlibEnum | AenumEnum | int | str' = Enum_ToSPrecedence.Routine, @@ -441,7 +464,7 @@ def make(self, protocol_default: 'Optional[int]' = None, protocol_namespace: 'Optional[dict[str, int] | dict[int, str] | Type[StdlibEnum] | Type[AenumEnum]]' = None, # pylint: disable=line-too-long protocol_reversed: 'bool' = False, - checksum: 'bytes' = b'\x00\x00', + checksum: 'Optional[bytes]' = None, src: 'IPv4Address | str | int | bytes' = '127.0.0.1', dst: 'IPv4Address | str | int | bytes' = '0.0.0.0', # nosec: B104 options: 'Optional[list[Schema_Option | tuple[Enum_OptionNumber, dict[str, Any]] | bytes] | Option]' = None, # pylint: disable=line-too-long @@ -484,7 +507,10 @@ def make(self, protocol_default: Default value of ``protocol``. protocol_namespace: Namespace of ``protocol``. protocol_reversed: If the namespace of ``protocol`` is reversed. - checksum: Checksum of the packet. + checksum: Header checksum; computed over the header (:rfc:`791`) + if :data:`None` and + :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE` is on, zero + if it is off. src: Source address of the packet. dst: Destination address of the packet. options: Options of the packet. @@ -538,7 +564,7 @@ def make(self, if total_length is None: total_length = ihl * 4 + len(payload) - return Schema_IPv4( + schema = Schema_IPv4( vihl={ 'version': 4, 'ihl': ihl, @@ -560,7 +586,7 @@ def make(self, }, ttl=ttl_val, proto=proto, # type: ignore[arg-type] - chksum=checksum, + chksum=b'\x00\x00' if checksum is None else checksum, src=src, dst=dst, options=options_value, @@ -568,6 +594,29 @@ def make(self, trailer=trailer, ) + if checksum is None and config.CHECKSUM_COMPUTE: + from pcapkit.protocols.misc.checksum.checksum import compute_checksum # isort: skip # pylint: disable=import-outside-toplevel + schema.chksum = compute_checksum(schema.pack()[:ihl * 4]) + + # NOTE: A payload that left its checksum out gets it over this header's + # pseudo-header, unless a fragment, whose checksum covers the datagram (#2471). + if (config.CHECKSUM_COMPUTE and getattr(payload, '_checksum_pending', False) + and cast('ProtocolBase', payload).__index__() == proto and not (offset or mf)): + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo # isort: skip # pylint: disable=import-outside-toplevel + + # NOTE: The header is parsed back for a source route's destination, + # quietly, as this layer parses it again (cf. _rebuilding). + token = _rebuilding.set(True) + try: + header = IPv4(schema.pack()[:ihl * 4]) + finally: + _rebuilding.reset(token) + # NOTE: The payload puts its own length in the pseudo-header. + pseudo = IP_Pseudo(ip=header, protocol=proto, upper_length=0) + upper = cast('ProtocolBase', payload) + schema.payload = upper.from_data(upper.info, checksum=None, pseudo=pseudo).data + return schema + @classmethod def id(cls) -> 'tuple[Literal["IPv4"]]': # type: ignore[override] """Index ID of the protocol. @@ -623,6 +672,28 @@ def __index__(cls) -> 'Enum_TransType': # pylint: disable=invalid-index-returne """ return Enum_TransType.IPv4 # type: ignore[return-value] + def __post_init__(self, file: 'Optional[IO[bytes] | bytes]' = None, + length: 'Optional[int]' = None, **kwargs: 'Any') -> 'None': + """Post initialisation hook. + + Once parsed, the checksum of the upper layer is verified over this + layer's pseudo-header if :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` + is on, see :func:`~pcapkit.protocols.misc.checksum.checksum.verify_upper`. + + Args: + file: Source packet stream. + length: Length of packet data. + **kwargs: Arbitrary keyword arguments. + + See Also: + For construction arguments, please refer to :meth:`make`. + + """ + super().__post_init__(file, length, **kwargs) + if config.CHECKSUM_VERIFY: + from pcapkit.protocols.misc.checksum.checksum import verify_upper # isort: skip # pylint: disable=import-outside-toplevel + verify_upper(self) + ########################################################################## # Utilities. ########################################################################## diff --git a/pcapkit/protocols/internet/ipv6.py b/pcapkit/protocols/internet/ipv6.py index 41d7a3ef07..8795f51dd7 100644 --- a/pcapkit/protocols/internet/ipv6.py +++ b/pcapkit/protocols/internet/ipv6.py @@ -26,8 +26,9 @@ .. [*] https://en.wikipedia.org/wiki/IPv6_packet """ +import contextlib import ipaddress -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, cast from pcapkit.const.ipv6.extension_header import ExtensionHeader as Enum_ExtensionHeader from pcapkit.const.reg.transtype import TransType as Enum_TransType @@ -46,7 +47,7 @@ if TYPE_CHECKING: from enum import IntEnum as StdlibEnum from ipaddress import IPv6Address - from typing import Any, Mapping, Optional, Type + from typing import IO, Any, ContextManager, Mapping, Optional, Type from aenum import IntEnum as AenumEnum from typing_extensions import Literal @@ -311,7 +312,7 @@ def make(self, if payload_length > IP_MAX_LENGTH or jumbo_payload_length(next_val, bytes(payload)) is not None: payload_length = 0 - return Schema_IPv6( + schema = Schema_IPv6( hextet={ 'version': 6, 'class': traffic_class, @@ -326,6 +327,19 @@ def make(self, trailer=trailer, ) + # NOTE: A payload that left its checksum out gets it over this header's + # pseudo-header; see pcapkit.protocols.misc.checksum.checksum (#2471). + if (config.CHECKSUM_COMPUTE and getattr(payload, '_checksum_pending', False) + and cast('ProtocolBase', payload).__index__() == next_val): + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo # isort: skip # pylint: disable=import-outside-toplevel + + # NOTE: The payload follows this header directly, so no extension + # header moves an address; it puts its own length in the pseudo-header. + pseudo = IP_Pseudo(src=src, dst=dst, protocol=next_val, upper_length=0) + upper = cast('ProtocolBase', payload) + schema.payload = upper.from_data(upper.info, checksum=None, pseudo=pseudo).data + return schema + @classmethod def id(cls) -> 'tuple[Literal["IPv6"]]': # type: ignore[override] """Index ID of the protocol. @@ -356,6 +370,28 @@ def __index__(cls) -> 'Enum_TransType': # pylint: disable=invalid-index-returne """ return Enum_TransType.IPv6 # type: ignore[return-value] + def __post_init__(self, file: 'Optional[IO[bytes] | bytes]' = None, + length: 'Optional[int]' = None, **kwargs: 'Any') -> 'None': + """Post initialisation hook. + + Once parsed, the checksum of the upper layer is verified over this + layer's pseudo-header if :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` + is on, see :func:`~pcapkit.protocols.misc.checksum.checksum.verify_upper`. + + Args: + file: Source packet stream. + length: Length of packet data. + **kwargs: Arbitrary keyword arguments. + + See Also: + For construction arguments, please refer to :meth:`make`. + + """ + super().__post_init__(file, length, **kwargs) + if config.CHECKSUM_VERIFY: + from pcapkit.protocols.misc.checksum.checksum import verify_upper # isort: skip # pylint: disable=import-outside-toplevel + verify_upper(self) + ########################################################################## # Utilities. ########################################################################## @@ -613,11 +649,19 @@ def _decode_next_layer(self, ipv6: 'Data_IPv6', proto: 'Optional[int]' = None, # of a slice of UDP payload (#1545). Reassembly takes # ``fragment`` above, which this leaves as it was. self._exthdr_depth = len(_protos) # the headers the upper layer follows + scope = contextlib.nullcontext() # type: ContextManager[None] + if config.CHECKSUM_VERIFY: + # NOTE: As in IPv4.read; an atomic fragment, offset zero and no more + # to come, is a whole packet (#2470). + from pcapkit.protocols.misc.checksum.checksum import fragment_payload # isort: skip # pylint: disable=import-outside-toplevel + frags = self._exthdr.getlist(Enum_ExtensionHeader.IPv6_Frag) # type: ignore[arg-type] + scope = fragment_payload(any(frag.info.get('offset') or frag.info.get('mf') for frag in frags)) token = _frame_depth.set(depth + len(_protos)) try: - return super()._decode_next_layer(ipv6, None if opaque else proto, raw_len, packet=packet, - ipv6_exthdr=ProtoChain.from_list(_protos), # type: ignore[arg-type] - payload=payload) + with scope: + return super()._decode_next_layer(ipv6, None if opaque else proto, raw_len, packet=packet, + ipv6_exthdr=ProtoChain.from_list(_protos), # type: ignore[arg-type] + payload=payload) finally: _frame_depth.reset(token) diff --git a/pcapkit/protocols/misc/checksum/__init__.py b/pcapkit/protocols/misc/checksum/__init__.py index 47b0fce9d2..99197224a6 100644 --- a/pcapkit/protocols/misc/checksum/__init__.py +++ b/pcapkit/protocols/misc/checksum/__init__.py @@ -4,11 +4,15 @@ .. module:: pcapkit.protocols.misc.checksum -:mod:`pcapkit.protocols.misc.checksum` contains the pseudo-header protocols -that upper-layer checksums cover, currently -:class:`~pcapkit.protocols.misc.checksum.ip_pseudo.IP_Pseudo` only. +:mod:`pcapkit.protocols.misc.checksum` contains the pseudo-header that +upper-layer checksums cover, +:class:`~pcapkit.protocols.misc.checksum.ip_pseudo.IP_Pseudo`, and in +:mod:`~pcapkit.protocols.misc.checksum.checksum` what the one's-complement +checksums share, such as +:class:`~pcapkit.protocols.misc.checksum.checksum.ChecksumStatus`. """ +from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo -__all__ = ['IP_Pseudo'] +__all__ = ['IP_Pseudo', 'ChecksumStatus'] diff --git a/pcapkit/protocols/misc/checksum/checksum.py b/pcapkit/protocols/misc/checksum/checksum.py new file mode 100644 index 0000000000..92d1222318 --- /dev/null +++ b/pcapkit/protocols/misc/checksum/checksum.py @@ -0,0 +1,281 @@ +# -*- coding: utf-8 -*- +"""Internet Checksum +===================== + +.. module:: pcapkit.protocols.misc.checksum.checksum + +:mod:`pcapkit.protocols.misc.checksum.checksum` contains what the +one's-complement Internet checksum (:rfc:`1071`) of the IPv4 header, TCP, UDP, +ICMP, ICMPv6 and OSPF shares: :class:`ChecksumStatus`, the outcome a parse +records, and the functions that compute and verify the checksum. + +Verification is opt-in and computation opt-out, by +:data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` and +:data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE` (:issue:`2471`). A +mismatch warns with a :exc:`~pcapkit.utilities.warnings.ChecksumWarning`, +which a filter can silence alone (:issue:`2470`). + +A checksum over the IP pseudo-header is left to the enclosing IP layer, which +alone knows the pseudo-header, and builds it as an +:class:`~pcapkit.protocols.misc.checksum.ip_pseudo.IP_Pseudo`, rather than the +upper layer reaching back to its parent. The upper layer takes part through +these members: + +* ``_verify_pseudo_checksum(pseudo)`` verifies the checksum against + ``pseudo``, or records :attr:`ChecksumStatus.NO_PSEUDO_HEADER` for + :data:`None`; the IP layer calls it once parsed, see :func:`verify_upper`; +* ``make()`` takes the ``pseudo`` it computes the checksum over, and replaces + the length it carries with the upper layer's own; +* ``_checksum_pending`` is :data:`True` once ``make()`` left out a checksum it + had no ``pseudo`` for. An IP layer constructed with that upper layer as its + payload then rebuilds it with its own pseudo-header. + +A checksum over the upper layer alone, as ICMP's, is verified as it is parsed, +unless the IP layer it is parsed from is a fragment, see :func:`fragment_payload`. + +.. versionadded:: 1.6.0 + +""" +import contextlib +import contextvars +import enum +import struct +from typing import TYPE_CHECKING + +from pcapkit.corekit.enum import EnumLookup +from pcapkit.protocols.protocol import _rebuilding +from pcapkit.utilities.exceptions import FieldValueError, ProtocolError, stacklevel +from pcapkit.utilities.warnings import ChecksumWarning, warn + +if TYPE_CHECKING: + from typing import Collection, Iterator, Optional + + from pcapkit.protocols.internet.ipv4 import IPv4 + from pcapkit.protocols.internet.ipv6 import IPv6 + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo + +__all__ = [ + 'ChecksumStatus', + 'internet_checksum', 'compute_checksum', 'partial_checksum', 'verify_checksum', + 'pseudo_header', 'verify_upper', 'fragment_payload', +] + +#: contextvars.ContextVar[bool]: Whether the layer being parsed is the payload +#: of an IP fragment, see :func:`fragment_payload`. +_in_fragment = contextvars.ContextVar('_in_fragment', default=False) + + +class ChecksumStatus(EnumLookup, enum.IntEnum): + """Outcome of verifying a checksum, as recorded in ``checksum_status``. + + ``get`` / ``get_all`` come from :class:`~pcapkit.corekit.enum.EnumLookup`; + the class defines no ``_missing_``, so an unknown value raises + :exc:`ValueError`. + + """ + + #: The checksum matches. + VALID = 0 + #: The checksum does not match, which warns with a + #: :exc:`~pcapkit.utilities.warnings.ChecksumWarning`. + INVALID = 1 + #: The field holds no checksum: one the protocol lets the sender leave + #: out, as UDP over IPv4 does with zero (:rfc:`768`), or what checksum + #: offload leaves in a capture taken on the sending host. This is the + #: expected state for such a capture, and is **not** an error. + NOT_COMPUTED = 2 + #: Fewer octets were captured than the checksum covers. + TRUNCATED = 3 + #: The enclosing IP layer does not give what the checksum covers: it is + #: a fragment, holding only part of the datagram, or the pseudo-header + #: cannot be derived from it. + NO_PSEUDO_HEADER = 4 + + +def _sum(data: 'bytes') -> 'int': + """One's-complement sum of ``data`` in 16-bit words (:rfc:`1071`). + + Args: + data: Octets to sum, padded with a zero octet if of odd length. + + Returns: + The sum, folded to 16 bits. + + """ + if len(data) % 2: + data += b'\x00' + total = sum(struct.unpack(f'!{len(data) // 2}H', data)) + while total > 0xFFFF: + total = (total & 0xFFFF) + (total >> 16) + return total + + +def internet_checksum(data: 'bytes') -> 'int': + """Compute the Internet checksum of ``data`` (:rfc:`1071`). + + Args: + data: Octets the checksum covers, with the checksum field zeroed. + + Returns: + The one's complement of the one's-complement sum of ``data``. + + """ + return ~_sum(data) & 0xFFFF + + +def compute_checksum(data: 'bytes', *, prefix: 'bytes' = b'', nonzero: 'bool' = False) -> 'bytes': + """Compute the checksum field of ``data``. + + Args: + data: Octets the checksum covers, with the checksum field zeroed. + prefix: Pseudo-header octets the checksum covers ahead of ``data``. + nonzero: Whether a computed zero is written as ``0xFFFF``, as UDP does + (:rfc:`768`), since its zero means no checksum. + + Returns: + The two octets of the checksum field. + + """ + value = internet_checksum(prefix + data) + if nonzero and not value: + value = 0xFFFF + return value.to_bytes(2, 'big') + + +def partial_checksum(prefix: 'bytes') -> 'bytes': + """The field checksum offload leaves for a pseudo-header checksum. + + A sender that offloads the checksum to its NIC writes the one's-complement + sum of the pseudo-header alone, uncomplemented, for the NIC to finish, as + Linux does for TCP and UDP; a capture taken on that host records it so. + + Args: + prefix: Pseudo-header octets. + + Returns: + The two octets of that partial checksum. + + """ + return _sum(prefix).to_bytes(2, 'big') + + +def verify_checksum(alias: 'str', data: 'bytes', field: 'bytes', *, length: 'Optional[int]' = None, + prefix: 'bytes' = b'', absent: 'Optional[bytes]' = None, + offload: 'Collection[bytes]' = ()) -> 'ChecksumStatus': + """Verify the checksum ``field`` of ``data``. + + Args: + alias: Protocol alias, to name in the warning. + data: Octets the checksum covers, the checksum field included. + field: The checksum field, as on the wire. + length: Octets the checksum covers, as the headers declare it; the + length of ``data`` by default. + prefix: Pseudo-header octets the checksum covers ahead of ``data``. + absent: Field value that means the sender left the checksum out. + offload: Field values checksum offload leaves on the sending host. + + Returns: + :attr:`~ChecksumStatus.VALID` if the sum of ``prefix`` and ``data`` is + all ones (:rfc:`1071`), or the reason it is not. + + Warns: + ChecksumWarning: If the checksum is :attr:`~ChecksumStatus.INVALID`, + unless a payload is being rebuilt: the layer it is rebuilt for + parses the same octets again, so a rebuilt frame warns once. + + """ + if length is not None: + if len(data) < length: + return ChecksumStatus.TRUNCATED + data = data[:length] + if absent is not None and field == absent: + return ChecksumStatus.NOT_COMPUTED + + total = _sum(prefix + data) + if total == 0xFFFF: + return ChecksumStatus.VALID + if field in offload: + return ChecksumStatus.NOT_COMPUTED + + # NOTE: Adding the field's complement takes it back out of the sum, so the + # value expected is found without knowing where the field sits. + value = int.from_bytes(field, 'big') + rest = total + (~value & 0xFFFF) + expected = ~((rest & 0xFFFF) + (rest >> 16)) & 0xFFFF + if not _rebuilding.get(): + warn(f'{alias}: checksum {value:#06x} is wrong; {expected:#06x} expected', + ChecksumWarning, stacklevel=stacklevel()) + return ChecksumStatus.INVALID + + +def pseudo_header(pseudo: 'IP_Pseudo', length: 'int') -> 'bytes': + """Octets of ``pseudo`` with ``length`` as its upper-layer length. + + Args: + pseudo: Pseudo-header of the enclosing IP layer. + length: Upper-layer length the checksum is computed for. + + Returns: + :attr:`pseudo.data `, made + anew if it carries another length. + + Raises: + FieldValueError: If ``length`` does not fit the length field. + + """ + if pseudo.info.len == length: + return pseudo.data + return type(pseudo)(src=pseudo.src, dst=pseudo.dst, protocol=pseudo.protocol, + upper_length=length, zero=pseudo.info.zero).data + + +def verify_upper(ip: 'IPv4 | IPv6') -> 'None': + """Verify the checksum of the upper layer ``ip`` carries over its pseudo-header. + + The upper layer is the payload of ``ip``, past any Authentication Header + of IPv4. It is verified if it defines ``_verify_pseudo_checksum``, against + the :class:`~pcapkit.protocols.misc.checksum.ip_pseudo.IP_Pseudo` of ``ip``, + or against :data:`None` if that cannot be derived or names another protocol. + + Args: + ip: Parsed IPv4 or IPv6 layer. + + """ + from pcapkit.protocols.internet.ah import AH # isort: skip # pylint: disable=import-outside-toplevel + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo # isort: skip # pylint: disable=import-outside-toplevel + + upper = ip.payload + # NOTE: IPv6 walks an AH as an extension header; IPv4 carries it as its payload. + while isinstance(upper, AH): + upper = upper.payload + verify = getattr(upper, '_verify_pseudo_checksum', None) + if verify is None: + return + + try: + pseudo = IP_Pseudo(ip=ip) # type: Optional[IP_Pseudo] + except (ProtocolError, FieldValueError): + pseudo = None + if pseudo is not None and pseudo.protocol != upper.__index__(): + pseudo = None + verify(pseudo) + + +@contextlib.contextmanager +def fragment_payload(fragment: 'bool') -> 'Iterator[None]': + """Mark the payload an IP layer parses as a fragment's, or not. + + A checksum over the upper layer alone, as ICMP's (:rfc:`792`), covers all + of the datagram's data, which a fragment carries part of, so the upper + layer of one records :attr:`~ChecksumStatus.NO_PSEUDO_HEADER` rather than + being verified (:issue:`2470`). The nearest enclosing IP layer decides. + + Args: + fragment: Whether the IP layer is a fragment; an atomic one is not. + + """ + token = _in_fragment.set(fragment) + try: + yield + finally: + _in_fragment.reset(token) diff --git a/pcapkit/protocols/misc/checksum/ip_pseudo.py b/pcapkit/protocols/misc/checksum/ip_pseudo.py index 9a7defa403..79442adf97 100644 --- a/pcapkit/protocols/misc/checksum/ip_pseudo.py +++ b/pcapkit/protocols/misc/checksum/ip_pseudo.py @@ -372,7 +372,8 @@ def _source(self, ip: 'IPv4 | IPv6') -> 'IPv4Address | IPv6Address': if any, the source address otherwise. Raises: - ProtocolError: If a Destination Options header is not parsed. + ProtocolError: If a Destination Options header is not parsed, + however deep in the chain. """ src = ip.info.src @@ -386,18 +387,39 @@ def _source(self, ip: 'IPv4 | IPv6') -> 'IPv4Address | IPv6Address': if option is not None: src = option.address - # If a malformed Destination Options header stopped the walk, it becomes the upper layer as Raw (#2480). - # Only raise an error if the Destination Options header is the next protocol but not in extension_headers. - if (ip.info.next == Enum_ExtensionHeader.IPv6_Opts and - not ip.extension_headers.getlist(Enum_ExtensionHeader.IPv6_Opts) and - hasattr(ip, 'payload') and ip.payload is not None): - from pcapkit.protocols.misc.raw import Raw - if type(ip.payload) is Raw: - raise ProtocolError(f'{self.alias}: Destination Options of {ip.alias} not ' - 'parsed; give src') - + # NOTE: A header the walk stopped on, at any depth, is unparsed too, as + # a malformed one is (#2480), and may hold a Home Address option. + if self._walk(ip)[0] == Enum_ExtensionHeader.IPv6_Opts: + raise ProtocolError(f'{self.alias}: Destination Options of {ip.alias} not ' + 'parsed; give src') return src + @staticmethod + def _walk(ip: 'IPv6') -> 'tuple[Enum_TransType, int]': + """Walk the extension headers of an IPv6 layer up to its upper layer. + + A Mobility Header or HIP (:rfc:`6275#section-6.1.1`, :rfc:`7401#section-5.1.1`) + is the upper layer itself, and an Encapsulating Security Payload + (:rfc:`4303`) ends the walk, its upper layer being encrypted. + + Args: + ip: IPv6 layer. + + Returns: + The code the walk stopped on, and the octets of the headers ahead of + it. The code is an extension header's if ``ip`` did not parse that + header: a malformed one (:issue:`2480`), one past the header limit, + or one not captured. + + """ + code, before = ip.info.next, 0 + for _, header in ip.extension_headers.items(multi=True): + if code in _TERMINAL_HEADERS or code == Enum_TransType.ESP: + break + before += header.length + code = header.info.next + return code, before + def _destination(self, ip: 'IPv4 | IPv6') -> 'IPv4Address | IPv6Address': """Read the pseudo-header destination address from an IP layer. @@ -444,14 +466,10 @@ def _destination(self, ip: 'IPv4 | IPv6') -> 'IPv4Address | IPv6Address': f'{route.type} not parsed; give dst') dst = final - # If a malformed Routing header stopped the walk, it becomes the upper layer as Raw (#2480). - # Only raise an error if the Routing header is the next protocol but not in extension_headers. - if (ip.info.next == Enum_ExtensionHeader.IPv6_Route and - not ip.extension_headers.getlist(Enum_ExtensionHeader.IPv6_Route) and - hasattr(ip, 'payload') and ip.payload is not None): - from pcapkit.protocols.misc.raw import Raw - if type(ip.payload) is Raw: - raise ProtocolError(f'{self.alias}: Routing header of {ip.alias} not parsed; give dst') + # NOTE: A header the walk stopped on, at any depth, is unparsed too, as + # a malformed one is (#2480), and may have segments left. + if self._walk(ip)[0] == Enum_ExtensionHeader.IPv6_Route: + raise ProtocolError(f'{self.alias}: Routing header of {ip.alias} not parsed; give dst') return dst def _upper_protocol(self, ip: 'IPv4 | IPv6') -> 'Enum_TransType': @@ -507,42 +525,11 @@ def _upper(self, ip: 'IPv4 | IPv6') -> 'tuple[Enum_TransType, int]': # NOTE: The upper layer of an ESP (RFC 4303) is encrypted, so it is never walked. unparsed = code in (Enum_TransType.AH, Enum_TransType.ESP) else: - code, before = ip.info.next, 0 - for _, header in ip.extension_headers.items(multi=True): - # NOTE: An ESP header is never walked, its upper layer being encrypted (RFC 4303). - if code in _TERMINAL_HEADERS or code == Enum_TransType.ESP: - break - before += header.length - code = header.info.next + code, before = self._walk(ip) unparsed = code not in _TERMINAL_HEADERS and self._is_extension_header(code) - # If a malformed extension header stopped the walk, it becomes the upper layer as Raw (#2480). - # Provide a more specific error message identifying which extension header failed. - if unparsed: - from pcapkit.protocols.misc.raw import Raw - - # Check if the upper layer is Raw, indicating an unparsed extension header. - # Only raise a specific error if this extension header is NOT in the parsed headers. - if hasattr(ip, 'payload') and type(ip.payload) is Raw: - # Try to identify which extension header failed. ``ProtocolError`` - # subclasses ``ValueError``, so only the enum lookup itself -- not - # the raises below -- may sit inside this ``except ValueError``. - try: - ext_header = Enum_ExtensionHeader(code) # type: 'Optional[Enum_ExtensionHeader]' - except ValueError: - ext_header = None - if ext_header is not None: - # Check if this header type was successfully parsed by looking in extension_headers - ext_list = ip.extension_headers.getlist(ext_header) - # If the header is not in the parsed list, it failed to parse - if not ext_list: - if ext_header == Enum_ExtensionHeader.IPv6_Opts: - raise ProtocolError(f'{self.alias}: Destination Options of {ip.alias} not ' - 'parsed; give src') - elif ext_header == Enum_ExtensionHeader.IPv6_Route: - raise ProtocolError(f'{self.alias}: Routing header of {ip.alias} not parsed; give dst') - # Fall through to generic error message if not a recognized case - + # NOTE: What is missing is the protocol and the length, even where an + # unparsed header also hides an address (#2513). if code is None or unparsed: raise ProtocolError(f'{self.alias}: upper layer of {ip.alias} not parsed; ' 'give protocol and upper_length') diff --git a/pcapkit/protocols/transport/tcp.py b/pcapkit/protocols/transport/tcp.py index 98fb687a32..158990c017 100644 --- a/pcapkit/protocols/transport/tcp.py +++ b/pcapkit/protocols/transport/tcp.py @@ -135,6 +135,7 @@ from pcapkit.protocols.schema.transport.tcp import UserTimeout as Schema_UserTimeout from pcapkit.protocols.schema.transport.tcp import WindowScale as Schema_WindowScale from pcapkit.protocols.transport.transport import Transport +from pcapkit.utilities import config from pcapkit.utilities.exceptions import ProtocolError from pcapkit.utilities.warnings import RegistryWarning, warn @@ -152,6 +153,7 @@ from pcapkit.protocols.data.transport.tcp import MPTCP as Data_MPTCP from pcapkit.protocols.data.transport.tcp import MPTCPJoin as Data_MPTCPJoin from pcapkit.protocols.data.transport.tcp import Option as Data_Option + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo from pcapkit.protocols.protocol import ProtocolBase from pcapkit.protocols.schema.transport.tcp import MPTCP as Schema_MPTCP from pcapkit.protocols.schema.transport.tcp import Flags as Schema_Flags @@ -328,6 +330,13 @@ class TCP(Transport[Data_TCP, Schema_TCP], present (:issue:`1215`, :issue:`1216`), and ADD_ADDR parses the :rfc:`8684` layout (:issue:`1217`). + .. versionchanged:: 1.6.0 + + The checksum is computed when :meth:`make` is given none, over the + ``pseudo`` it is given or by the enclosing IP layer, and verified by that + layer under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` + (:issue:`2473`). + """ ########################################################################## @@ -437,6 +446,10 @@ class TCP(Transport[Data_TCP, Schema_TCP], }, ) # type: DefaultDict[int, str | tuple[MPOptionParser, MPOptionConstructor]] + #: Whether :meth:`make` left the checksum for the enclosing IP layer to + #: compute, see :mod:`pcapkit.protocols.misc.checksum.checksum`. + _checksum_pending = False + ########################################################################## # Properties. ########################################################################## @@ -590,12 +603,13 @@ def make(self, syn: 'bool' = False, fin: 'bool' = False, window: 'int' = 65535, # reasonable default value - checksum: 'bytes' = b'\x00\x00', + checksum: 'Optional[bytes]' = None, urgent: 'int' = 0, options: 'Optional[list[Schema_Option | tuple[Enum_Option, dict[str, Any]] | bytes] | Option]' = None, # pylint: disable=line-too-long payload: 'bytes | ProtocolBase | Schema' = b'', padding: 'bytes' = b'', reserved: 'int' = 0, + pseudo: 'Optional[IP_Pseudo]' = None, **kwargs: 'Any') -> 'Schema_TCP': """Make (construct) packet data. @@ -614,13 +628,18 @@ def make(self, syn: Synchronize sequence numbers. fin: Last packet from sender. window: Window size. - checksum: Checksum. + checksum: Checksum; if :data:`None` and + :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE` is on, + computed over ``pseudo`` (:rfc:`793#section-3.1`), or left to + an IP layer this segment is the payload of; zero otherwise. urgent: Urgent pointer. options: TCP options. payload: Payload of the packet. padding: Octets after the End of Option List option; dropped unless ``options`` end with one. reserved: Reserved bits 4-6 of octet 12. + pseudo: Pseudo-header of the enclosing IP layer; its length is + replaced by this segment's own. **kwargs: Arbitrary keyword arguments. Returns: @@ -683,7 +702,7 @@ def make(self, offset = math.ceil((TCP_MIN_HEADER_LEN + total_length) / 4) - return Schema_TCP( + schema = Schema_TCP( srcport=self._make_port(srcport, Enum_TransportProtocol.tcp), dstport=self._make_port(dstport, Enum_TransportProtocol.tcp), seq=seq_no, @@ -695,12 +714,22 @@ def make(self, }, flags=flags, window=window, - checksum=checksum, + checksum=b'\x00\x00' if checksum is None else checksum, urgent=urgent, options=options_value, payload=payload, ) + self._checksum_pending = False + if checksum is None and config.CHECKSUM_COMPUTE: + if pseudo is None: + self._checksum_pending = True + else: + from pcapkit.protocols.misc.checksum.checksum import compute_checksum, pseudo_header # isort: skip # pylint: disable=import-outside-toplevel + octets = schema.pack() + schema.checksum = compute_checksum(octets, prefix=pseudo_header(pseudo, len(octets))) + return schema + @classmethod def register_option(cls, code: 'Enum_Option', meth: 'str | tuple[OptionParser, OptionConstructor]') -> 'None': """Register an option parser. @@ -809,6 +838,29 @@ def _make_data(cls, data: 'Data_TCP') -> 'dict[str, Any]': # type: ignore[overr 'payload': cls._make_payload(data), } + def _verify_pseudo_checksum(self, pseudo: 'Optional[IP_Pseudo]') -> 'None': + """Verify the checksum over the pseudo-header of the enclosing IP layer. + + That layer calls this once parsed, see + :func:`~pcapkit.protocols.misc.checksum.checksum.verify_upper`. A zero + checksum, or the pseudo-header's sum alone, is what checksum offload + leaves on the sending host, and is + :attr:`~pcapkit.protocols.misc.checksum.checksum.ChecksumStatus.NOT_COMPUTED`. + + Args: + pseudo: Pseudo-header, or :data:`None` if the IP layer gives none. + + """ + from pcapkit.protocols.misc.checksum.checksum import ( # isort: skip # pylint: disable=import-outside-toplevel + ChecksumStatus, partial_checksum, verify_checksum) + + if pseudo is None: + status = ChecksumStatus.NO_PSEUDO_HEADER + else: + status = verify_checksum(self.alias, self._data, self._info.checksum, length=pseudo.info.len, + prefix=pseudo.data, offload=(b'\x00\x00', partial_checksum(pseudo.data))) + self._info.__update__(checksum_status=status) + def _read_tcp_options(self, size: 'int') -> 'Option': """Read TCP option list. diff --git a/pcapkit/protocols/transport/udp.py b/pcapkit/protocols/transport/udp.py index 744d32537b..968c3b8bff 100644 --- a/pcapkit/protocols/transport/udp.py +++ b/pcapkit/protocols/transport/udp.py @@ -32,6 +32,7 @@ from pcapkit.protocols.data.transport.udp import UDP as Data_UDP from pcapkit.protocols.schema.transport.udp import UDP as Schema_UDP from pcapkit.protocols.transport.transport import Transport +from pcapkit.utilities import config if TYPE_CHECKING: from typing import Any, ClassVar, Optional, Type @@ -39,6 +40,7 @@ from typing_extensions import Literal from pcapkit.corekit.enum import EnumRegistry + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo from pcapkit.protocols.protocol import ProtocolBase from pcapkit.protocols.schema.schema import Schema @@ -80,6 +82,13 @@ class UDP(Transport[Data_UDP, Schema_UDP], (:issue:`1155`), and ``trailer``, so that the octets past the Length field rebuild as captured (:issue:`1461`). + .. versionchanged:: 1.6.0 + + The checksum is computed when :meth:`make` is given none, over the + ``pseudo`` it is given or by the enclosing IP layer, and verified by that + layer under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY` + (:issue:`2474`). + """ ########################################################################## @@ -138,6 +147,10 @@ class UDP(Transport[Data_UDP, Schema_UDP], #: member carries one (:issue:`2268`). __proto_key__: 'ClassVar[tuple[Type[EnumRegistry], bool]]' = (Enum_AppType, True) + #: Whether :meth:`make` left the checksum for the enclosing IP layer to + #: compute, see :mod:`pcapkit.protocols.misc.checksum.checksum`. + _checksum_pending = False + ########################################################################## # Properties. ########################################################################## @@ -217,22 +230,28 @@ def read(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_UDP': def make(self, srcport: 'Enum_AppType | int' = 0, dstport: 'Enum_AppType | int' = 0, - checksum: 'bytes' = b'\x00\x00', + checksum: 'Optional[bytes]' = None, total_length: 'Optional[int]' = None, payload: 'bytes | Schema | ProtocolBase' = b'', trailer: 'bytes' = b'', + pseudo: 'Optional[IP_Pseudo]' = None, **kwargs: 'Any') -> 'Schema_UDP': """Make (construct) packet data. Args: srcport: Source port. dstport: Destination port. - checksum: Checksum. + checksum: Checksum; if :data:`None` and + :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE` is on, + computed over ``pseudo`` (:rfc:`768`), or left to an IP layer + this datagram is the payload of; zero, i.e. none, otherwise. total_length: Length of the datagram, header included; computed from the header and the payload when omitted. payload: Payload data. trailer: Octets after the datagram, outside the Length field; written as is and not counted in a computed ``total_length``. + pseudo: Pseudo-header of the enclosing IP layer; its length is + replaced by ``total_length``, as the Length field gives it. **kwargs: Arbitrary keyword arguments. Returns: @@ -259,15 +278,30 @@ def make(self, if total_length is None: total_length = 8 + len(payload) - return Schema_UDP( + schema = Schema_UDP( srcport=self._make_port(srcport, Enum_TransportProtocol.udp), dstport=self._make_port(dstport, Enum_TransportProtocol.udp), len=total_length, - checksum=checksum, + checksum=b'\x00\x00' if checksum is None else checksum, payload=payload, trailer=trailer, ) + self._checksum_pending = False + if checksum is None and config.CHECKSUM_COMPUTE: + if pseudo is None: + self._checksum_pending = True + else: + from pcapkit.protocols.misc.checksum.checksum import compute_checksum, pseudo_header # isort: skip # pylint: disable=import-outside-toplevel + + # NOTE: Over the datagram the Length field gives, or all but the trailer + # for a jumbogram's zero one (RFC 2675); a computed zero is all ones. + octets = schema.pack() + length = total_length or len(octets) - len(trailer) + schema.checksum = compute_checksum(octets[:length], nonzero=True, + prefix=pseudo_header(pseudo, length)) + return schema + ########################################################################## # Data models. ########################################################################## @@ -315,3 +349,30 @@ def _make_data(cls, data: 'Data_UDP') -> 'dict[str, Any]': # type: ignore[overr # NOTE: And the octets captured past the Length field (#1461). 'trailer': data.get('trailer', b''), } + + def _verify_pseudo_checksum(self, pseudo: 'Optional[IP_Pseudo]') -> 'None': + """Verify the checksum over the pseudo-header of the enclosing IP layer. + + That layer calls this once parsed, see + :func:`~pcapkit.protocols.misc.checksum.checksum.verify_upper`. A zero + checksum over IPv4, which means none (:rfc:`768`), and the + pseudo-header's sum alone, which checksum offload leaves on the sending + host, are + :attr:`~pcapkit.protocols.misc.checksum.checksum.ChecksumStatus.NOT_COMPUTED`. + A zero one over IPv6 is wrong, as one is mandatory there + (:rfc:`8200#section-8.1`). + + Args: + pseudo: Pseudo-header, or :data:`None` if the IP layer gives none. + + """ + from pcapkit.protocols.misc.checksum.checksum import ( # isort: skip # pylint: disable=import-outside-toplevel + ChecksumStatus, partial_checksum, verify_checksum) + + if pseudo is None: + status = ChecksumStatus.NO_PSEUDO_HEADER + else: + status = verify_checksum(self.alias, self._data, self._info.checksum, length=pseudo.info.len, + prefix=pseudo.data, absent=b'\x00\x00' if pseudo.version == 4 else None, + offload=(partial_checksum(pseudo.data),)) + self._info.__update__(checksum_status=status) diff --git a/pcapkit/utilities/config.py b/pcapkit/utilities/config.py index 9f91206830..724db1f934 100644 --- a/pcapkit/utilities/config.py +++ b/pcapkit/utilities/config.py @@ -5,8 +5,9 @@ .. module:: pcapkit.utilities.config :mod:`pcapkit.utilities.config` holds every tunable limit of :mod:`pcapkit` -(:issue:`1726`). Each limit is read from a ``PCAPKIT_*`` environment variable -once, when this module is imported, and validated there: +(:issue:`1726`), and its checksum flags (:issue:`2471`). Each is read from a +``PCAPKIT_*`` environment variable once, when this module is imported, and +validated there; a flag as :func:`read_flag` describes, and a limit as follows: * A value that is not an integer, or is below the variable's minimum, is reported with a :exc:`~pcapkit.utilities.warnings.ConfigWarning`, and the @@ -53,6 +54,8 @@ 'FRAME_LAYER_LIMIT', 'EXTENSION_HEADER_LIMIT', 'IPV6_TCP_WARNED_LIMIT', 'FALLBACK_LIMIT', 'DETECT_CACHE_SIZE', 'READ_CHUNK', 'MAX_ZERO_PAD_LENGTH', 'MAX_ZERO_PAD_SHORTFALL', 'ZERO_PAD_BUDGET_RATIO', + # flags + 'CHECKSUM_VERIFY', 'CHECKSUM_COMPUTE', # readers 'read_limit', 'read_flag', ] @@ -380,3 +383,29 @@ def read_flag(name: 'str', default: 'bool' = False) -> 'bool': ZERO_PAD_BUDGET_RATIO = read_limit('ZERO_PAD_BUDGET_RATIO', 0x10, risk=( 'a capture whose blocks all lie about their lengths can amplify each octet read ' 'into that many zero octets in memory')) + +############################################################################### +# Flags +############################################################################### + +#: bool: Whether parsing verifies the one's-complement checksums, of the IPv4 +#: header, TCP, UDP, ICMP, ICMPv6 and OSPF, :envvar:`PCAPKIT_CHECKSUM_VERIFY`. +#: +#: Off by default, since checksumming every packet costs time on a large +#: capture (:issue:`2471`). When on, each records a +#: :class:`~pcapkit.protocols.misc.checksum.checksum.ChecksumStatus` as +#: ``checksum_status`` in its info, and a wrong checksum warns with a +#: :exc:`~pcapkit.utilities.warnings.ChecksumWarning` rather than raising. +#: +#: .. versionadded:: 1.6.0 +CHECKSUM_VERIFY = read_flag('CHECKSUM_VERIFY', False) + +#: bool: Whether construction computes a checksum the caller leaves out, for +#: the protocols :data:`CHECKSUM_VERIFY` names, :envvar:`PCAPKIT_CHECKSUM_COMPUTE`. +#: +#: On by default (:issue:`2471`). A checksum given explicitly, as +#: :meth:`~pcapkit.protocols.protocol.Protocol.from_data` gives the parsed one, +#: is written as given either way. Off, a checksum left out is zero. +#: +#: .. versionadded:: 1.6.0 +CHECKSUM_COMPUTE = read_flag('CHECKSUM_COMPUTE', True) diff --git a/pcapkit/utilities/warnings.py b/pcapkit/utilities/warnings.py index 9a851733d5..c5fa2c12d5 100644 --- a/pcapkit/utilities/warnings.py +++ b/pcapkit/utilities/warnings.py @@ -89,7 +89,7 @@ # ImportWarning 'FormatWarning', 'EngineWarning', 'InvalidVendorWarning', # RuntimeWarning - 'FileWarning', 'LayerWarning', 'ProtocolWarning', 'AttributeWarning', + 'FileWarning', 'LayerWarning', 'ProtocolWarning', 'ChecksumWarning', 'AttributeWarning', 'DevModeWarning', 'VendorRequestWarning', 'VendorRuntimeWarning', 'UnknownFieldWarning', 'RegistryWarning', 'SchemaWarning', 'InfoWarning', 'SeekWarning', 'ExtractionWarning', 'ConfigWarning', @@ -380,6 +380,17 @@ class ProtocolWarning(BaseWarning, RuntimeWarning): """Unrecognised protocol.""" +class ChecksumWarning(ProtocolWarning): + """Checksum mismatch. + + Reported when parsing verifies a checksum and finds it wrong, so that a + filter can silence it without silencing other protocol warnings. + + .. versionadded:: 1.6.0 + + """ + + class AttributeWarning(BaseWarning, RuntimeWarning): """Unsupported attribute.""" diff --git a/tests/project/test_rfc_constants_one_home_1739_unit.py b/tests/project/test_rfc_constants_one_home_1739_unit.py index a349ab8595..26806dc3d4 100644 --- a/tests/project/test_rfc_constants_one_home_1739_unit.py +++ b/tests/project/test_rfc_constants_one_home_1739_unit.py @@ -397,7 +397,11 @@ 'pcapkit/protocols/schema/internet/ipv6_option.py': ((16,), 'an MPL Seed-ID of 16 octets'), 'pcapkit/protocols/schema/application/httpv2.py': ((6, 32), 'an HTTP/2 setting size and a flag bit'), - 'pcapkit/protocols/application/ospf.py': ((24, 65535), 'OSPF header length and field bounds'), + 'pcapkit/protocols/application/ospf.py': ((16, 24, 65535), + 'OSPF header length and field bounds; 16 is the ' + 'offset of the Authentication field the checksum ' + 'leaves out (RFC 2328 appendix A.3.1), not a PCAP ' + 'record header'), 'pcapkit/protocols/schema/application/ospf.py': ((24,), 'the OSPF header length'), 'pcapkit/protocols/internet/esp.py': ((12, 16, 20, 24, 32), 'ESP ICV and key sizes'), 'pcapkit/protocols/internet/igmpv3.py': ((16,), "the floating-point Max Resp Code/QQIC " @@ -456,6 +460,10 @@ 'pcapkit/protocols/schema/link/l2tp.py': ((6, 12), 'an L2TP header field offset, and the ' 'version bit-field'), 'pcapkit/protocols/schema/link/vlan.py': ((12,), 'the 12-bit VLAN identifier'), + 'pcapkit/protocols/misc/checksum/checksum.py': ((16, 65535), + "the one's-complement sum's 16-bit word mask " + 'and carry shift (RFC 1071), not an IP total ' + 'length or a PCAP record header'), 'pcapkit/protocols/misc/pcapng.py': ((65535,), 'a 16-bit mask'), 'pcapkit/protocols/schema/misc/pcapng.py': ((9, 65535), 'an epb_flags bit offset, and a ' '16-bit field default'), diff --git a/tests/protocols/application/test_ospf_checksum_2475_unit.py b/tests/protocols/application/test_ospf_checksum_2475_unit.py new file mode 100644 index 0000000000..08bb92dcb2 --- /dev/null +++ b/tests/protocols/application/test_ospf_checksum_2475_unit.py @@ -0,0 +1,140 @@ +# -*- coding: utf-8 -*- +"""OSPFv2 verifies and computes its checksum. + +GitHub issue #2475: :meth:`OSPF.read ` +verifies the OSPFv2 checksum under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`, +and :meth:`OSPF.make ` computes one it +is not given under :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE`. It covers the +packet but its Authentication field, with no pseudo-header +(:rfc:`2328#appendix-A.3.1`), and is unused under cryptographic authentication +(:rfc:`2328#appendix-D.4.3`). The packets were built with ``scapy.contrib.ospf`` +2.7.0, and are cross-checked with the hand-written sum of +``tests/foundation/_roundtrip.py``. + +""" +from __future__ import annotations + +import importlib.util +import unittest +import warnings +from unittest import mock + +from tests._support import reimport_once_per_class +from tests.foundation import _roundtrip as wire + +#: Test tier (#1958): hand-picked parse -> rebuild and make -> parse round trips. +__tier__ = 'ALL' + +#: An OSPFv2 Hello body: mask, interval, options, priority, dead interval, DR, +#: BDR and one neighbour. +HELLO = bytes.fromhex('ffffff00000a020100000028c000020100000000c0000202') + +#: Hello packets as scapy builds them, with null and with simple password +#: authentication; only the AuType differs in what the checksum covers. +BUILT = ( + bytes.fromhex('02010030c000020100000000b69400000000000000000000') + HELLO, + bytes.fromhex('02010030c000020100000000b69300017365637265742121') + HELLO, +) + +HAS_SCAPY = importlib.util.find_spec('scapy') is not None + + +def expected(packet: bytes) -> bytes: + """The checksum ``packet`` should carry: over all but the Authentication field.""" + return wire.checksum(packet[:12] + b'\x00\x00' + packet[14:16] + packet[24:]).to_bytes(2, 'big') + + +def with_checksum(packet: bytes, field: bytes) -> bytes: + """``packet`` with ``field`` as its checksum.""" + return packet[:12] + field + packet[14:] + + +class OSPFChecksumTests(unittest.TestCase): + """Reading and making the OSPFv2 checksum.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + from pcapkit.protocols.application.ospf import OSPF + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus + from pcapkit.utilities import config + from pcapkit.utilities.warnings import ChecksumWarning, ProtocolWarning + + self.OSPF = OSPF + self.Status = ChecksumStatus + self.config = config + self.ProtocolWarning = ProtocolWarning + self.ChecksumWarning = ChecksumWarning + + def parse(self, octets: bytes, *, verify: bool = True) -> 'tuple[object, list[tuple[type, str]]]': + """The OSPF packet parsed from ``octets``, and its ProtocolWarnings as ``(category, message)``.""" + with mock.patch.object(self.config, 'CHECKSUM_VERIFY', verify), \ + warnings.catch_warnings(record=True) as caught: + warnings.simplefilter('always') + ospf = self.OSPF(octets) + return ospf, [(item.category, str(item.message)) + for item in caught if issubclass(item.category, self.ProtocolWarning)] + + def status(self, octets: bytes, **kwargs: object) -> 'tuple[object, list[tuple[type, str]]]': + """The checksum status of ``octets``, and its ProtocolWarnings as ``(category, message)``.""" + ospf, messages = self.parse(octets, **kwargs) + return ospf.info.get('checksum_status'), messages + + def test_built_checksums_verify(self) -> None: + for packet in BUILT: + with self.subTest(packet=packet.hex()): + self.assertEqual(expected(packet), packet[12:14]) + self.assertEqual(self.status(packet), (self.Status.VALID, [])) + + def test_a_corrupted_checksum_warns(self) -> None: + status, messages = self.status(with_checksum(BUILT[0], b'\xb6\x95')) + self.assertIs(status, self.Status.INVALID) + self.assertEqual(messages, [(self.ChecksumWarning, 'OSPFv2: checksum 0xb695 is wrong; 0xb694 expected')]) + + def test_cryptographic_authentication_leaves_it_unused(self) -> None: + packet = BUILT[0][:14] + b'\x00\x02' + bytes([0, 0, 1, 16]) + bytes(4) + HELLO + self.assertEqual(self.status(packet), (self.Status.NOT_COMPUTED, [])) + made = self.OSPF(type='Hello', router_id='192.0.2.1', auth_type=2, + auth_data={'key_id': 1, 'len': 16, 'seq': 0}, payload=HELLO).data + self.assertEqual(made[12:14], b'\x00\x00') + + def test_truncated_and_other_versions(self) -> None: + self.assertEqual(self.status(BUILT[0][:-4]), (self.Status.TRUNCATED, [])) + self.assertEqual(self.status(b'\x03' + BUILT[0][1:]), (None, [])) + self.assertEqual(self.OSPF(version=3, payload=HELLO).data[12:14], b'\x00\x00') + + def test_off_verifies_nothing(self) -> None: + self.assertEqual(self.status(with_checksum(BUILT[0], b'\xb6\x95'), verify=False), (None, [])) + + def test_parse_rebuild_is_exact(self) -> None: + for packet in (*BUILT, with_checksum(BUILT[0], b'\xb6\x95'), BUILT[0][:-4]): + for verify in (True, False): + with self.subTest(packet=packet.hex(), verify=verify): + ospf, _ = self.parse(packet, verify=verify) + self.assertEqual(self.OSPF.from_data(ospf.info).data, packet) + + def test_make_computes_the_checksum(self) -> None: + for packet, kwargs in zip(BUILT, ({}, {'auth_type': 1, 'auth_data': b'secret!!'})): + with self.subTest(kwargs=kwargs): + made = self.OSPF(type='Hello', router_id='192.0.2.1', payload=HELLO, **kwargs).data + self.assertEqual(made, packet) + ospf, messages = self.parse(made) + self.assertEqual((ospf.info.checksum_status, messages), (self.Status.VALID, [])) + self.assertEqual(self.OSPF.from_data(ospf.info).data, made) + + @unittest.skipUnless(HAS_SCAPY, 'scapy not installed') + def test_make_agrees_with_scapy(self) -> None: + from scapy.contrib.ospf import OSPF_Hdr + + made = self.OSPF(type='Hello', router_id='192.0.2.1', payload=HELLO).data + packet = OSPF_Hdr(made) + del packet.chksum + self.assertEqual(bytes(packet), made) + + def test_off_or_given_is_written_as_is(self) -> None: + with mock.patch.object(self.config, 'CHECKSUM_COMPUTE', False): + self.assertEqual(self.OSPF(payload=HELLO).data[12:14], b'\x00\x00') + self.assertEqual(self.OSPF(checksum=b'\xbe\xef', payload=HELLO).data[12:14], b'\xbe\xef') + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/protocols/internet/test_icmp_checksum_2470_unit.py b/tests/protocols/internet/test_icmp_checksum_2470_unit.py new file mode 100644 index 0000000000..25003a6a06 --- /dev/null +++ b/tests/protocols/internet/test_icmp_checksum_2470_unit.py @@ -0,0 +1,234 @@ +# -*- coding: utf-8 -*- +"""ICMP and ICMPv6 verify and compute their checksums. + +GitHub issue #2470: :meth:`ICMPv4.read ` +verifies the checksum over the message alone (:rfc:`792`), and the enclosing IPv6 +layer verifies the ICMPv6 checksum over its pseudo-header (:rfc:`4443#section-2.3`), +both under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`; ``make()`` computes one +it is not given under :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE`. The ICMPv6 +messages are from the tracked :file:`examples/captures/in.pcap`; expected checksums +are the hand-written sum of ``tests/foundation/_roundtrip.py``. + +""" +from __future__ import annotations + +import importlib.util +import unittest +import warnings +from unittest import mock + +from tests._support import reimport_once_per_class +from tests.foundation import _roundtrip as wire + +#: Test tier (#1958): hand-picked parse -> rebuild and make -> parse round trips. +__tier__ = 'ALL' + +#: An IPv4 packet carrying an ICMP Echo request, with its sender's checksum. +ECHO = bytes.fromhex('4500003c4e2100004001db730a141e830a141e820800083a3f210001211ba46a081f6a04' + '9680ef6445e70f48a870edac63255fe850848af33d4c2bf6') + +#: IPv6 packets carrying ICMPv6 as captured (``in.pcap`` frames 0 and 1): a +#: Neighbor Solicitation and its Advertisement. +CAPTURED = ( + bytes.fromhex('6000000000203afffe8000000000000000a687f9279316eefe800000000000001ccd7c77bac746b7' + '87000eaa00000000fe800000000000001ccd7c77bac746b70101a45e60d96b97'), + bytes.fromhex('6000000000183afffe800000000000001ccd7c77bac746b7fe8000000000000000a687f9279316ee' + '88003f8240000000fe800000000000001ccd7c77bac746b7'), +) + +HAS_SCAPY = importlib.util.find_spec('scapy') is not None + + +def pseudo6(packet: bytes, length: int) -> bytes: + """The ICMPv6 pseudo-header of ``packet``'s addresses (RFC 8200 section 8.1).""" + return packet[8:40] + length.to_bytes(4, 'big') + b'\x00\x00\x00\x3a' + + +def expected(prefix: bytes, message: bytes) -> bytes: + """The checksum ``message`` should carry over ``prefix``.""" + return wire.checksum(prefix + message[:2] + b'\x00\x00' + message[4:]).to_bytes(2, 'big') + + +def with_checksum(packet: bytes, field: bytes, at: int) -> bytes: + """``packet`` with ``field`` at ``at``.""" + return packet[:at] + field + packet[at + 2:] + + +class ChecksumTestCase(unittest.TestCase): + """Imports and helpers both protocols' tests share.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + from pcapkit.protocols.internet.icmpv4 import ICMPv4 + from pcapkit.protocols.internet.icmpv6 import ICMPv6 + from pcapkit.protocols.internet.ipv4 import IPv4 + from pcapkit.protocols.internet.ipv6 import IPv6 + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo + from pcapkit.utilities import config + from pcapkit.utilities.warnings import ChecksumWarning, ProtocolWarning + + self.ICMPv4, self.ICMPv6, self.IPv4, self.IPv6, self.IP_Pseudo = ICMPv4, ICMPv6, IPv4, IPv6, IP_Pseudo + self.Status = ChecksumStatus + self.config = config + self.ProtocolWarning = ProtocolWarning + self.ChecksumWarning = ChecksumWarning + + def parse(self, octets: bytes, cls: type, *, verify: bool = True) -> 'tuple[object, list[tuple[type, str]]]': + """``octets`` parsed as ``cls``, and its ProtocolWarnings as ``(category, message)``.""" + with mock.patch.object(self.config, 'CHECKSUM_VERIFY', verify), \ + warnings.catch_warnings(record=True) as caught: + warnings.simplefilter('always') + layer = cls(octets) + return layer, [(item.category, str(item.message)) + for item in caught if issubclass(item.category, self.ProtocolWarning)] + + +class ICMPv4ChecksumTests(ChecksumTestCase): + """Reading and making the ICMP checksum.""" + + def status(self, octets: bytes, **kwargs: object) -> 'tuple[object, list[tuple[type, str]]]': + """The ICMP checksum status of the IPv4 packet ``octets``, and its ProtocolWarnings.""" + layer, messages = self.parse(octets, self.IPv4, **kwargs) + return layer['ICMP'].info.get('checksum_status'), messages + + def test_the_captured_checksum_verifies(self) -> None: + self.assertEqual(expected(b'', ECHO[20:]), ECHO[22:24]) + self.assertEqual(self.status(ECHO), (self.Status.VALID, [])) + icmp, messages = self.parse(ECHO[20:], self.ICMPv4) + self.assertEqual((icmp.info.checksum_status, messages), (self.Status.VALID, [])) + + def test_a_corrupted_checksum_warns(self) -> None: + status, messages = self.status(with_checksum(ECHO, b'\x08\x3b', 22)) + self.assertIs(status, self.Status.INVALID) + self.assertEqual(messages, [(self.ChecksumWarning, 'ICMP: checksum 0x083b is wrong; 0x083a expected')]) + + def test_truncated_and_off(self) -> None: + self.assertEqual(self.status(ECHO[:-1]), (self.Status.TRUNCATED, [])) + self.assertEqual(self.status(with_checksum(ECHO, b'\x08\x3b', 22), verify=False), (None, [])) + + def test_a_fragment_is_not_verified(self) -> None: + # The first 32 of the Echo's 40 octets, More Fragments set: the checksum + # covers all 40, so the fragment cannot be checked, as TCP's cannot. + message = ECHO[20:] + packets = { + 'IPv4': (self.IPv4, wire.ipv4(message[:32], mf=True, proto=1)), + 'behind AH': (self.IPv4, wire.ipv4(bytes([1, 4]) + bytes(22) + message[:32], mf=True, proto=51)), + 'IPv6': (self.IPv6, wire.ipv6_fragment(message[:32], mf=True, nxt=1)), + } + for name, (cls, packet) in packets.items(): + with self.subTest(name): + layer, messages = self.parse(packet, cls) + self.assertEqual((layer['ICMP'].info.checksum_status, messages), + (self.Status.NO_PSEUDO_HEADER, [])) + self.assertEqual(cls.from_data(layer.info).data, packet) + # An atomic fragment is the whole datagram. + layer, messages = self.parse(wire.ipv6_fragment(message, nxt=1), self.IPv6) + self.assertEqual((layer['ICMP'].info.checksum_status, messages), (self.Status.VALID, [])) + + def test_parse_rebuild_is_exact(self) -> None: + for packet in (ECHO, with_checksum(ECHO, b'\x08\x3b', 22), with_checksum(ECHO, b'\x00\x00', 22)): + for verify in (True, False): + with self.subTest(packet=packet.hex(), verify=verify): + layer, _ = self.parse(packet, self.IPv4, verify=verify) + self.assertEqual(self.IPv4.from_data(layer.info).data, packet) + + def test_make_computes_the_checksum(self) -> None: + body = {'id': 0x3f21, 'seq': 1, 'data': b'odd'} + made = self.ICMPv4(type='Echo', body=body).data + self.assertEqual(made[2:4], expected(b'', made)) + icmp, messages = self.parse(made, self.ICMPv4) + self.assertEqual((icmp.info.checksum_status, messages), (self.Status.VALID, [])) + self.assertEqual(self.ICMPv4.from_data(icmp.info).data, made) + with mock.patch.object(self.config, 'CHECKSUM_COMPUTE', False): + self.assertEqual(self.ICMPv4(type='Echo', body=body).data[2:4], b'\x00\x00') + self.assertEqual(self.ICMPv4(type='Echo', body=body, checksum=b'\xbe\xef').data[2:4], b'\xbe\xef') + + @unittest.skipUnless(HAS_SCAPY, 'scapy not installed') + def test_make_agrees_with_scapy(self) -> None: + from scapy.layers.inet import ICMP + + made = self.ICMPv4(type='Echo', body={'id': 7, 'seq': 2, 'data': b'ping'}).data + packet = ICMP(made) + del packet.chksum + self.assertEqual(bytes(packet), made) + + +class ICMPv6ChecksumTests(ChecksumTestCase): + """Verifying and computing the ICMPv6 checksum.""" + + def status(self, octets: bytes, **kwargs: object) -> 'tuple[object, list[tuple[type, str]]]': + """The ICMPv6 checksum status of the IPv6 packet ``octets``, and its ProtocolWarnings.""" + layer, messages = self.parse(octets, self.IPv6, **kwargs) + return layer['ICMPv6'].info.get('checksum_status'), messages + + def test_captured_checksums_verify(self) -> None: + for packet in CAPTURED: + with self.subTest(packet=packet.hex()): + self.assertEqual(expected(pseudo6(packet, len(packet) - 40), packet[40:]), packet[42:44]) + self.assertEqual(self.status(packet), (self.Status.VALID, [])) + + def test_a_corrupted_checksum_warns(self) -> None: + status, messages = self.status(with_checksum(CAPTURED[0], b'\x0e\xab', 42)) + self.assertIs(status, self.Status.INVALID) + self.assertEqual(messages, [(self.ChecksumWarning, 'ICMPv6: checksum 0x0eab is wrong; 0x0eaa expected')]) + + def test_off_or_alone_verifies_nothing(self) -> None: + self.assertEqual(self.status(with_checksum(CAPTURED[0], b'\x0e\xab', 42), verify=False), (None, [])) + icmp, _ = self.parse(CAPTURED[0][40:], self.ICMPv6) + self.assertNotIn('checksum_status', icmp.info) + + def test_a_first_fragment_has_no_pseudo_header(self) -> None: + packet = wire.ipv6_fragment(CAPTURED[0][40:], mf=True, nxt=58) + self.assertEqual(self.status(packet), (self.Status.NO_PSEUDO_HEADER, [])) + + def test_behind_an_extension_header(self) -> None: + # A Hop-by-Hop Options header with one PadN; the pseudo-header names ICMPv6. + message = CAPTURED[0][40:] + packet = CAPTURED[0][:4] + (len(message) + 8).to_bytes(2, 'big') + b'\x00' + CAPTURED[0][7:40] \ + + b'\x3a\x00\x01\x04\x00\x00\x00\x00' + message + self.assertEqual(self.status(packet), (self.Status.VALID, [])) + + def test_parse_rebuild_is_exact(self) -> None: + for packet in (*CAPTURED, with_checksum(CAPTURED[0], b'\x0e\xab', 42)): + for verify in (True, False): + with self.subTest(packet=packet.hex(), verify=verify): + layer, _ = self.parse(packet, self.IPv6, verify=verify) + self.assertEqual(self.IPv6.from_data(layer.info).data, packet) + + def make(self, **kwargs: object) -> bytes: + """The octets of an IPv6 packet carrying an Echo Request made with ``kwargs``.""" + icmp = self.ICMPv6(type='Echo_Request', body={'id': 7, 'seq': 2, 'data': b'odd'}, **kwargs) + return self.IPv6(src='2001:db8::1', dst='2001:db8::2', next='IPv6_ICMP', payload=icmp).data + + def test_the_ipv6_layer_computes_it(self) -> None: + made = self.make() + self.assertEqual(made[42:44], expected(pseudo6(made, 11), made[40:])) + layer, messages = self.parse(made, self.IPv6) + self.assertEqual((layer['ICMPv6'].info.checksum_status, messages), (self.Status.VALID, [])) + self.assertEqual(self.IPv6.from_data(layer.info).data, made) + + def test_make_takes_the_pseudo_header(self) -> None: + pseudo = self.IP_Pseudo(src='2001:db8::1', dst='2001:db8::2', protocol='IPv6_ICMP', upper_length=0) + message = self.ICMPv6(type='Echo_Request', body={'id': 7, 'seq': 2, 'data': b'odd'}, pseudo=pseudo).data + self.assertEqual(message, self.make()[40:]) + # Alone, with no pseudo-header to compute it over, it is left zero. + self.assertEqual(self.ICMPv6(type='Echo_Request').data[2:4], b'\x00\x00') + + @unittest.skipUnless(HAS_SCAPY, 'scapy not installed') + def test_make_agrees_with_scapy(self) -> None: + from scapy.layers.inet6 import IPv6 + + made = self.make() + packet = IPv6(made) + del packet.payload.cksum + self.assertEqual(bytes(packet), made) + + def test_off_or_given_is_written_as_is(self) -> None: + with mock.patch.object(self.config, 'CHECKSUM_COMPUTE', False): + self.assertEqual(self.make()[42:44], b'\x00\x00') + self.assertEqual(self.make(checksum=b'\xbe\xef')[42:44], b'\xbe\xef') + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/protocols/internet/test_icmpv4_roundtrip_unit.py b/tests/protocols/internet/test_icmpv4_roundtrip_unit.py index 46cbf94f52..03c93e536c 100644 --- a/tests/protocols/internet/test_icmpv4_roundtrip_unit.py +++ b/tests/protocols/internet/test_icmpv4_roundtrip_unit.py @@ -124,10 +124,9 @@ def test_construct_with_no_arguments(self) -> None: from pcapkit.protocols.internet.icmpv4 import ICMPv4 proto = ICMPv4() - # Type 8 (Echo), code 0, an uncomputed (zero) RFC 792 checksum over - # an all-zero 4-octet id/seq body, then that same body: this class - # does not compute or verify the checksum (#2470). - self.assertEqual(bytes(proto), bytes.fromhex('0800000000000000')) + # Type 8 (Echo), code 0, the RFC 792 checksum computed over an all-zero + # 4-octet id/seq body, then that same body (#2470). + self.assertEqual(bytes(proto), bytes.fromhex('0800f7ff00000000')) def test_dispatches_from_ipv4(self) -> None: """IPv4's protocol-number dispatch reaches :class:`ICMPv4`.""" diff --git a/tests/protocols/internet/test_ipv4_checksum_2472_unit.py b/tests/protocols/internet/test_ipv4_checksum_2472_unit.py new file mode 100644 index 0000000000..a838f8e1bf --- /dev/null +++ b/tests/protocols/internet/test_ipv4_checksum_2472_unit.py @@ -0,0 +1,142 @@ +# -*- coding: utf-8 -*- +"""IPv4 verifies and computes its header checksum. + +GitHub issue #2472: :meth:`IPv4.read ` +verifies the header checksum (:rfc:`791#section-3.1`) under +:data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`, and :meth:`IPv4.make +` computes one it is not given under +:data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE`. The captured headers are from +the tracked :file:`examples/captures/in.pcap` and :file:`examples/captures/dhcp.pcapng`. + +""" +from __future__ import annotations + +import importlib.util +import unittest +import warnings +from unittest import mock + +from tests._support import reimport_once_per_class +from tests.foundation import _roundtrip as wire + +#: Test tier (#1958): hand-picked parse -> rebuild and make -> parse round trips. +__tier__ = 'ALL' + +#: IPv4 headers as captured: TCP from a remote host (``in.pcap`` frame 2) and a +#: DHCP Discover (``dhcp.pcapng`` frame 0), each with the sender's checksum. +CAPTURED = ( + bytes.fromhex('45000028c53440002b067a867b81d287c0a80164'), + bytes.fromhex('4500012ca8360000fa11178b00000000ffffffff'), +) + +HAS_SCAPY = importlib.util.find_spec('scapy') is not None + + +def with_checksum(header: bytes, field: bytes) -> bytes: + """``header`` with ``field`` as its checksum.""" + return header[:10] + field + header[12:] + + +class IPv4ChecksumTests(unittest.TestCase): + """Reading and making the IPv4 header checksum.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + from pcapkit.protocols.internet.ipv4 import IPv4 + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus + from pcapkit.utilities import config + from pcapkit.utilities.warnings import ChecksumWarning, ProtocolWarning + + self.IPv4 = IPv4 + self.Status = ChecksumStatus + self.config = config + self.ProtocolWarning = ProtocolWarning + self.ChecksumWarning = ChecksumWarning + + def parse(self, octets: bytes, *, verify: bool = True) -> 'tuple[object, list[tuple[type, str]]]': + """The IPv4 layer parsed from ``octets``, and its ProtocolWarnings as ``(category, message)``.""" + with mock.patch.object(self.config, 'CHECKSUM_VERIFY', verify), \ + warnings.catch_warnings(record=True) as caught: + warnings.simplefilter('always') + ipv4 = self.IPv4(octets) + return ipv4, [(item.category, str(item.message)) + for item in caught if issubclass(item.category, self.ProtocolWarning)] + + def test_captured_checksums_verify(self) -> None: + for header in CAPTURED: + with self.subTest(header=header.hex()): + ipv4, messages = self.parse(header) + self.assertIs(ipv4.info.checksum_status, self.Status.VALID) + self.assertEqual(messages, []) + + def test_a_corrupted_checksum_warns(self) -> None: + header = with_checksum(CAPTURED[0], b'\x7a\x87') + ipv4, messages = self.parse(header) + self.assertIs(ipv4.info.checksum_status, self.Status.INVALID) + self.assertEqual(messages, [(self.ChecksumWarning, 'IPv4: checksum 0x7a87 is wrong; 0x7a86 expected')]) + + def test_a_zero_checksum_is_offload(self) -> None: + ipv4, messages = self.parse(with_checksum(CAPTURED[0], b'\x00\x00')) + self.assertIs(ipv4.info.checksum_status, self.Status.NOT_COMPUTED) + self.assertEqual(messages, []) + + def test_off_verifies_nothing(self) -> None: + ipv4, messages = self.parse(with_checksum(CAPTURED[0], b'\x7a\x87'), verify=False) + self.assertNotIn('checksum_status', ipv4.info) + self.assertEqual(messages, []) + + def test_a_rebuilt_frame_warns_once(self) -> None: + from pcapkit.protocols.link.ethernet import Ethernet + + frame = wire.ethernet(with_checksum(CAPTURED[0], b'\x7a\x87'), 0x0800) + with mock.patch.object(self.config, 'CHECKSUM_VERIFY', True), \ + warnings.catch_warnings(record=True) as caught: + warnings.simplefilter('always') + rebuilt = Ethernet.from_data(Ethernet(frame).info) + self.assertEqual(rebuilt.data, frame) + self.assertEqual([(item.category, str(item.message)) + for item in caught if issubclass(item.category, self.ProtocolWarning)], + [(self.ChecksumWarning, 'IPv4: checksum 0x7a87 is wrong; 0x7a86 expected')] * 2) + + def test_parse_rebuild_is_exact(self) -> None: + for header in (*CAPTURED, with_checksum(CAPTURED[0], b'\x7a\x87'), with_checksum(CAPTURED[0], b'\x00\x00')): + for verify in (True, False): + with self.subTest(header=header.hex(), verify=verify): + ipv4, _ = self.parse(header, verify=verify) + self.assertEqual(self.IPv4.from_data(ipv4.info).data, header) + + def make(self, compute: bool = True, **kwargs: object) -> bytes: + """The octets of an IPv4 header made with ``kwargs``.""" + with mock.patch.object(self.config, 'CHECKSUM_COMPUTE', compute): + return self.IPv4(src='192.0.2.1', dst='198.51.100.2', ttl=64, id=0x1234, + protocol='UDP', **kwargs).data + + def test_make_computes_the_checksum(self) -> None: + from pcapkit.const.ipv4.option_number import OptionNumber + + for options in (None, [(OptionNumber.RR, {'counts': 2, 'route': ['203.0.113.9']})]): + with self.subTest(options=options): + octets = self.make(options=options, payload=b'\x00' * 8) + header = octets[:(octets[0] & 0x0F) * 4] + self.assertEqual(header[10:12], wire.checksum(with_checksum(header, b'\x00\x00')).to_bytes(2, 'big')) + ipv4, messages = self.parse(octets) + self.assertIs(ipv4.info.checksum_status, self.Status.VALID) + self.assertEqual(messages, []) + self.assertEqual(self.IPv4.from_data(ipv4.info).data, octets) + + @unittest.skipUnless(HAS_SCAPY, 'scapy not installed') + def test_make_agrees_with_scapy(self) -> None: + from scapy.layers.inet import IP + + octets = self.make(payload=b'\x00' * 8) + packet = IP(octets) + del packet.chksum + self.assertEqual(bytes(packet), octets) + + def test_off_or_given_is_written_as_is(self) -> None: + self.assertEqual(self.make(compute=False)[10:12], b'\x00\x00') + self.assertEqual(self.make(checksum=b'\xbe\xef')[10:12], b'\xbe\xef') + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/protocols/internet/test_ipv4_unit.py b/tests/protocols/internet/test_ipv4_unit.py index 38eb742e7c..bec91ebd66 100644 --- a/tests/protocols/internet/test_ipv4_unit.py +++ b/tests/protocols/internet/test_ipv4_unit.py @@ -355,7 +355,9 @@ def test_schema_pack_packs_a_second_real_protocol_as_payload(self) -> None: with warnings.catch_warnings(): warnings.simplefilter('ignore') - udp = UDP(srcport=1234, dstport=53, payload=b'\xbb' * 8) + # NOTE: Explicit checksums, which the IP layer writes as given rather + # than computing them over its pseudo-header (#2473, #2474). + udp = UDP(srcport=1234, dstport=53, payload=b'\xbb' * 8, checksum=b'\x00\x00') udp_octets = bytes(udp) # 8 octets of UDP header and the 8 the caller gave it @@ -393,7 +395,7 @@ def test_schema_pack_packs_a_second_real_protocol_as_payload(self) -> None: with warnings.catch_warnings(): warnings.simplefilter('ignore') others = [ - TCP(srcport=1234, dstport=80, payload=b'\xcc' * 8), + TCP(srcport=1234, dstport=80, payload=b'\xcc' * 8, checksum=b'\x00\x00'), IPv4(protocol=6, src='192.0.2.1', dst='198.51.100.1', payload=b'\xcc' * 8), ] diff --git a/tests/protocols/misc/checksum/test_internet_checksum_2471_unit.py b/tests/protocols/misc/checksum/test_internet_checksum_2471_unit.py new file mode 100644 index 0000000000..6faba98ab7 --- /dev/null +++ b/tests/protocols/misc/checksum/test_internet_checksum_2471_unit.py @@ -0,0 +1,152 @@ +# -*- coding: utf-8 -*- +"""The one's-complement Internet checksum the IP family's checksums share. + +GitHub issue #2471 settled the policy: verification is opt-in and computation +opt-out, each a ``PCAPKIT_*`` flag; a wrong checksum warns rather than raises; +and the outcome is recorded as a +:class:`~pcapkit.protocols.misc.checksum.checksum.ChecksumStatus`. These tests pin +:mod:`pcapkit.protocols.misc.checksum.checksum` against :rfc:`1071`, cross-checked +with the hand-written sum of ``tests/foundation/_roundtrip.py``. + +""" +from __future__ import annotations + +import enum +import random +import unittest +import warnings + +from tests._support import reimport_once_per_class +from tests.foundation import _roundtrip as wire + +#: Test tier (#1958): exercises bytes and struct edge cases, odd lengths and folding. +__tier__ = 'ALL' + + +class InternetChecksumTests(unittest.TestCase): + """:func:`internet_checksum` and :func:`compute_checksum`.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + from pcapkit.protocols.misc.checksum import checksum + + self.checksum = checksum + + def test_rfc_1071_example(self) -> None: + # RFC 1071 section 3: the words sum to 0xddf2. + self.assertEqual(self.checksum.internet_checksum(bytes.fromhex('0001f203f4f5f6f7')), ~0xDDF2 & 0xFFFF) + + def test_odd_length_is_padded(self) -> None: + self.assertEqual(self.checksum.internet_checksum(b'\x01'), ~0x0100 & 0xFFFF) + self.assertEqual(self.checksum.internet_checksum(b''), 0xFFFF) + + def test_agrees_with_an_independent_sum(self) -> None: + rng = random.Random(2471) + for size in (1, 2, 3, 20, 41, 1500, 65535): + data = bytes(rng.randrange(256) for _ in range(size)) + with self.subTest(size=size): + self.assertEqual(self.checksum.internet_checksum(data), wire.checksum(data)) + self.assertEqual(self.checksum.compute_checksum(data[2:], prefix=data[:2]), + wire.checksum(data).to_bytes(2, 'big')) + + def test_a_computed_zero(self) -> None: + self.assertEqual(self.checksum.compute_checksum(b'\xff\xff'), b'\x00\x00') + self.assertEqual(self.checksum.compute_checksum(b'\xff\xff', nonzero=True), b'\xff\xff') + + def test_the_partial_checksum_is_the_uncomplemented_sum(self) -> None: + prefix = bytes.fromhex('c0000201' 'c6336402' '00' '06' '0014') + self.assertEqual(self.checksum.partial_checksum(prefix), (~wire.checksum(prefix) & 0xFFFF).to_bytes(2, 'big')) + + +class VerifyChecksumTests(unittest.TestCase): + """:func:`verify_checksum` returns each :class:`ChecksumStatus` it can.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + from pcapkit.protocols.misc.checksum import checksum + from pcapkit.utilities.warnings import ChecksumWarning, ProtocolWarning + + self.checksum = checksum + self.Status = checksum.ChecksumStatus + self.ProtocolWarning = ProtocolWarning + self.ChecksumWarning = ChecksumWarning + body = b'\x45\x00\x00\x1c\x12\x34\x00\x00\x40\x11\x00\x00\xc0\x00\x02\x01\xc6\x33\x64\x02' + self.field = wire.checksum(body).to_bytes(2, 'big') + self.data = body[:10] + self.field + body[12:] + + def verify(self, data: bytes, field: bytes, **kwargs: object) -> 'tuple[object, list[tuple[type, str]]]': + """The status of ``data``, and its ProtocolWarnings as ``(category, message)``.""" + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter('always') + status = self.checksum.verify_checksum('TEST', data, field, **kwargs) + return status, [(item.category, str(item.message)) + for item in caught if issubclass(item.category, self.ProtocolWarning)] + + def corrupt(self, field: bytes) -> bytes: + """``self.data`` with ``field`` in place of its checksum.""" + return self.data[:10] + field + self.data[12:] + + def test_valid(self) -> None: + self.assertEqual(self.verify(self.data, self.field), (self.Status.VALID, [])) + + def test_invalid_warns_with_the_expected_value(self) -> None: + status, messages = self.verify(self.corrupt(b'\xbe\xef'), b'\xbe\xef') + self.assertIs(status, self.Status.INVALID) + self.assertEqual(messages, [(self.ChecksumWarning, + f'TEST: checksum 0xbeef is wrong; 0x{self.field.hex()} expected')]) + + def test_both_zeros_verify(self) -> None: + # RFC 1071: a receiver sums the field in, so 0x0000 and 0xffff are both zero. + data = b'\xff\xff\x00\x00' + self.assertEqual(self.verify(data, b'\x00\x00'), (self.Status.VALID, [])) + self.assertEqual(self.verify(b'\xff\xff\xff\xff', b'\xff\xff'), (self.Status.VALID, [])) + + def test_truncated(self) -> None: + self.assertEqual(self.verify(self.data, self.field, length=len(self.data) + 1), + (self.Status.TRUNCATED, [])) + self.assertEqual(self.verify(self.data + b'junk', self.field, length=len(self.data)), + (self.Status.VALID, [])) + + def test_absent_is_checked_first(self) -> None: + data = self.corrupt(b'\x00\x00') + self.assertEqual(self.verify(data, b'\x00\x00', absent=b'\x00\x00'), (self.Status.NOT_COMPUTED, [])) + + def test_offload_only_once_wrong(self) -> None: + self.assertEqual(self.verify(self.corrupt(b'\x12\x34'), b'\x12\x34', offload=(b'\x12\x34',)), + (self.Status.NOT_COMPUTED, [])) + self.assertEqual(self.verify(self.data, self.field, offload=(self.field,)), (self.Status.VALID, [])) + + def test_the_status_is_shaped_like_esp_status(self) -> None: + from pcapkit.corekit.enum import EnumLookup + + self.assertEqual(self.Status.__bases__, (EnumLookup, enum.IntEnum)) + self.assertEqual([member.name for member in self.Status], + ['VALID', 'INVALID', 'NOT_COMPUTED', 'TRUNCATED', 'NO_PSEUDO_HEADER']) + self.assertIs(self.Status.get('NOT_COMPUTED'), self.Status.NOT_COMPUTED) + + +class PseudoHeaderTests(unittest.TestCase): + """:func:`pseudo_header` gives the upper layer's own length.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + from pcapkit.protocols.misc.checksum.checksum import pseudo_header + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo + + self.pseudo_header = pseudo_header + self.IP_Pseudo = IP_Pseudo + + def test_the_length_is_replaced(self) -> None: + for src, dst, offset, width in (('192.0.2.1', '198.51.100.2', 10, 2), + ('2001:db8::1', '2001:db8::2', 32, 4)): + pseudo = self.IP_Pseudo(src=src, dst=dst, protocol=6, upper_length=0) + with self.subTest(src=src): + self.assertIs(self.pseudo_header(pseudo, 0), pseudo.data) + octets = self.pseudo_header(pseudo, 0x1234) + self.assertEqual(octets[offset:offset + width], (0x1234).to_bytes(width, 'big')) + self.assertEqual(octets[:offset] + octets[offset + width:], + pseudo.data[:offset] + pseudo.data[offset + width:]) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/protocols/misc/checksum/test_ip_pseudo_2471_unit.py b/tests/protocols/misc/checksum/test_ip_pseudo_2471_unit.py index 589f46abfa..bb0dc50251 100644 --- a/tests/protocols/misc/checksum/test_ip_pseudo_2471_unit.py +++ b/tests/protocols/misc/checksum/test_ip_pseudo_2471_unit.py @@ -606,8 +606,12 @@ def chain(count: int) -> bytes: for index in range(count)) self.assertEqual(IP_Pseudo(ip=self.ipv6(60, chain(32) + self.UDP)).info.len, 8) + # It may hold a Home Address option, and it hides the upper layer (#2513). + ip = self.ipv6(60, chain(33) + self.UDP) + with self.assertRaisesRegex(ProtocolError, 'Destination Options of IPv6 not parsed; give src'): + IP_Pseudo(ip=ip) with self.assertRaisesRegex(ProtocolError, 'upper layer of IPv6 not parsed'): - IP_Pseudo(ip=self.ipv6(60, chain(33) + self.UDP)) + IP_Pseudo(ip=ip, src='2001:db8::1') def test_truncated_headers_are_unparsed(self) -> None: from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo @@ -631,6 +635,41 @@ def test_malformed_extension_headers_raise_protocol_error(self) -> None: with self.assertRaisesRegex(ProtocolError, 'Routing header of IPv6 not parsed; give dst'): IP_Pseudo(ip=self.ipv6(43, bytes([17, 1, 0, 1, 0, 0, 0, 0]) + bytes(8))) + def test_a_malformed_header_past_the_first_is_unparsed(self) -> None: + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo + from pcapkit.utilities.exceptions import ProtocolError + + # A valid 8-octet header, then a malformed one as above: each may hold an + # address the pseudo-header takes, so giving the protocol and the length + # does not do without it (#2513). + opts = bytes([17, 2]) + bytes(22) + route = bytes([17, 1, 0, 1, 0, 0, 0, 0]) + bytes(8) + cases = ( + ('Destination Options', 'src', self.ipv6(0, bytes([60, 0]) + bytes(6) + opts, payload=16)), + ('Destination Options', 'src', self.ipv6(60, bytes([60, 0]) + bytes(6) + opts, payload=16)), + ('Routing header', 'dst', self.ipv6(0, bytes([43, 0]) + bytes(6) + route)), + ) + for index, (header, value, ip) in enumerate(cases): + with self.subTest(index): + with self.assertRaisesRegex(ProtocolError, f'{header} of IPv6 not parsed; give {value}'): + IP_Pseudo(ip=ip, protocol=17, upper_length=8) + pseudo = IP_Pseudo(ip=ip, protocol=17, upper_length=8, **{value: '2001:db8::99'}) + self.assertEqual(getattr(pseudo, value), ipaddress.IPv6Address('2001:db8::99')) + + def test_the_upper_layer_needs_its_protocol_and_length(self) -> None: + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo + from pcapkit.utilities.exceptions import ProtocolError + + # The malformed headers above, with the address given: what the walk still + # cannot read is the upper layer's protocol and length (#2513). + opts = self.ipv6(60, bytes([17, 2]) + bytes(22), payload=8) + route = self.ipv6(43, bytes([17, 1, 0, 1, 0, 0, 0, 0]) + bytes(8)) + for ip, value in ((opts, 'src'), (route, 'dst')): + with self.subTest(value): + with self.assertRaisesRegex(ProtocolError, 'upper layer of IPv6 not parsed; ' + 'give protocol and upper_length'): + IP_Pseudo(ip=ip, **{value: '2001:db8::99'}) + def test_last_fragment_needs_the_reassembled_length(self) -> None: from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo from pcapkit.utilities.exceptions import ProtocolError diff --git a/tests/protocols/test_make_raw_item_read_unit.py b/tests/protocols/test_make_raw_item_read_unit.py index 399debb452..9e163b2439 100644 --- a/tests/protocols/test_make_raw_item_read_unit.py +++ b/tests/protocols/test_make_raw_item_read_unit.py @@ -98,7 +98,7 @@ def test_well_formed_raw_item_still_builds(self) -> None: # the wider ``try`` leaves a raw item that does parse alone ipv4 = _attr('pcapkit.protocols.internet.ipv4.IPv4') proto = ipv4(options=[b'\x94\x04\x00\x00'], protocol=253) - self.assertEqual(proto.data.hex(), '460000180000000000fd00007f0000010000000094040000') + self.assertEqual(proto.data.hex(), '460000180000000000fda5e47f0000010000000094040000') if __name__ == '__main__': diff --git a/tests/protocols/test_protocol_make_unit.py b/tests/protocols/test_protocol_make_unit.py index cbe939a438..73be3808fc 100644 --- a/tests/protocols/test_protocol_make_unit.py +++ b/tests/protocols/test_protocol_make_unit.py @@ -50,7 +50,7 @@ def _attr(path: 'str') -> 'Any': '1106206100000000' + '00' * 32 + '0041000c' + '00' * 12), 'IPv4 options': ('pcapkit.protocols.internet.ipv4.IPv4', {'options': [b'\x94\x04\x00\x00'], 'protocol': 253}, - '460000180000000000fd00007f0000010000000094040000'), + '460000180000000000fda5e47f0000010000000094040000'), 'TCP options': ('pcapkit.protocols.transport.tcp.TCP', {'options': [b'\x02\x04\x05\xb4']}, '0000000000000000000000006000ffff00000000020405b4'), diff --git a/tests/protocols/transport/test_tcp_checksum_2473_unit.py b/tests/protocols/transport/test_tcp_checksum_2473_unit.py new file mode 100644 index 0000000000..03f599c702 --- /dev/null +++ b/tests/protocols/transport/test_tcp_checksum_2473_unit.py @@ -0,0 +1,227 @@ +# -*- coding: utf-8 -*- +"""TCP verifies and computes its checksum over the IP pseudo-header. + +GitHub issue #2473: the enclosing IPv4 or IPv6 layer verifies the TCP checksum +(:rfc:`793#section-3.1`) once parsed, under +:data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`, and computes it for a segment +constructed as its payload under :data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE`; +:meth:`TCP.make ` also takes the +``pseudo`` itself. The captured segments are from the tracked +:file:`examples/captures/in.pcap`; expected checksums are the hand-written sum of +``tests/foundation/_roundtrip.py`` over a hand-built pseudo-header. + +""" +from __future__ import annotations + +import importlib.util +import unittest +import warnings +from unittest import mock + +from tests._support import reimport_once_per_class +from tests.foundation import _roundtrip as wire + +#: Test tier (#1958): hand-picked parse -> rebuild and make -> parse round trips. +__tier__ = 'ALL' + +#: IPv4 packets carrying TCP as captured (``in.pcap`` frames 2 to 4); the last +#: two were sent by the capturing host, which computed its checksums. +CAPTURED = ( + bytes.fromhex('45000028c53440002b067a867b81d287c0a801640050d7c0d59fb154a67aa0ff50117cb07c8e0000'), + bytes.fromhex('450000280000400040062abbc0a801647b81d287d7c00050a67aa0ffd59fb1555010fffff93e0000'), + bytes.fromhex('450000280000400040062abbc0a801647b81d287d7b000502dd48d216d65746e5011ffff2af40000'), +) + +HAS_SCAPY = importlib.util.find_spec('scapy') is not None + + +def pseudo4(packet: bytes, length: int) -> bytes: + """The IPv4 pseudo-header of ``packet``, an IPv4 header without options.""" + return packet[12:20] + b'\x00\x06' + length.to_bytes(2, 'big') + + +def pseudo6(src: bytes, dst: bytes, length: int) -> bytes: + """The IPv6 pseudo-header of TCP (RFC 8200 section 8.1).""" + return src + dst + length.to_bytes(4, 'big') + b'\x00\x00\x00\x06' + + +def expected(prefix: bytes, segment: bytes) -> bytes: + """The checksum ``segment`` should carry over ``prefix``.""" + return wire.checksum(prefix + segment[:16] + b'\x00\x00' + segment[18:]).to_bytes(2, 'big') + + +def with_checksum(packet: bytes, field: bytes, at: int = 20 + 16) -> bytes: + """``packet`` with ``field`` at ``at``, the TCP checksum behind a 20-octet IPv4 header.""" + return packet[:at] + field + packet[at + 2:] + + +class TCPChecksumTests(unittest.TestCase): + """Verifying and computing the TCP checksum.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + from pcapkit.protocols.internet.ipv4 import IPv4 + from pcapkit.protocols.internet.ipv6 import IPv6 + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo + from pcapkit.protocols.transport.tcp import TCP + from pcapkit.utilities import config + from pcapkit.utilities.warnings import ChecksumWarning, ProtocolWarning + + self.IPv4, self.IPv6, self.TCP, self.IP_Pseudo = IPv4, IPv6, TCP, IP_Pseudo + self.Status = ChecksumStatus + self.config = config + self.ProtocolWarning = ProtocolWarning + self.ChecksumWarning = ChecksumWarning + + def parse(self, octets: bytes, cls: 'type | None' = None, *, + verify: bool = True) -> 'tuple[object, list[tuple[type, str]]]': + """``octets`` parsed as ``cls``, IPv4 by default, and its ProtocolWarnings as ``(category, message)``.""" + with mock.patch.object(self.config, 'CHECKSUM_VERIFY', verify), \ + warnings.catch_warnings(record=True) as caught: + warnings.simplefilter('always') + layer = (cls or self.IPv4)(octets) + return layer, [(item.category, str(item.message)) + for item in caught if issubclass(item.category, self.ProtocolWarning)] + + def status(self, octets: bytes, **kwargs: object) -> 'tuple[object, list[tuple[type, str]]]': + """The TCP checksum status of ``octets``, and its ProtocolWarnings as ``(category, message)``.""" + layer, messages = self.parse(octets, **kwargs) + return layer['TCP'].info.get('checksum_status'), messages + + def test_captured_checksums_verify(self) -> None: + for packet in CAPTURED: + with self.subTest(packet=packet.hex()): + self.assertEqual(expected(pseudo4(packet, 20), packet[20:]), packet[36:38]) + self.assertEqual(self.status(packet), (self.Status.VALID, [])) + + def test_a_corrupted_checksum_warns(self) -> None: + status, messages = self.status(with_checksum(CAPTURED[0], b'\x7c\x8f')) + self.assertIs(status, self.Status.INVALID) + self.assertEqual(messages, [(self.ChecksumWarning, 'TCP: checksum 0x7c8f is wrong; 0x7c8e expected')]) + + def test_offload_is_not_computed(self) -> None: + # Linux offload leaves the pseudo-header's sum alone, uncomplemented. + partial = (~wire.checksum(pseudo4(CAPTURED[1], 20)) & 0xFFFF).to_bytes(2, 'big') + for field in (b'\x00\x00', partial): + with self.subTest(field=field.hex()): + self.assertEqual(self.status(with_checksum(CAPTURED[1], field)), (self.Status.NOT_COMPUTED, [])) + + def test_off_verifies_nothing(self) -> None: + self.assertEqual(self.status(with_checksum(CAPTURED[0], b'\x7c\x8f'), verify=False), (None, [])) + + def test_without_a_pseudo_header(self) -> None: + # A first fragment: the segment's length is only known once reassembled. + header = bytearray(CAPTURED[0][:6] + b'\x20\x00' + CAPTURED[0][8:10] + b'\x00\x00' + CAPTURED[0][12:20]) + header[10:12] = wire.checksum(bytes(header)).to_bytes(2, 'big') + status, messages = self.status(bytes(header) + CAPTURED[0][20:]) + self.assertEqual((status, messages), (self.Status.NO_PSEUDO_HEADER, [])) + # Alone, a segment has no IP layer to verify it. + tcp, _ = self.parse(CAPTURED[0][20:], self.TCP) + self.assertNotIn('checksum_status', tcp.info) + + def test_behind_an_authentication_header(self) -> None: + # IPv4 carries AH as its payload, ahead of the segment (RFC 4302 section 2.2). + segment = self.TCP(srcport=40000, dstport=80, payload=b'signed', checksum=b'\x00\x00').data + ah = bytes([6, 4, 0, 0]) + (0x1234).to_bytes(4, 'big') + (1).to_bytes(4, 'big') + bytes(12) + header = bytearray(b'\x45\x00' + (20 + len(ah) + len(segment)).to_bytes(2, 'big') + bytes(4) + + b'\x40\x33\x00\x00' + wire.IPV4_A + wire.IPV4_B) + header[10:12] = wire.checksum(bytes(header)).to_bytes(2, 'big') + prefix = wire.IPV4_A + wire.IPV4_B + b'\x00\x06' + len(segment).to_bytes(2, 'big') + packet = bytes(header) + ah + segment[:16] + expected(prefix, segment) + segment[18:] + self.assertEqual(self.status(packet), (self.Status.VALID, [])) + + def test_an_upper_layer_the_pseudo_header_does_not_name(self) -> None: + # IPv6 walks a Mobility Header as an extension header, but it is the + # pseudo-header's upper layer (RFC 6275 section 6.1.1), not the segment. + segment = self.TCP(srcport=40000, dstport=80, checksum=b'\x00\x00').data + mh = bytes([6, 0, 0, 0, 0, 0, 0, 0]) + packet = (b'\x60\x00\x00\x00' + (len(mh) + len(segment)).to_bytes(2, 'big') + bytes([135, 64]) + + wire.IPV6_A + wire.IPV6_B + mh + segment) + layer, messages = self.parse(packet, self.IPv6) + self.assertEqual(str(layer.protochain), 'IPv6:MH:TCP') + self.assertEqual((layer['TCP'].info.checksum_status, messages), (self.Status.NO_PSEUDO_HEADER, [])) + + def test_ipv6_and_truncation(self) -> None: + segment = self.TCP(srcport=80, dstport=40000, seq_no=1, ack=True, ack_no=2, payload=b'odd').data + with mock.patch.object(self.config, 'CHECKSUM_COMPUTE', True): + packet = self.IPv6(src='2001:db8::1', dst='2001:db8::2', next='TCP', payload=segment).data + self.assertEqual(self.status(packet, cls=self.IPv6), (self.Status.NOT_COMPUTED, [])) + + made = self.IPv6(src='2001:db8::1', dst='2001:db8::2', next='TCP', + payload=self.TCP(srcport=80, dstport=40000, payload=b'odd')).data + self.assertEqual(made[56:58], expected(pseudo6(wire.IPV6_A, wire.IPV6_B, 23), made[40:])) + self.assertEqual(self.status(made, cls=self.IPv6), (self.Status.VALID, [])) + self.assertEqual(self.status(made[:-1], cls=self.IPv6), (self.Status.TRUNCATED, [])) + + def test_parse_rebuild_is_exact(self) -> None: + for packet in (*CAPTURED, with_checksum(CAPTURED[0], b'\x7c\x8f'), with_checksum(CAPTURED[1], b'\x00\x00')): + for verify in (True, False): + with self.subTest(packet=packet.hex(), verify=verify): + layer, _ = self.parse(packet, verify=verify) + self.assertEqual(self.IPv4.from_data(layer.info).data, packet) + + def make(self, compute: bool = True, **kwargs: object) -> bytes: + """The octets of an IPv4 packet carrying a TCP segment made with ``kwargs``.""" + with mock.patch.object(self.config, 'CHECKSUM_COMPUTE', compute): + tcp = self.TCP(srcport=40000, dstport=80, seq_no=7, syn=True, payload=b'hello', **kwargs) + return self.IPv4(src='192.0.2.1', dst='198.51.100.2', ttl=64, protocol='TCP', payload=tcp).data + + def test_the_ip_layer_computes_it(self) -> None: + octets = self.make() + self.assertEqual(octets[36:38], expected(pseudo4(octets, 25), octets[20:])) + layer, messages = self.parse(octets) + self.assertIs(layer['TCP'].info.checksum_status, self.Status.VALID) + self.assertEqual(messages, []) + self.assertEqual(self.IPv4.from_data(layer.info).data, octets) + + def test_a_source_route_names_the_destination(self) -> None: + from pcapkit.const.ipv4.option_number import OptionNumber + + tcp = self.TCP(srcport=40000, dstport=80, payload=b'routed') + octets = self.IPv4(src='192.0.2.1', dst='198.51.100.2', protocol='TCP', payload=tcp, + options=[(OptionNumber.LSR, {'counts': 2, 'route': ['203.0.113.1'], + 'remaining': ['203.0.113.9']})]).data + hdr = (octets[0] & 0x0F) * 4 + prefix = octets[12:16] + bytes([203, 0, 113, 9]) + b'\x00\x06' + (len(octets) - hdr).to_bytes(2, 'big') + self.assertEqual(octets[hdr + 16:hdr + 18], expected(prefix, octets[hdr:])) + self.assertEqual(self.status(octets), (self.Status.VALID, [])) + + def test_a_wrong_header_checksum_warns_once(self) -> None: + with mock.patch.object(self.config, 'CHECKSUM_VERIFY', True), \ + warnings.catch_warnings(record=True) as caught: + warnings.simplefilter('always') + ipv4 = self.IPv4(src='192.0.2.1', dst='198.51.100.2', protocol='TCP', checksum=b'\xbe\xef', + payload=self.TCP(srcport=40000, dstport=80)) + messages = [(item.category, str(item.message)) + for item in caught if issubclass(item.category, self.ProtocolWarning)] + self.assertEqual(len(messages), 1, messages) + self.assertIs(messages[0][0], self.ChecksumWarning, messages) + self.assertTrue(messages[0][1].startswith('IPv4: checksum 0xbeef is wrong'), messages) + self.assertIs(ipv4['TCP'].info.checksum_status, self.Status.VALID) + + def test_make_takes_the_pseudo_header(self) -> None: + pseudo = self.IP_Pseudo(src='192.0.2.1', dst='198.51.100.2', protocol='TCP', upper_length=0) + segment = self.TCP(srcport=40000, dstport=80, payload=b'alone', pseudo=pseudo).data + prefix = wire.IPV4_A + wire.IPV4_B + b'\x00\x06' + len(segment).to_bytes(2, 'big') + self.assertEqual(segment[16:18], expected(prefix, segment)) + + @unittest.skipUnless(HAS_SCAPY, 'scapy not installed') + def test_make_agrees_with_scapy(self) -> None: + from scapy.layers.inet import IP + from scapy.layers.inet import TCP as ScapyTCP + + octets = self.make() + packet = IP(octets) + del packet[ScapyTCP].chksum + self.assertEqual(bytes(packet), octets) + + def test_off_or_given_is_written_as_is(self) -> None: + self.assertEqual(self.make(compute=False)[36:38], b'\x00\x00') + self.assertEqual(self.make(checksum=b'\xbe\xef')[36:38], b'\xbe\xef') + alone = self.TCP(srcport=40000, dstport=80) + self.assertEqual(alone.data[16:18], b'\x00\x00') + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/protocols/transport/test_udp_checksum_2474_unit.py b/tests/protocols/transport/test_udp_checksum_2474_unit.py new file mode 100644 index 0000000000..e65dd1df4e --- /dev/null +++ b/tests/protocols/transport/test_udp_checksum_2474_unit.py @@ -0,0 +1,206 @@ +# -*- coding: utf-8 -*- +"""UDP verifies and computes its checksum over the IP pseudo-header. + +GitHub issue #2474: the enclosing IPv4 or IPv6 layer verifies the UDP checksum +(:rfc:`768`) once parsed, under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`, +and computes it for a datagram constructed as its payload under +:data:`~pcapkit.utilities.config.CHECKSUM_COMPUTE`. A zero checksum means none over +IPv4 and is wrong over IPv6 (:rfc:`8200#section-8.1`), and a computed zero is sent +as all ones. The captured datagram is from the tracked +:file:`examples/captures/in.pcap`; expected checksums are the hand-written sum of +``tests/foundation/_roundtrip.py`` over a hand-built pseudo-header. + +""" +from __future__ import annotations + +import importlib.util +import unittest +import warnings +from unittest import mock + +from tests._support import reimport_once_per_class +from tests.foundation import _roundtrip as wire + +#: Test tier (#1958): hand-picked parse -> rebuild and make -> parse round trips. +__tier__ = 'ALL' + +#: An IPv4 packet carrying UDP as captured (``in.pcap`` frame 5), a router's +#: broadcast to port 5001. +CAPTURED = bytes.fromhex( + '4500009100004000401178b3c0a80101ffffffff92441389007d63b101010e00e12b83c7f98b0067000000' + '06000a544c2d57445236333030000b0003362e3000070001010005001142382d46382d38332d41352d4639' + '2d34370008000b3139322e3136382e312e310009000a74706c6f67696e2e636e000a000e544c2d57445236' + '33303020362e30000c0005312e372e34' +) + +HAS_SCAPY = importlib.util.find_spec('scapy') is not None + + +def pseudo4(src: bytes, dst: bytes, length: int) -> bytes: + """The IPv4 pseudo-header of UDP (RFC 768).""" + return src + dst + b'\x00\x11' + length.to_bytes(2, 'big') + + +def pseudo6(src: bytes, dst: bytes, length: int) -> bytes: + """The IPv6 pseudo-header of UDP (RFC 8200 section 8.1).""" + return src + dst + length.to_bytes(4, 'big') + b'\x00\x00\x00\x11' + + +def expected(prefix: bytes, datagram: bytes) -> bytes: + """The checksum ``datagram`` should carry over ``prefix``; zero is sent as all ones.""" + value = wire.checksum(prefix + datagram[:6] + b'\x00\x00' + datagram[8:]) + return (value or 0xFFFF).to_bytes(2, 'big') + + +def with_checksum(packet: bytes, field: bytes, at: int) -> bytes: + """``packet`` with ``field`` at ``at``.""" + return packet[:at] + field + packet[at + 2:] + + +class UDPChecksumTests(unittest.TestCase): + """Verifying and computing the UDP checksum.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + from pcapkit.protocols.internet.ipv4 import IPv4 + from pcapkit.protocols.internet.ipv6 import IPv6 + from pcapkit.protocols.misc.checksum.checksum import ChecksumStatus + from pcapkit.protocols.misc.checksum.ip_pseudo import IP_Pseudo + from pcapkit.protocols.transport.udp import UDP + from pcapkit.utilities import config + from pcapkit.utilities.warnings import ChecksumWarning, ProtocolWarning + + self.IPv4, self.IPv6, self.UDP, self.IP_Pseudo = IPv4, IPv6, UDP, IP_Pseudo + self.Status = ChecksumStatus + self.config = config + self.ProtocolWarning = ProtocolWarning + self.ChecksumWarning = ChecksumWarning + + def parse(self, octets: bytes, cls: 'type | None' = None, *, + verify: bool = True) -> 'tuple[object, list[tuple[type, str]]]': + """``octets`` parsed as ``cls``, IPv4 by default, and its ProtocolWarnings as ``(category, message)``.""" + with mock.patch.object(self.config, 'CHECKSUM_VERIFY', verify), \ + warnings.catch_warnings(record=True) as caught: + warnings.simplefilter('always') + layer = (cls or self.IPv4)(octets) + return layer, [(item.category, str(item.message)) + for item in caught if issubclass(item.category, self.ProtocolWarning)] + + def status(self, octets: bytes, **kwargs: object) -> 'tuple[object, list[tuple[type, str]]]': + """The UDP checksum status of ``octets``, and its ProtocolWarnings as ``(category, message)``.""" + layer, messages = self.parse(octets, **kwargs) + return layer['UDP'].info.get('checksum_status'), messages + + def make6(self, **kwargs: object) -> bytes: + """The octets of an IPv6 packet carrying a UDP datagram made with ``kwargs``.""" + udp = self.UDP(srcport=40000, dstport=9, payload=b'hello', **kwargs) + return self.IPv6(src='2001:db8::1', dst='2001:db8::2', next='UDP', payload=udp).data + + def test_the_captured_checksum_verifies(self) -> None: + self.assertEqual(expected(pseudo4(CAPTURED[12:16], CAPTURED[16:20], 125), CAPTURED[20:]), CAPTURED[26:28]) + self.assertEqual(self.status(CAPTURED), (self.Status.VALID, [])) + + def test_a_corrupted_checksum_warns(self) -> None: + status, messages = self.status(with_checksum(CAPTURED, b'\x63\xb2', 26)) + self.assertIs(status, self.Status.INVALID) + self.assertEqual(messages, [(self.ChecksumWarning, 'UDP: checksum 0x63b2 is wrong; 0x63b1 expected')]) + + def test_zero_is_none_over_ipv4_only(self) -> None: + self.assertEqual(self.status(with_checksum(CAPTURED, b'\x00\x00', 26)), (self.Status.NOT_COMPUTED, [])) + status, messages = self.status(with_checksum(self.make6(), b'\x00\x00', 46), cls=self.IPv6) + self.assertIs(status, self.Status.INVALID) + self.assertEqual([category for category, _ in messages], [self.ChecksumWarning]) + + def test_offload_is_not_computed(self) -> None: + prefix4 = pseudo4(CAPTURED[12:16], CAPTURED[16:20], 125) + partial4 = (~wire.checksum(prefix4) & 0xFFFF).to_bytes(2, 'big') + self.assertEqual(self.status(with_checksum(CAPTURED, partial4, 26)), (self.Status.NOT_COMPUTED, [])) + partial6 = (~wire.checksum(pseudo6(wire.IPV6_A, wire.IPV6_B, 13)) & 0xFFFF).to_bytes(2, 'big') + self.assertEqual(self.status(with_checksum(self.make6(), partial6, 46), cls=self.IPv6), + (self.Status.NOT_COMPUTED, [])) + + def test_a_first_fragment_has_no_pseudo_header(self) -> None: + header = bytearray(CAPTURED[:6] + b'\x20\x00' + CAPTURED[8:10] + b'\x00\x00' + CAPTURED[12:20]) + header[10:12] = wire.checksum(bytes(header)).to_bytes(2, 'big') + self.assertEqual(self.status(bytes(header) + CAPTURED[20:]), (self.Status.NO_PSEUDO_HEADER, [])) + + def test_off_verifies_nothing(self) -> None: + self.assertEqual(self.status(with_checksum(CAPTURED, b'\x63\xb2', 26), verify=False), (None, [])) + + def test_parse_rebuild_is_exact(self) -> None: + for packet in (CAPTURED, with_checksum(CAPTURED, b'\x63\xb2', 26), with_checksum(CAPTURED, b'\x00\x00', 26)): + for verify in (True, False): + with self.subTest(packet=packet.hex(), verify=verify): + layer, _ = self.parse(packet, verify=verify) + self.assertEqual(self.IPv4.from_data(layer.info).data, packet) + + def test_the_ip_layer_computes_it(self) -> None: + octets = self.IPv4(src='192.0.2.1', dst='198.51.100.2', protocol='UDP', + payload=self.UDP(srcport=40000, dstport=9, payload=b'hello')).data + self.assertEqual(octets[26:28], expected(pseudo4(wire.IPV4_A, wire.IPV4_B, 13), octets[20:])) + self.assertEqual(self.status(octets), (self.Status.VALID, [])) + + octets = self.make6() + self.assertEqual(octets[46:48], expected(pseudo6(wire.IPV6_A, wire.IPV6_B, 13), octets[40:])) + layer, messages = self.parse(octets, self.IPv6) + self.assertEqual((layer['UDP'].info.checksum_status, messages), (self.Status.VALID, [])) + self.assertEqual(self.IPv6.from_data(layer.info).data, octets) + + def test_a_fragment_leaves_it_out(self) -> None: + # The checksum covers the whole datagram, which a fragment does not hold. + for kwargs in ({'mf': True}, {'offset': 185}): + with self.subTest(**kwargs): + udp = self.UDP(srcport=40000, dstport=9, payload=b'hello') + octets = self.IPv4(src='192.0.2.1', dst='198.51.100.2', protocol='UDP', payload=udp, **kwargs).data + self.assertEqual(octets[20:], udp.data) + self.assertEqual(octets[26:28], b'\x00\x00') + + def test_the_length_field_bounds_it(self) -> None: + # The trailer past the Length field is neither summed nor in the pseudo-header. + udp = self.UDP(srcport=40000, dstport=9, payload=b'hello', trailer=b'pad') + octets = self.IPv4(src='192.0.2.1', dst='198.51.100.2', protocol='UDP', payload=udp).data + self.assertEqual(octets[26:28], expected(pseudo4(wire.IPV4_A, wire.IPV4_B, 13), octets[20:33])) + self.assertEqual(self.status(octets), (self.Status.VALID, [])) + + def test_a_computed_zero_is_sent_as_all_ones(self) -> None: + # Two payload octets chosen so that the sum over everything else is all ones. + prefix = pseudo4(wire.IPV4_A, wire.IPV4_B, 10) + rest = ~wire.checksum(prefix + b'\x9c\x40\x00\x09\x00\x0a\x00\x00') & 0xFFFF + word = (0xFFFF - rest).to_bytes(2, 'big') + octets = self.IPv4(src='192.0.2.1', dst='198.51.100.2', protocol='UDP', + payload=self.UDP(srcport=40000, dstport=9, payload=word)).data + self.assertEqual(wire.checksum(prefix + octets[20:26] + b'\x00\x00' + octets[28:]), 0) + self.assertEqual(octets[26:28], b'\xff\xff') + self.assertEqual(self.status(octets), (self.Status.VALID, [])) + + def test_make_takes_the_pseudo_header(self) -> None: + pseudo = self.IP_Pseudo(src='192.0.2.1', dst='198.51.100.2', protocol='UDP', upper_length=0) + datagram = self.UDP(srcport=40000, dstport=9, payload=b'alone', pseudo=pseudo).data + self.assertEqual(datagram[6:8], expected(pseudo4(wire.IPV4_A, wire.IPV4_B, 13), datagram)) + + # A jumbogram's zero Length (RFC 2675): the datagram is all but the trailer. + pseudo = self.IP_Pseudo(src='2001:db8::1', dst='2001:db8::2', protocol='UDP', upper_length=0) + datagram = self.UDP(srcport=40000, dstport=9, payload=b'alone', total_length=0, + trailer=b'pad', pseudo=pseudo).data + self.assertEqual(datagram[6:8], expected(pseudo6(wire.IPV6_A, wire.IPV6_B, 13), datagram[:13])) + + @unittest.skipUnless(HAS_SCAPY, 'scapy not installed') + def test_make_agrees_with_scapy(self) -> None: + from scapy.layers.inet import IP + from scapy.layers.inet import UDP as ScapyUDP + + octets = self.IPv4(src='192.0.2.1', dst='198.51.100.2', protocol='UDP', + payload=self.UDP(srcport=40000, dstport=9, payload=b'hello')).data + packet = IP(octets) + del packet[ScapyUDP].chksum + self.assertEqual(bytes(packet), octets) + + def test_off_or_given_is_written_as_is(self) -> None: + with mock.patch.object(self.config, 'CHECKSUM_COMPUTE', False): + self.assertEqual(self.make6()[46:48], b'\x00\x00') + self.assertEqual(self.make6(checksum=b'\xbe\xef')[46:48], b'\xbe\xef') + self.assertEqual(self.UDP(srcport=40000, dstport=9).data[6:8], b'\x00\x00') + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/utilities/test_checksum_warning_2470_unit.py b/tests/utilities/test_checksum_warning_2470_unit.py new file mode 100644 index 0000000000..5256b45af5 --- /dev/null +++ b/tests/utilities/test_checksum_warning_2470_unit.py @@ -0,0 +1,89 @@ +# -*- coding: utf-8 -*- +"""A wrong checksum warns with a category of its own. + +GitHub issue #2470 ruled that a checksum mismatch warns with +:exc:`~pcapkit.utilities.warnings.ChecksumWarning`, a +:exc:`~pcapkit.utilities.warnings.ProtocolWarning`, so that a filter can silence +mismatches alone. Each protocol's checksum tests pin that its mismatch warns with +it; these pin the class and what the filters do with it. + +""" +from __future__ import annotations + +import unittest +import warnings +from unittest import mock + +from tests._support import reimport_once_per_class +from tests.foundation import _roundtrip as wire + + +def chain(count: int) -> bytes: + """``count`` Destination Options headers, then a TCP segment.""" + return b''.join(bytes([60 if index < count - 1 else 6, 0, 1, 4, 0, 0, 0, 0]) + for index in range(count)) + wire.tcp(b'x', seq=1) + + +def with_wrong_checksum(packet: bytes) -> bytes: + """``packet`` with the low bit of its IPv4 header checksum flipped.""" + return packet[:11] + bytes([packet[11] ^ 1]) + packet[12:] + + +#: IPv4 with a wrong header checksum, carrying more Destination Options headers +#: than an extension-header limit of 2: a ChecksumWarning and a ProtocolWarning. +PACKET = with_wrong_checksum(wire.ipv4(chain(4), proto=60)) + + +class ChecksumWarningTests(unittest.TestCase): + """:exc:`ChecksumWarning` under the warning filters.""" + + def setUp(self) -> None: + reimport_once_per_class(self) + from pcapkit.protocols.internet.ipv4 import IPv4 + from pcapkit.utilities import config + from pcapkit.utilities import warnings as module + + self.IPv4 = IPv4 + self.config = config + self.module = module + self.ChecksumWarning = module.ChecksumWarning + self.ProtocolWarning = module.ProtocolWarning + + def parse(self, *filters: 'tuple[str, type]') -> 'list[tuple[type, str]]': + """The ProtocolWarnings of :data:`PACKET` under ``filters``, as ``(category, message)``.""" + with mock.patch.object(self.config, 'CHECKSUM_VERIFY', True), \ + mock.patch.object(self.config, 'EXTENSION_HEADER_LIMIT', 2), \ + warnings.catch_warnings(record=True) as caught: + warnings.simplefilter('always') + for action, category in filters: + warnings.filterwarnings(action, category=category) + self.IPv4(PACKET) + return [(item.category, str(item.message)) + for item in caught if issubclass(item.category, self.ProtocolWarning)] + + def test_a_protocol_warning(self) -> None: + self.assertTrue(issubclass(self.ChecksumWarning, self.ProtocolWarning)) + self.assertIn('ChecksumWarning', self.module.__all__) + + def categories(self, *filters: 'tuple[str, type]') -> 'list[str]': + """The categories :meth:`parse` gives, by name and sorted.""" + return sorted(category.__name__ for category, _ in self.parse(*filters)) + + def test_the_packet_gives_both(self) -> None: + self.assertEqual(self.categories(), ['ChecksumWarning', 'ProtocolWarning']) + + def test_a_protocol_warning_filter_still_catches_a_mismatch(self) -> None: + self.assertEqual(self.parse(('ignore', self.ProtocolWarning)), []) + self.assertEqual(self.categories(('ignore', Warning), ('always', self.ProtocolWarning)), + ['ChecksumWarning', 'ProtocolWarning']) + with self.assertRaisesRegex(self.ChecksumWarning, r'^IPv4: checksum 0x\w+ is wrong'): + self.parse(('error', self.ProtocolWarning)) + + def test_ignoring_checksum_warnings_keeps_the_others(self) -> None: + records = self.parse(('ignore', self.ChecksumWarning)) + self.assertEqual([category for category, _ in records], [self.ProtocolWarning]) + self.assertIn('reached the limit of 2 headers', records[0][1]) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/utilities/test_config_unit.py b/tests/utilities/test_config_unit.py index 0757603752..8dc69bf15c 100644 --- a/tests/utilities/test_config_unit.py +++ b/tests/utilities/test_config_unit.py @@ -80,6 +80,13 @@ ('ZERO_PAD_BUDGET_RATIO', 16, 0, 'amplify each octet read'), ) +#: Every flag :mod:`pcapkit.utilities.config` declares, as ``(name, default)`` +#: (#2471: verification off by default, computation on). +FLAGS = ( + ('CHECKSUM_VERIFY', False), + ('CHECKSUM_COMPUTE', True), +) + def options(nxt: int) -> bytes: """An 8-octet Destination Options header: Hdr Ext Len 0 and one PadN.""" @@ -235,9 +242,19 @@ def load(self, name: str, text: 'str | None') -> 'tuple[object, list[str]]': importlib.reload(self.config) def test_the_table_names_every_variable(self) -> None: - names = {row[0] for row in DECLARED} + names = {row[0] for row in DECLARED} | {row[0] for row in FLAGS} self.assertEqual(names, set(self.config.__all__) - READERS) + def test_each_flag_default_and_both_values(self) -> None: + for name, default in FLAGS: + with self.subTest(name=name): + self.assertEqual(self.load(name, None), (default, [])) + self.assertEqual(self.load(name, 'on'), (True, [])) + self.assertEqual(self.load(name, 'off'), (False, [])) + value, messages = self.load(name, 'maybe') + self.assertEqual((value, len(messages)), (default, 1), messages) + self.assertIn(f"PCAPKIT_{name}='maybe' is not one of", messages[0]) + def test_each_variable_is_read_by_its_own_name(self) -> None: for name, default, _, _ in DECLARED: with self.subTest(name=name): @@ -305,7 +322,7 @@ def test_every_variable_is_documented(self) -> None: from pcapkit.utilities import config docs = (PACKAGE.parent / 'docs' / 'source' / 'pcapkit' / 'utilities' / 'config.rst').read_text('utf-8') - # A limit is named after its variable. + # A limit or a flag is named after its variable. for name in set(config.__all__) - READERS: with self.subTest(name=name): self.assertIn(f'.. envvar:: PCAPKIT_{name}\n', docs) diff --git a/tests/utilities/test_warning_filters.py b/tests/utilities/test_warning_filters.py index afe1eaf6af..25fd68863e 100644 --- a/tests/utilities/test_warning_filters.py +++ b/tests/utilities/test_warning_filters.py @@ -37,7 +37,7 @@ CATEGORY_NAMES = [ 'BaseWarning', 'FormatWarning', 'EngineWarning', 'InvalidVendorWarning', - 'FileWarning', 'LayerWarning', 'ProtocolWarning', 'AttributeWarning', + 'FileWarning', 'LayerWarning', 'ProtocolWarning', 'ChecksumWarning', 'AttributeWarning', 'DevModeWarning', 'VendorRequestWarning', 'VendorRuntimeWarning', 'UnknownFieldWarning', 'RegistryWarning', 'SchemaWarning', 'InfoWarning', 'SeekWarning', 'ExtractionWarning', 'ConfigWarning',