feat(application): implement TLS/SSL and DTLS record layer + handshake framing

docs/source/contributing/pep.rst

@@ -43,14 +43,18 @@ the ones outstanding are,
 * Link Layer: DSL, EAPOL, FDDI, ISDN, LINUX_SLL, LINUX_SLL2, NDP, PPP
 * Internet Layer: ECN, ICMP, ICMPv6, IGMP, Shim6
 * Transport Layer: DCCP, RSVP
-* Application Layer: BGP, DHCP, DHCPv6, DNS, DTLS, IMAP, LDAP, MQTT, NNTP,
-  NTP, ONC/RPC, POP, QUIC, RIP, RTP, SIP, SMTP, SNMP, SSH, Telnet, TLS/SSL,
+* Application Layer: BGP, DHCP, DHCPv6, DNS, IMAP, LDAP, MQTT, NNTP,
+  NTP, ONC/RPC, POP, QUIC, RIP, RTP, SIP, SMTP, SNMP, SSH, Telnet,
   XMPP
 
 Several of these come in pairs that want a shared abstract base rather than two
 independent implementations, in the way
 :class:`~pcapkit.protocols.internet.ip.IP` already covers its family: ICMP with
-ICMPv6, TLS/SSL with DTLS, and ``LINUX_SLL`` with ``LINUX_SLL2``.
+ICMPv6, and ``LINUX_SLL`` with ``LINUX_SLL2``. **Done** for TLS/SSL with
+DTLS: :class:`~pcapkit.protocols.application.ssl.SSL` is the shared
+record-layer base of :class:`~pcapkit.protocols.application.tls.TLS` and
+:class:`~pcapkit.protocols.application.dtls.DTLS` -- see the `DTLS`_ section
+below.
 :class:`~pcapkit.protocols.link.vlan.VLAN` is the closer precedent for a pair
 whose *layout* is identical -- it holds the whole of the tag, and
 :class:`~pcapkit.protocols.link.c_tag.C_Tag` and
@@ -62,7 +66,7 @@ whose *layout* is identical -- it holds the whole of the tag, and
    ``NotImplemented`` folder. **Those files are not a roadmap.** They are
    scratch reminders, so a stub's presence does not mean a protocol is planned or
    claimed, and its absence does not mean the protocol is unwanted:
-   ``LINUX_SLL``, ``LINUX_SLL2``, ``QUIC`` and **DTLS** have no stub and are on the
+   ``LINUX_SLL``, ``LINUX_SLL2`` and ``QUIC`` have no stub and are on the
    list above,
    while **NGAP** has none either and is implemented. Take this page as the
    record and ignore the folder.
@@ -222,35 +226,41 @@ looks like an ordinary field and is not:
 DTLS
 ~~~~
 
-**Not started**, and it has no stub in the tree -- of the TLS/SSL and DTLS pair,
-only TLS/SSL does. It earns its own entry because the registries
-have moved ahead of it, and a growing part of SCTP's surface names a protocol
-that does not exist:
-
-* the DTLS chunk, type 65 of :class:`~pcapkit.const.sctp.chunk.Chunk`, and the
-  DTLS Key Management chunk parameter, ``0x8006`` of
-  :class:`~pcapkit.const.sctp.parameter.Parameter` -- both fall through to the
-  generic handlers, so they parse as opaque;
-* its error causes, 100 to 103 of
-  :class:`~pcapkit.const.sctp.cause_code.CauseCode`;
-* payload protocol identifiers -- 47 (Diameter over DTLS/SCTP), 66 to 69
-  (NGAP, XnAP, F1AP and E1AP, each over DTLS over SCTP) and 4242 (DTLS chunk
-  key management).
-
-That last group is the one that bites. NGAP over DTLS over SCTP is PPID 66, and
-even with an NGAP dissector registered against it the bytes arriving are a DTLS
-record rather than an NGAP PDU, so they can only degrade to
-:class:`~pcapkit.protocols.misc.raw.Raw`. DTLS therefore blocks NGAP over DTLS,
+**Done** for the record layer and handshake message framing:
+:class:`~pcapkit.protocols.application.tls.TLS` (SSLv3, :rfc:`6101`; TLS
+1.0, :rfc:`2246`; TLS 1.1, :rfc:`4346`; TLS 1.2, :rfc:`5246`; TLS 1.3,
+:rfc:`9846`) and :class:`~pcapkit.protocols.application.dtls.DTLS` (DTLS
+1.0, :rfc:`4347`; DTLS 1.2, :rfc:`6347`; DTLS 1.3, :rfc:`9147`) share
+:class:`~pcapkit.protocols.application.ssl.SSL` as their record-layer base --
+content type, legacy version, length, and content-type dispatch -- with
+DTLS's own divergence (the epoch/sequence-number pair in its record header,
+and the fragment header each handshake message carries) staying in
+:class:`~pcapkit.protocols.application.dtls.DTLS` alone. ``ClientHello`` and
+``ServerHello`` decode down to their cipher suite(s), compression method(s)
+and extension list (TYPE+LENGTH framed, extension bodies undecoded); every
+other handshake message is message-TYPE-level framing only. Decryption is a
+separate question and is not needed to make the record structure legible, in
+the same way :class:`ESP <pcapkit.protocols.internet.esp.ESP>` parses
+without keys.
+
+**Not done: reassembling a handshake message DTLS has fragmented across
+several records** (:rfc:`6347#section-4.2.3`, :issue:`2440`). That needs a
+:class:`~pcapkit.foundation.reassembly.reassembly.Reassembly` subclass keyed
+on the UDP 5-tuple plus epoch. A fragmented message's body is kept as that one
+fragment's raw octets -- see
+:attr:`DTLSHandshake.is_fragmented <pcapkit.protocols.data.application.dtls.DTLSHandshake.is_fragmented>`
+-- rather than decoded from a slice of the message. An unfragmented message is
+unaffected.
+
+NGAP over DTLS over SCTP (PPID 66,
+:class:`~pcapkit.const.sctp.payload_protocol_identifier.PayloadProtocolIdentifier`),
 asked for in `discussion #251
-<https://github.com/JarryShaw/PyPCAPKit/discussions/251>`__, independently of
-whether NGAP itself is implemented.
-
-A record-layer dissector is enough to unblock that -- content type, version,
-epoch, sequence number, length, and the fragment offset and length DTLS adds to
-handshake messages. Decryption is a separate question and is not needed to make
-the record structure legible, in the same way :class:`ESP
-<pcapkit.protocols.internet.esp.ESP>` parses without keys. DTLS over UDP is the
-other half of the same work and wants the same dissector.
+<https://github.com/JarryShaw/PyPCAPKit/discussions/251>`__, is **not**
+decoded: :attr:`SCTP.__proto__ <pcapkit.protocols.transport.sctp.SCTP.__proto__>`
+maps PPID 66 to :class:`~pcapkit.protocols.application.ngap.NGAP` rather than
+through :class:`~pcapkit.protocols.application.dtls.DTLS`, so those bytes
+degrade to :class:`~pcapkit.protocols.misc.raw.Raw`. Rewiring that dispatch
+needs a decision on what an undecrypted DTLS-wrapped NGAP PDU reports.
 
 Registered, but Not Dissected
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -307,10 +317,13 @@ Some wanted only a table entry, because the dissector was already there:
   *name* rather than a parse -- which is the right answer for a data channel
   carrying an arbitrary file.
 * **Done.** HTTP is additionally bound on 8080 (IANA ``http-alt``, "HTTP
-  Alternate (see port 80)") on both TCP and UDP. **8443 is deliberately left
-  unbound**: IANA registers it as ``pcsync-https``, not as an HTTP alternate,
-  and de-facto 8443 traffic is TLS-wrapped, which pcapkit cannot parse -- see
-  the ``tls`` stub above. Binding it would feed a TLS record to an HTTP parser.
+  Alternate (see port 80)") on both TCP and UDP. TCP 443 (IANA ``https``) is
+  bound to :class:`~pcapkit.protocols.application.tls.TLS` and UDP 5684 (IANA
+  ``coaps``) to :class:`~pcapkit.protocols.application.dtls.DTLS`. **8443 is deliberately left unbound**: IANA registers it as
+  ``pcsync-https``, not as an HTTP alternate, and binding it to HTTP would
+  feed a TLS record to an HTTP parser; binding it to TLS instead would be a
+  guess this registry does not make for a service IANA does not actually
+  name there.
 * **Done.** The service VLAN tag identifier (S-Tag), ``0x88A8``, is bound to
   :class:`~pcapkit.protocols.link.s_tag.S_Tag`, and the customer tag ``0x8100``
   to :class:`~pcapkit.protocols.link.c_tag.C_Tag`. Both subclass the abstract
@@ -1078,6 +1091,11 @@ pair alone would answer it twice. ``LINUX_SLL`` matters out of proportion to its
 size: every ``tcpdump -i any`` capture uses it, and neither variant has even a
 stub.
 
+**Done within wave 2 — TLS/SSL with DTLS.** The `DTLS`_ section above lists
+what is decoded (the shared :class:`~pcapkit.protocols.application.ssl.SSL`
+base, TLS, DTLS, and handshake message framing) and what is not (DTLS
+handshake-fragment reassembly, :issue:`2440`).
+
 **Done ahead of wave 2 — handlers for the** ``DLT_NULL`` **and** ``DLT_RAW``
 **link types.** Both are genuine DLTs that a capture can legitimately be rooted
 in — ``DLT_NULL`` (0) is BSD's loopback encapsulation, ``DLT_RAW`` (101) an IP

docs/source/ext.rst

@@ -89,6 +89,14 @@ The protocol classes :mod:`pcapkit` ships:
 |                                                                  |                | :class:`pcapkit.protocols.application.rarp.RARP`                                    |
 +                                                                  + RARP Family    +-----------------------+-------------------------------------------------------------+
 |                                                                  |                | :class:`pcapkit.protocols.application.rarp.DRARP`                                   |
++                                                                  +----------------+-----------------------+-------------------------------------------------------------+
+|                                                                  |                | :class:`pcapkit.protocols.application.ssl.SSL`                                      |
++                                                                  +                +-----------------------+-------------------------------------------------------------+
+|                                                                  |                | :class:`pcapkit.protocols.application.tls.TLS`                                      |
++                                                                  + SSL Family     +-----------------------+-------------------------------------------------------------+
+|                                                                  |                | :class:`pcapkit.protocols.application.tls.SSLv3`                                    |
++                                                                  +                +-----------------------+-------------------------------------------------------------+
+|                                                                  |                | :class:`pcapkit.protocols.application.dtls.DTLS`                                    |
 +------------------------------------------------------------------+----------------+-----------------------+-------------------------------------------------------------+
 |                                                                  |                | :class:`pcapkit.protocols.misc.pcap.header.Header`                                  |
 +                                                                  + PCAP Format    +-----------------------+-------------------------------------------------------------+

docs/source/pcapkit/const/index.rst

@@ -104,6 +104,7 @@ Application Layer
    http
    ngap
    ospf
+   tls
 
 Miscellaneous
 -------------

docs/source/pcapkit/const/tls.rst

@@ -0,0 +1,53 @@
+=====================================================================
+:class:`~pcapkit.protocols.application.ssl.SSL` Constant Enumerations
+=====================================================================
+
+.. module:: pcapkit.const.tls
+
+This module contains the constant enumerations shared by the record-layer
+implementations under the :class:`~pcapkit.protocols.application.ssl.SSL`
+umbrella -- :class:`~pcapkit.protocols.application.tls.TLS` (SSLv3 through
+TLS 1.3) and :class:`~pcapkit.protocols.application.dtls.DTLS`. Available
+enumerations include:
+
+.. list-table::
+
+   * - :class:`TLS_ContentType <pcapkit.const.tls.content_type.ContentType>`
+     - TLS/DTLS Content Types [*]_
+   * - :class:`TLS_HandshakeType <pcapkit.const.tls.handshake_type.HandshakeType>`
+     - TLS/DTLS HandshakeType [*]_
+
+.. versionadded:: 1.6.0
+
+TLS/DTLS Content Types
+======================
+
+.. module:: pcapkit.const.tls.content_type
+
+This module contains the constant enumeration for **TLS/DTLS Content Types**,
+hand-curated from IANA's ``TLS ContentType`` registry rather than generated by
+a :class:`~pcapkit.vendor.default.Vendor` crawler.
+
+.. autoclass:: pcapkit.const.tls.content_type.ContentType
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+TLS/DTLS HandshakeType
+======================
+
+.. module:: pcapkit.const.tls.handshake_type
+
+This module contains the constant enumeration for **TLS/DTLS HandshakeType**,
+hand-curated from IANA's ``TLS HandshakeType`` registry, including the
+DTLS-only messages :rfc:`9147#section-5.2` adds to it.
+
+.. autoclass:: pcapkit.const.tls.handshake_type.HandshakeType
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. rubric:: Footnotes
+
+.. [*] https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-5
+.. [*] https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-7

docs/source/pcapkit/protocols/application/dtls.rst

@@ -0,0 +1,112 @@
+DTLS - Datagram Transport Layer Security
+==============================================
+
+.. module:: pcapkit.protocols.application.dtls
+
+:mod:`pcapkit.protocols.application.dtls` contains
+:class:`~pcapkit.protocols.application.dtls.DTLS` only, which implements an
+extractor for Datagram Transport Layer Security (DTLS) [*]_ -- DTLS 1.0
+(:rfc:`4347`), DTLS 1.2 (:rfc:`6347`) and DTLS 1.3's legacy-format records
+(:rfc:`9147`) -- at the record layer, plus the epoch/sequence-number fields and
+the handshake-message fragment header that set DTLS apart from
+:class:`~pcapkit.protocols.application.tls.TLS`, whose reliable, ordered
+transport needs neither.
+
+Version Identification
+----------------------
+
+As with :class:`~pcapkit.protocols.application.tls.TLS`, one class parses
+every version and :meth:`~pcapkit.protocols.application.dtls.DTLS.read` sets
+:attr:`~pcapkit.protocols.application.dtls.DTLS.alias` from the record;
+:meth:`~pcapkit.protocols.application.dtls.DTLS.id` stays ``('DTLS',)``.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Legacy record version
+     - :attr:`~pcapkit.protocols.application.dtls.DTLS.alias`
+   * - ``0xFEFF``
+     - ``DTLS 1.0``
+   * - ``0xFEFD``
+     - ``DTLS 1.2``
+   * - ``0xFEFD``, or ``0xFEFF`` on an initial ``ClientHello``, whose
+       ``supported_versions`` extension lists ``0xFEFC``
+     - ``DTLS 1.3``
+   * - any other ``0xFExx``
+     - ``DTLS``
+
+DTLS 1.3 never puts its own ``0xFEFC`` on the record layer
+(:rfc:`9147#section-4`), so it is told apart only by the
+``supported_versions`` peek. A version the table does not name keeps the
+generic ``DTLS`` alias rather than claiming a specific one.
+
+Known Limitations
+-----------------
+
+**A handshake message split across DTLS records is not reassembled**
+(:rfc:`6347#section-4.2.3`, :issue:`2440`). Such a message is flagged
+:attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.is_fragmented`
+and its :attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.body`
+kept as raw octets.
+
+**DTLS 1.3's unified record header is not decoded** (:issue:`2467`). It has
+no content-type octet (:rfc:`9147#section-4`), so it degrades to
+:class:`~pcapkit.protocols.misc.raw.Raw`. DTLS 1.3's cleartext records use
+the legacy ``DTLSPlaintext`` header and are decoded.
+
+**Connection ID records are not decoded** (``tls12_cid``, :rfc:`9146`): the
+connection ID's length is not carried in the record, so they degrade to
+:class:`~pcapkit.protocols.misc.raw.Raw`.
+
+.. versionadded:: 1.6.0
+
+.. raw:: html
+
+   <br />
+
+.. autoclass:: pcapkit.protocols.application.dtls.DTLS
+   :no-members:
+   :show-inheritance:
+
+   .. autoproperty:: name
+   .. autoproperty:: alias
+   .. autoproperty:: length
+
+   .. automethod:: id
+
+   .. automethod:: read
+   .. automethod:: make
+
+   .. automethod:: __length_hint__
+
+   .. automethod:: _make_data
+
+Header Schema
+-------------
+
+.. module:: pcapkit.protocols.schema.application.dtls
+
+.. autoclass:: pcapkit.protocols.schema.application.dtls.DTLS
+   :members:
+   :show-inheritance:
+
+Data Models
+-----------
+
+.. module:: pcapkit.protocols.data.application.dtls
+
+.. autoclass:: pcapkit.protocols.data.application.dtls.DTLS
+   :members:
+   :show-inheritance:
+
+.. autoclass:: pcapkit.protocols.data.application.dtls.DTLSHandshake
+   :members:
+   :show-inheritance:
+
+See :doc:`ssl` for ``ClientHello``, ``ServerHello``, ``Extension``, ``Alert``
+and ``ChangeCipherSpec``, shared verbatim with
+:class:`~pcapkit.protocols.application.tls.TLS`.
+
+.. rubric:: Footnotes
+
+.. [*] https://en.wikipedia.org/wiki/Datagram_Transport_Layer_Security

docs/source/pcapkit/protocols/application/index.rst

@@ -22,11 +22,14 @@ function rather than encapsulation, see
    ngap
    ospf
    rarp
+   ssl
+   tls
+   dtls
 
 .. todo::
 
    Implement BGP, DHCP, DHCPv6, DNS, IMAP, LDAP, MQTT, NNTP, NTP, ONC:RPC,
-   POP, RIP, RTP, SIP, SMTP, SNMP, SSH, TELNET, TLS/SSL, XMPP.
+   POP, RIP, RTP, SIP, SMTP, SNMP, SSH, TELNET, XMPP.
 
 Protocol Registry
 -----------------

docs/source/pcapkit/protocols/application/ngap.rst

@@ -91,8 +91,11 @@ Known Limitations
 -----------------
 
 * **PPID 66 payloads are not decoded.** NGAP over DTLS over SCTP wraps the
-  ``NGAP-PDU`` in a DTLS record, and :mod:`pcapkit` implements no DTLS, so the
-  bytes reaching :meth:`NGAP.read` are not an APER encoding. The PPID is
+  ``NGAP-PDU`` in a DTLS record, and :attr:`SCTP.__proto__
+  <pcapkit.protocols.transport.sctp.SCTP.__proto__>` maps PPID 66 to
+  :class:`~pcapkit.protocols.application.ngap.NGAP` rather than
+  through :class:`~pcapkit.protocols.application.dtls.DTLS`, so the bytes
+  reaching :meth:`NGAP.read` are not an APER encoding. The PPID is
   registered so that it is *named* rather than anonymous; the payload itself
   degrades to :class:`~pcapkit.protocols.misc.raw.Raw`.
 * **The specification version is that of the installed decoder, not this