Skip to content

fix(ftp,httpv1): keep the raw command case, HTTP/1 whitespace and field octets, and parse whole multi-line FTP replies - #1347

Merged
JarryShaw merged 1 commit into
mainfrom
fix/ftp-http1-raw-1302-1303-1333
Oct 7, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/ftp-http1-raw-1302-1303-1333

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner
  • Searched for similar pull requests
  • Followed the coding style (make pylint, make mypy, make isort): ran on the four changed pcapkit/ files. mypy and isort are clean. pylint reports only the 4 warnings main already has.
  • make test passes, and a test case covers the change: ran the new module, 10 FTP/HTTP modules, tests/project and the protocols --exclude protocols/internet ordering leg (910 tests, 0 failures)
  • Added a changelog entry under docs/source/changelog/ and regenerated CHANGELOG.md, if the change is user-visible — N/A, added centrally

What is the purpose of your pull request?

  • fix — corrects a defect

Description of your pull request and other information

Per the #1333 ruling, the original token or line is kept next to the normalised value, and make writes it whenever it still reads back as that value. A value that has been edited is built from the edit.

Before → after, via from_data(info).data: quit → QUIT; 220-a\r\n220 b → ProtocolError; X-Name: caf\xe9 → caf\xc3\xa9; 200 \tOK / /a\t / /a\x0c → stripped; Host:a → Host: a; X: a\r\n b → X: a b. All 11 probes now rebuild byte-exactly.

info is unchanged apart from the new keys:

  • 471 HTTP layers across examples/captures/* (frames and reassembled streams), and 189 bytes literals from the application tests, parsed as FTP and as HTTP/1.
  • Compared against origin/main with the new keys dropped, every layer is identical.
  • The only difference is 220-a\r\n220 b\r\n, which now parses where it used to raise.

Fails without the fix: the new module gives 12 passed / 48 subtests with the fix. Against main the first revision failed 31 / passed 5. Against that revision (d104be9cf) the new review tests fail 16 subtests: the 4 injection shapes, 3 raw_line shapes, 4 unclosed or over-long replies, 2 content-matching cases and 3 bad lines values.

Rebuilds: all 471 capture layers rebuild byte-exactly, and none raise.

Contract changes:

  • New data-class fields: Request.raw_cmmd, Response.lines, RequestHeader/ResponseHeader.raw_line and HTTP.raw_header.
  • New make() kwargs with defaults.
  • FTP.make raises ProtocolError when lines are not a closing tail of bytes for a more reply.
  • HTTP.make raises ProtocolError on the CR/LF case above.
  • Two tests pin the exact _make_data dict, so I added the new keys to them in test_ftp_unit.py and test_http_unit.py.

Closes #1302
Closes #1303
Closes #1333

@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) breaking Breaks public-facing behaviour or API (apply alongside the type label) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review verdict on d104be9cf: NEEDS CHANGES (Sonnet reviewed; author Opus). I re-derived both blockers in the PR worktree.

1. make() writes bare CR/LF from raw_header. The "same values" test normalises with split(b'\r\n')/strip(), which hides lone \r and \n:

raw_header (with headers={'X':'a'}) written
(b'X: a\r',) X: a\r\r\n
(b'X: a\n',) X: a\n\r\n
(b'\nX: a',) \nX: a\r\n

The reviewer also got X: a\r\n \nInjected: b, which a lenient peer reads as a second field, while the parsed values carry no newline. Reject any CR or LF in a raw line other than CRLF followed by SP/HTAB (obs-fold). test_raw_lines_cannot_add_lines passes on main too, so it does not cover this path.

2. #1303 never checks the RFC 959 closing line. Per §4.2, a multi-line reply ends at a line with the same code followed by SP. Here everything after a code- first line is accepted:

  • 220-a\r\nfoo\r\n parses with no closing line.
  • 220-a\r\n221 b\r\n parses with the wrong code.
  • 220-a\r\n220 b\r\n331 x\r\n absorbs the next reply into lines.

Accept the segment only when it ends at the first <code> line, and treat anything after that as the next reply.

Not blocking: raw lines are matched to headers by position, so removing or inserting one header normalises every unedited header after it. Matching by content would keep them exact. make(lines=<str>) raises TypeError rather than ProtocolError.

Held up:

  • Edits: every edit (value, key case, version, status, reorder, add, remove) is built from the edit, and no stale raw line is written.
  • Unusual messages: 13 rebuild byte-exactly, covering tabs, an empty reason phrase, X : a, obs-fold with \t, and 0x80–0xFF octets.
  • Fixtures: info matches main once the new keys are dropped, and all 231 HTTP frame layers rebuild.
  • Revert proof: 31 failed / 5 passed against origin/main.
  • Tests: the leg is 906 tests, 0 failures.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
…ld octets, and parse whole multi-line FTP replies

- FTP requests keep the command token as received (`raw_cmmd`); `make`
  writes it whenever it differs from `cmmd` only in case (#1333).
- A multi-line FTP reply arriving whole in one segment parses when it ends
  at its first `<code> SP` line; the lines after the first are kept
  verbatim in `lines` and rebuilt (#1303).
- HTTP/1 keeps the start line (`receipt.raw_line`) and each field line,
  obs-folds included (`raw_header`); `make` writes them whenever they read
  back as the given values, matching field lines by content, so whitespace
  and non-UTF-8 field values rebuild byte-exactly (#1302, #1333). A raw
  line carrying a CR or LF its values lack raises `ProtocolError`.
  Parsed values are unchanged.

Closes #1302
Closes #1303
Closes #1333
@JarryShaw
JarryShaw force-pushed the fix/ftp-http1-raw-1302-1303-1333 branch from d104be9 to 5017fb9 Compare October 7, 2026 20:43
@JarryShaw

Copy link
Copy Markdown
Owner Author

Round-2 verdict on 5017fb94e: GOOD TO GO. I re-ran my round-1 probes against this head myself.

  • Injection is closed. raw_header of (b'X: a\r',), (b'X: a\n',) or (b'\nX: a',) now raises ProtocolError: raw line carries a CR or LF its values do not. This rule is narrower than the one I asked for: a CR or LF is allowed where the parsed value itself carries one. That is correct, because X: a\nb on the wire rebuilds byte-exactly on main, and rejecting every bare LF would have broken it. The obs-fold X: a\r\n b still rebuilds exactly.
  • RFC 959 §4.2 closing line enforced. 220-a\r\nfoo\r\n, 220-a\r\n221 b\r\n and 220-a\r\n220 b\r\n331 x\r\n all raise. 220-a\r\n220 b\r\n rebuilds exactly, and so does an intermediate line starting with digits (123 x). quit\r\n stays lower-case.
  • Content matching. Raw field lines now match headers by content, so removing or inserting one header keeps its unedited neighbours byte-exact.
  • Tests. The new tests fail 16 subtests on d104be9cf. The leg runs 910 tests with 0 failures, and all 471 HTTP capture layers match main apart from the new keys.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Coverage: 88.98% (unit tier, Python 3.14, 5017fb94e, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 4 93.55%
pcapkit/const 18777 1032 2348 843 90.27%
pcapkit/corekit 1949 84 606 20 94.91%
pcapkit/dumpkit 189 1 68 1 99.22%
pcapkit/foundation 2443 102 852 39 94.39%
pcapkit/interface 112 7 40 5 92.11%
pcapkit/protocols 16069 211 4208 178 98.04%
pcapkit/toolkit 539 74 168 3 84.58%
pcapkit/utilities 429 4 122 4 98.55%
pcapkit/vendor 4409 2359 1006 158 42.84%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw
JarryShaw merged commit 0a32b2f into main Oct 7, 2026
37 checks passed
@JarryShaw
JarryShaw deleted the fix/ftp-http1-raw-1302-1303-1333 branch October 7, 2026 23:00
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 7, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking Breaks public-facing behaviour or API (apply alongside the type label) fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

1 participant