Skip to content

fix(ftp,httpv1): FTP and HTTP/1 lines do not rebuild byte for byte - #1301

Merged
JarryShaw merged 1 commit into
mainfrom
fix/ftp-http1-roundtrip
Oct 7, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/ftp-http1-roundtrip

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner
  • Searched for similar pull requests
  • Followed the coding style (make pylint, make mypy, make isort) — ran pylint/mypy/isort on the touched files: no new findings (httpv1.py's isort complaint and the existing pylint design warnings predate this)
  • make test passes, and a test case covers the change — modules listed below, not the full suite
  • Added a changelog entry — N/A, added centrally

What is the purpose of your pull request?

  • fix — corrects a defect

Description

Closes #1238
Closes #1239
Closes #1240
Closes #1241

FTP make appends CRLF, omits the separator when args is None and after a reply hyphen (RFC 959 §4.2). Only the single SP is syntax, so USER x, 220- x and QUIT keep their bytes. \Z replaces $. For #1241, text that would not re-encode as UTF-8 records the detected charset (or latin-1 as a fallback). protocol.py's chardet decoding is unchanged, and the displayed str is the same as before.

Breaking: FTP Request/Response and HTTP/1 RequestHeader/ResponseHeader have a new required charset field. FTP args is None for argument-less lines. 220- Ready now parses to args=' Ready', which updates the existing pin in test_ftp_unit.py. make also accepts charset.

Probe: before the fix, USER anonymous\r\n raises ProtocolError, QUIT\r\n raises TypeError, 220-Welcome\r\n raises ProtocolError, and GET /caf\xe9 rebuilds as /caf\xc3\xa9. After it, all of them plus the latin-1 FTP and HTTP reason phrase rebuild byte-exactly. All 241 HTTP layers in examples/captures/* round-trip, both before and after.

Tests: test_ftp_http1_roundtrip_unit passes 8/8 with 16 subtests; with the fix reverted, 5 tests and 15 subtests fail (only the UTF-8 HTTP control passes). test_ftp_unit 10, test_http_unit 65, test_http_runtime 5, test_http_from_data_1154 4, test_option_roundtrip 6, tests/project 389 passed. The run_unittest_leg.py protocols --exclude protocols/internet leg ran 858 tests, OK.

Out of scope: HTTP/1 header field values are still re-encoded as UTF-8, and a multi-line FTP reply in one segment still fails to parse.

- FTP `make` ends every line with CRLF (#1238).
- A line with no arguments parses with `args=None`, and `make` writes no
  separator for it (#1239).
- A multi-line reply's text follows the hyphen immediately, per RFC 959
  section 4.2; only one SP is syntax elsewhere, the rest is argument (#1240).
- Text whose decoding does not encode back as UTF-8 records its charset in a
  new `charset` field on the FTP request/response and HTTP/1 start-line data,
  and `make` takes `charset` to re-encode with it (#1241).
- The FTP regexes anchor with `\Z`, so a trailing `\n` is no longer dropped.
@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) breaking Breaks public-facing behaviour or API (apply alongside the type label) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Coverage: 88.90% (unit tier, Python 3.14, 4a60fb89f, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 4 93.55%
pcapkit/const 18757 1034 2342 845 90.24%
pcapkit/corekit 1878 91 580 22 94.34%
pcapkit/dumpkit 142 0 42 0 100.00%
pcapkit/foundation 2442 102 852 39 94.38%
pcapkit/interface 112 7 40 5 92.11%
pcapkit/protocols 15802 196 4046 167 98.14%
pcapkit/toolkit 487 65 144 1 85.42%
pcapkit/utilities 429 4 122 4 98.55%
pcapkit/vendor 4409 2359 1006 158 42.84%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review verdict on 4a60fb89f: GOOD TO GO (Sonnet reviewed; the author ran on Opus).

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
@JarryShaw
JarryShaw merged commit c9676c5 into main Oct 7, 2026
41 checks passed
@JarryShaw
JarryShaw deleted the fix/ftp-http1-roundtrip branch October 7, 2026 19:10
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 7, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking Breaks public-facing behaviour or API (apply alongside the type label) fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

1 participant