Conversation
Some servers hand back a message with the bare-LF line endings it was stored with. The MIME parser only recognises CRLF CRLF as the header/body separator, so such a body came back empty: the client showed the headers and nothing else. 2.2.1 added the same tolerance to TextMimeData, but a BODY[] literal arrives as bytes and goes through BinaryMimeData, which was not covered. The bytes cannot simply be decoded to a String to reuse that fix: the only charset that could be assumed at this point is wrong, because a message declares its charset per part in headers that have not been parsed yet. Decoding with Utf8Decoder(allowMalformed: true) turns every byte of a windows-1252 or latin-1 body into U+FFFD before the part's own charset= is ever read, so `Teší ma` would arrive as `Te ma` with no way to recover it. FetchParser therefore inserts the missing CR on the raw bytes, leaving every other byte exactly as the server sent it, and returns the input untouched (no copy) in the common conforming-CRLF case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Some servers hand back a message with the bare-LF line endings it was stored with. The MIME parser only recognises
\r\n\r\nas the header/body separator, so such a body came back empty: the client showed the headers and nothing else.2.2.1 (#272) added bare-LF tolerance to
TextMimeData, but aBODY[]literal arrives as bytes and goes throughBinaryMimeData, which that fix does not cover.Fix
FetchParser._parseBodyFullinserts the missing CR before each bare LF on the raw bytes before constructingBinaryMimeData. Every other byte is left exactly as the server sent it, and the input is returned untouched (no copy) in the common conforming-CRLF case, since this runs over every fetched body.The bytes deliberately are not decoded to a
Stringto reuse theTextMimeDatafix. The only charset that could be assumed at that point is wrong: a message declares its charset per part, in headers that have not been parsed yet. Decoding withUtf8Decoder(allowMalformed: true)turns every byte of a windows-1252 or latin-1 body into U+FFFD before the part's owncharset=is read, soTeší maarrives asTe mawith no way to recover it. (That was our first attempt, and the existing 8-bit tests caught it.)Tests
Four new tests in
fetch_parser_test.dart: a bare-LF body is not lost; a bare-LF windows-1252 body keeps its declared charset; a conforming CRLF message passes through untouched; a lone CR gains no LF. Three of the four fail againstmain.