Repository navigation
Stream and parser performance - #368
Merged
Merged
Conversation
std::copy() and the string range constructor can only use memmove() when the source and destination types are the same. Copying between char and byte_t (unsigned char) made GCC generate byte-by-byte loops: - inputStreamStringAdapter::read() copied from string iterators to the byte_t buffer. Copy from a byte_t pointer to the string data instead. - parserInputStreamAdapter::extract() read into a temporary byte_t buffer, then built the string from it. Read directly into the string. extract() matters most: components which are only parsed from strings, such as the header, extract the rest of their part (the header of a body part is extracted with the body which follows it). Parsing a 20 MiB message with an attachment takes about 45% less time, and parsing a 1 MiB message nested 100 levels deep is about 10 times faster.
Header can only be parsed from a string, so parsing it from a stream extracted everything up to the end of the part, body included: the contents of each part were copied once more for each enclosing level, and parsing a message with a 50 MB attachment needed 50 MB of memory more than the message itself, which defeats the purpose of parsing from a stream. header::parseImpl() now only extracts up to the first empty line when parsing from a stream. The header parser never goes beyond it, so the result does not change. Parsing a 20 MB message with an attachment is about 2.5 times faster (9 ms instead of 23 ms).
maildirMessage::fetch(), when only the header is needed, reads the message file by blocks of 1 KB and stops at the empty line which ends the header. After each block, it searched for this empty line in all the data read so far, so the time grew with the square of the header size: fetching a 4 MB header took 20 s. Only search the new block (and the 2 bytes before it) instead. The search also recognizes an empty line after a line ending with a single LF. getParsedMessage() (maildir and POP3) extracted the message into a std::ostringstream, whose str() returned one more copy of it. Extract it directly into a string.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.