Repository navigation
Conversation
Thread walks missed replies and scanned the whole lore table once per email. get_lore_emails_referencing() put the message-id into regexp_like() unescaped, so a reply that names an id containing `+` only in References was never found, and a `.` in an id matched any character. In a production lore table of 997,890 emails, 54,124 message-ids contain `+` and 955,356 contain `.`. Every call also scanned the `references` column of the whole table (about 170 ms there), once per thread member, so `lore -m <id> --thread` slowed with thread size. get_lore_emails_referencing_any() replaces it and takes a whole thread level, one query per 64 ids. Each query matches `in_reply_to IN (...)` or a literal contains(`references`, id) per id, with ids escaped only for the SQL string literal. Rows are then kept only where a header names an id as a whole id, so `<a@x>` does not match `<a@x.y>`. That match follows the headers as lore stores them: - It ignores the leading space a folded Message-ID or In-Reply-To header keeps. 16,878 stored message-ids have one; the old regex carried the space, so it found their replies only where the id was not the first entry in References, and the In-Reply-To equality found them only when the reply's header was folded too. - It accepts References entries separated by commas (727 rows have a comma after an id) or by nothing (`<a@x><b@x>`, 50 rows), which a split on whitespace would miss. Backslashes in an id are no longer doubled: the filter parser reads them literally, so the doubling broke the In-Reply-To equality for any id containing one. The four thread walks, two in lore_writers.rs and two in search.rs, were the same breadth-first walk, so they now share collect_thread_descendants(), which asks for each level at once. Within a level, emails keep the order the per-email walk gave them. Walking three threads of that table through pylance 4.0.0 with the old and new filters took 1.8 s -> 0.5 s (6 emails), 14.4 s -> 2.5 s (51) and 78.3 s -> 12.0 s (249), with the same emails in the same order. A level shares one scan of `references`, but each id still adds a containment pass, so a walk's cost still grows with thread size. Some cases go unmatched. Newly: 11 rows whose header glues a stored id to another character (`<<id>>`, `_<id>_`, `<id>;`), which the old substring regex matched and the whole-id match rejects. As before: a reply whose In-Reply-To names more than one id or carries a comment and whose References omits the parent (103 rows in that table: 77 name more than one id, 26 carry a comment or other text), and looking up the starting message by an id stored with a leading space. Tests in tests/lore_thread.rs cover `+` and `.` in ids, quotes and backslashes, the stored header forms above, a lookup split across queries, order within a level, and one lookup per thread level.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thread walks missed replies and scanned the whole lore table once per email. get_lore_emails_referencing() put the message-id into regexp_like() unescaped, so a reply that names an id containing
+only in References was never found, and a.in an id matched any character. In a production lore table of 997,890 emails, 54,124 message-ids contain+and 955,356 contain.. Every call also scanned thereferencescolumn of the whole table (about 170 ms there), once per thread member, solore -m <id> --threadslowed with thread size.get_lore_emails_referencing_any() replaces it and takes a whole thread level, one query per 64 ids. Each query matches
in_reply_to IN (...)or a literal contains(references, id) per id, with ids escaped only for the SQL string literal. Rows are then kept only where a header names an id as a whole id, so<a@x>does not match<a@x.y>. That match follows the headers as lore stores them:<a@x><b@x>, 50 rows), which a split on whitespace would miss.Backslashes in an id are no longer doubled: the filter parser reads them literally, so the doubling broke the In-Reply-To equality for any id containing one.
The four thread walks, two in lore_writers.rs and two in search.rs, were the same breadth-first walk, so they now share collect_thread_descendants(), which asks for each level at once. Within a level, emails keep the order the per-email walk gave them.
Walking three threads of that table through pylance 4.0.0 with the old and new filters took 1.8 s -> 0.5 s (6 emails), 14.4 s -> 2.5 s (51) and 78.3 s -> 12.0 s (249), with the same emails in the same order. A level shares one scan of
references, but each id still adds a containment pass, so a walk's cost still grows with thread size.Some cases go unmatched. Newly: 11 rows whose header glues a stored id to another character (
<<id>>,_<id>_,<id>;), which the old substring regex matched and the whole-id match rejects. As before: a reply whose In-Reply-To names more than one id or carries a comment and whose References omits the parent (103 rows in that table: 77 name more than one id, 26 carry a comment or other text), and looking up the starting message by an id stored with a leading space.Tests in tests/lore_thread.rs cover
+and.in ids, quotes and backslashes, the stored header forms above, a lookup split across queries, order within a level, and one lookup per thread level.Merging with #112 needs a one-line change in the test helper in
tests/lore_thread.rs, because #112 adds an argument toinsert_lore_emails(); whichever of the two lands second carries it.