Skip to content

lore: find thread replies by literal match, a level at a time - #121

Open
vladwing wants to merge 1 commit into
facebookexperimental:mainfrom
vladwing:lore-thread-lookup
Open

vladwing wants to merge 1 commit into
facebookexperimental:mainfrom
vladwing:lore-thread-lookup

Conversation

@vladwing

Copy link
Copy Markdown
Contributor

Thread walks missed replies and scanned the whole lore table once per email. get_lore_emails_referencing() put the message-id into regexp_like() unescaped, so a reply that names an id containing + only in References was never found, and a . in an id matched any character. In a production lore table of 997,890 emails, 54,124 message-ids contain + and 955,356 contain .. Every call also scanned the references column of the whole table (about 170 ms there), once per thread member, so lore -m <id> --thread slowed with thread size.

get_lore_emails_referencing_any() replaces it and takes a whole thread level, one query per 64 ids. Each query matches in_reply_to IN (...) or a literal contains(references, id) per id, with ids escaped only for the SQL string literal. Rows are then kept only where a header names an id as a whole id, so <a@x> does not match <a@x.y>. That match follows the headers as lore stores them:

  • It ignores the leading space a folded Message-ID or In-Reply-To header keeps. 16,878 stored message-ids have one; the old regex carried the space, so it found their replies only where the id was not the first entry in References, and the In-Reply-To equality found them only when the reply's header was folded too.
  • It accepts References entries separated by commas (727 rows have a comma after an id) or by nothing (<a@x><b@x>, 50 rows), which a split on whitespace would miss.

Backslashes in an id are no longer doubled: the filter parser reads them literally, so the doubling broke the In-Reply-To equality for any id containing one.

The four thread walks, two in lore_writers.rs and two in search.rs, were the same breadth-first walk, so they now share collect_thread_descendants(), which asks for each level at once. Within a level, emails keep the order the per-email walk gave them.

Walking three threads of that table through pylance 4.0.0 with the old and new filters took 1.8 s -> 0.5 s (6 emails), 14.4 s -> 2.5 s (51) and 78.3 s -> 12.0 s (249), with the same emails in the same order. A level shares one scan of references, but each id still adds a containment pass, so a walk's cost still grows with thread size.

Some cases go unmatched. Newly: 11 rows whose header glues a stored id to another character (<<id>>, _<id>_, <id>;), which the old substring regex matched and the whole-id match rejects. As before: a reply whose In-Reply-To names more than one id or carries a comment and whose References omits the parent (103 rows in that table: 77 name more than one id, 26 carry a comment or other text), and looking up the starting message by an id stored with a leading space.

Tests in tests/lore_thread.rs cover + and . in ids, quotes and backslashes, the stored header forms above, a lookup split across queries, order within a level, and one lookup per thread level.

Merging with #112 needs a one-line change in the test helper in tests/lore_thread.rs, because #112 adds an argument to insert_lore_emails(); whichever of the two lands second carries it.

Thread walks missed replies and scanned the whole lore table once per
email. get_lore_emails_referencing() put the message-id into
regexp_like() unescaped, so a reply that names an id containing `+`
only in References was never found, and a `.` in an id matched any
character. In a production lore table of 997,890 emails, 54,124
message-ids contain `+` and 955,356 contain `.`. Every call also
scanned the `references` column of the whole table (about 170 ms
there), once per thread member, so `lore -m <id> --thread` slowed with
thread size.

get_lore_emails_referencing_any() replaces it and takes a whole thread
level, one query per 64 ids. Each query matches `in_reply_to IN (...)`
or a literal contains(`references`, id) per id, with ids escaped only
for the SQL string literal. Rows are then kept only where a header names
an id as a whole id, so `<a@x>` does not match `<a@x.y>`. That match
follows the headers as lore stores them:

- It ignores the leading space a folded Message-ID or In-Reply-To
  header keeps. 16,878 stored message-ids have one; the old regex
  carried the space, so it found their replies only where the id was
  not the first entry in References, and the In-Reply-To equality
  found them only when the reply's header was folded too.
- It accepts References entries separated by commas (727 rows have a
  comma after an id) or by nothing (`<a@x><b@x>`, 50 rows), which a
  split on whitespace would miss.

Backslashes in an id are no longer doubled: the filter parser reads
them literally, so the doubling broke the In-Reply-To equality for any
id containing one.

The four thread walks, two in lore_writers.rs and two in search.rs,
were the same breadth-first walk, so they now share
collect_thread_descendants(), which asks for each level at once.
Within a level, emails keep the order the per-email walk gave them.

Walking three threads of that table through pylance 4.0.0 with the old
and new filters took 1.8 s -> 0.5 s (6 emails), 14.4 s -> 2.5 s (51) and
78.3 s -> 12.0 s (249), with the same emails in the same order. A level
shares one scan of `references`, but each id still adds a containment
pass, so a walk's cost still grows with thread size.

Some cases go unmatched. Newly: 11 rows whose header glues a stored id
to another character (`<<id>>`, `_<id>_`, `<id>;`), which the old
substring regex matched and the whole-id match rejects. As before: a
reply whose In-Reply-To names more than one id or carries a comment and
whose References omits the parent (103 rows in that table: 77 name more
than one id, 26 carry a comment or other text), and looking up the
starting message by an id stored with a leading space.

Tests in tests/lore_thread.rs cover `+` and `.` in ids, quotes and
backslashes, the stored header forms above, a lookup split across
queries, order within a level, and one lookup per thread level.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant