Repository navigation
Search Outlook mail stored in .pst files (Windows first) #69
Description
Use case
On Windows, many people keep years of mail in Outlook data files (.pst): tens of thousands of messages, with attachments. Finding "the mail where the supplier announced a price increase" in them is exactly what DigUp is good at, and Outlook's own search is not. I'd like DigUp to search those mails the way it searches documents.
What it would look like
- A .pst inside a chosen folder is read like any other file. Each mail in it becomes a searchable item, like the pages of a PDF: its subject, sender, recipients, date, body (without the quoted history) and its attachments' text.
- A result shows the mail (subject, sender, date, folder in the PST, snippet). Enter opens it in Outlook. A PST message's EntryID can be computed from the file itself (4 zero bytes, the store's record key, the message's node id in little-endian), so no Outlook is needed to index, only to open.
- No Outlook, MAPI or COM to index: the file is read directly, read-only.
Reading PST files: MIT only
microsoft/outlook-pst-rs is Microsoft's own read-only, clean-room implementation of the MS-PST spec. It is MIT licensed and cross-platform, and it includes compressed RTF. libpff, the usual alternative, is LGPL, so I left it out.
It's Rust, like the Windows shell. Two ways it could fit, which is your call:
- a small helper (
digup pst-reador similar) that streams mails as JSON lines to the index service, the way the helpers already talk; - a static library with a C ABI that the engine links, like llama.cpp.
Things I found that shape it
- Outlook is reported to lock an open PST for exclusive access, so nothing else can read or copy it while Outlook runs (I'll confirm on Windows). DigUp would read it when Outlook is closed, and otherwise treat it as "can't read now, try again at the next sync".
- The file changes while Outlook has it open, even without new mail. Re-reading it is cheap (see below); what costs is embedding. Mails compared by node id and modification time, plus DigUp's usual checks, keep that to new or changed mails.
- Size and speed: a large PST is several GB and around 100,000 mails. I prototyped on public Enron PSTs (CC BY 3.0). A 189 MB PST with 7,289 items reads in 0.39 s: about 18,700 mails/s with bodies and attachment lists, and 11 MB peak memory. Indexing time is all embeddings, not reading.
- outlook-pst-rs 1.2.0 needs a small fix: it rejects messages that have no subnode tree, which MS-PST allows (a zero
bidSub). That was 47% of that PST. With that fix, already proposed upstream in feat: tolerate real-world archive quirks (recall fork) microsoft/outlook-pst-rs#62, it reads the same 7,229 mails as libpff, and the 60 failures are damaged blocks in the files themselves.
Questions
- Is mail search something you'd want in DigUp, or outside its direction?
- Where should PST reading live: a helper, or in the engine?
- How should a file holding thousands of items show up in results and in the index (one row per mail, grouped under the PST)?
- Windows only, or the Mac too (a PST copied over can be read there; opening a mail would then export it to .eml)?
I'm happy to do the work and to shape it the way you prefer. I can also share the prototype and the test PSTs.
Activity
njeanselme commentedon Oct 14, 2026
Follow-up, now that the Windows beta is out. I prototyped and checked things so the choice is easier; it's yours.
What I verified (Windows 11, classic Outlook 16.0)
GetItemFromIDopens the mail with them (checked on a PST that Outlook created).PR_BODY.digup estimateskips .pst, .eml, .msg, .xlsx, .xls, .pptx, .ods and .odp.Two ways to get there
Native PST in DigUp.
pst-read.exebesidedigup.exeinhelper\, found and run the way ffmpeg is.archive.pst::<EntryID>, so that results stay one per item.Export outside DigUp.
.emlper mail into a local folder that DigUp indexes. It updates only new or changed mails, and deletes the ones that are gone..emlreader, which is small, and also useful for Thunderbird or Apple Mail exports..htmlinstead would work with today's beta without any change, but results would then open in a browser.)Which direction fits DigUp, if either? I'm happy to do the work either way, and I can start with a standalone
.emlreader PR if that's useful on its own.leivorn commentedon Oct 16, 2026
Disclosure: I build Mailward, a browser-based PST/OST viewer, so I've been on the reading side of this. One thing worth knowing before "Windows first" settles on .pst only:
On Windows, a Microsoft 365 / Exchange account's mail usually lives in an .ost, not a .pst, and the current outlook-pst-rs rejects every .ost at the header.
UnicodeHeader::readandAnsiHeader::read(crates/pst/src/ndb/header.rs) requirewMagicClientto beSMbefore they readwVer; an OST carriesSOthere, soopen_store()fails withInvalid HEADER wMagicClient: 0x4F53for any .ost, whatever its page size. OSTs can also use the 4 KB-page variant of the format: headerwVer36 (0x24), which libpff documents as "64-bit Unicode format with 4k pages", first found in Outlook 2013 OSTs.NdbVersion::try_fromaccepts only 14–15 (ANSI) and 23 (Unicode), so a file marked as a PST (SM) with that version would stop one step later, withInvalidNdbVersion(36). For many people the .ost is the bigger mailbox, so it may be worth deciding early whether .ost is in scope and how the reader reports it.If 4K support gets added (here or upstream), one trap I hit in a different reader (pst-extractor 1.12.0): 4K files have larger, deflate-compressed data blocks, so a Table Context's row matrix can't be walked with the Unicode PST's 8,176-byte block arithmetic. In pst-extractor's own test OST (MIT), the contents table of the 160-item "United States holidays" folder is a single block that inflates to 54,240 bytes (160 rows × 339). Walked as 8 KB blocks, row 24 is read from offset 8,176 instead of 8,136, its row id is garbage, and the folder ends at 24 of 160 items without any error. Checking each row's
dwRowIDagainst the table's row index (the BTH keyed by row id) catches it. That OST is a ready-made 4K test file.What I can't tell you: whether EntryIDs computed from the file open OST items in Outlook the way they do for a PST (I haven't tested OSTs against Outlook), and I haven't had a real 4K file whose row matrix spans several blocks.
Edit (24 Oct): corrected the second paragraph. I first wrote that a 4 KB OST stops with
InvalidNdbVersion(36). The crate checkswMagicClientbeforewVer, so an .ost of any page size stops one step earlier, withInvalid HEADER wMagicClient: 0x4F53; I checked by replaying the header checks on pst-extractor's test OST (magic and partial CRC pass,wMagicClientisSO).InvalidNdbVersion(36)is what a file marked as a PST with 4 KB pages gets.Malcomle commentedon Oct 16, 2026
I tried the Apple Mail side of this on my Mac (0.6.1): a small
.emlxreader, plus letting the crawl go into Mail's.mboxfolders, which it skips today because they're packages.On my
~/Library/Mail(6 accounts, about 30,800 messages) the estimate first said 19 h. Most of that was the estimate itself: atsize / 2000, a 40 KB.emlxthat's mostly HTML and base64 counts as ~20 chunks. The rest came from mail that isn't worth searching. Gmail's Spam alone had 2,437 messages, the trash mailboxes another ~2,200, and Mail'sAttachmentsfolders held 1,600 PNGs, mostly logos and signatures.What I ended up doing: read only the message part of the
.emlx, prefer text/plain over HTML, cut the quoted thread ("On … wrote:", "Le … a écrit :",>lines) since every message is indexed on its own anyway, cap the body at about two chunks (one for mail withList-Unsubscribe), drop links, skip spam/trash/outbox mailboxes, and keep only PDFs among the attachments. That's 26,000 messages and 1.6 h, with ~2.35 chunks per message on a sample. "payment problem" finds the bank and billing mails without the exact words.Two things I didn't solve: results show
12345.emlxinstead of the subject, and~/Library/Mailneeds Full Disk Access. If Apple Mail fits where DigUp is going, I can open a PR on main with tests and the evals. Or take whatever helps.njeanselme commentedon Oct 16, 2026
Thanks both, this is useful.
@leivorn: agreed, and #88 reads .pst only, on purpose. Besides the 4K format (I checked: outlook-pst-rs stops at
wVer36), an .ost is Outlook's cache of a mailbox that lives on the server. Outlook keeps it open whenever it runs, so DigUp could rarely read it, and its items have Exchange EntryIDs rather than ones built from the node id, so opening them would need another way. The case #69 is about is archives kept as .pst. A 4K .pst would fail cleanly today (the PST is marked failed); a clearer message for it is an easy follow-up. Thanks for thedwRowIDcheck, worth passing to outlook-pst-rs if it adds 4K pages.@Malcomle: the subject-as-title part is solved in #88: a mail is a row at a path inside its file, named by its subject. That could carry over to .emlx if Apple Mail fits. Your other findings apply to PSTs too, and #88 doesn't do them yet: skipping Deleted Items and Junk, cutting quoted replies, and a lower cap on chunks per mail. On Enron (7,229 mails), #88 averages 2.06 chunks per mail, and its estimate is close (about 12,600 chunks estimated, 14,904 real), but each reply in a thread indexes the text it quotes again. Happy to add those to #88 if @ARahim3 wants them there.
ARahim3 commentedon Oct 21, 2026
Thanks @njeanselme, and @leivorn and @Malcomle for the details. I've read #88. Here's where I land: a mailbox is a different kind of source from a file (one file, thousands of items, locked while Outlook runs, another program to ship and keep up), and I don't want to take that on now. If mail comes to DigUp, the way that fits is one file per mail: an .eml/.emlx reader with the subject as the title, which also covers Thunderbird and Apple Mail exports, and a PST exported that way would be searchable with it. That's a later item, not a promise of when. Leaving this open for it.