diff --git a/CHANGELOG.md b/CHANGELOG.md index f387f42..c4cdf1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -138,6 +138,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- DOCX uses far less memory: body blocks are converted while `document.xml` is parsed and freed + right after, and the XML tree is compact (interned names and values, exact-size lists). A 1.5 MB + `document.xml` peaks at 3.9 MiB instead of 18.5 MiB, with half the allocations. Markdown + rendering allocates less for every format. +- HTML peak memory down about 35% and allocations about halved on large pages (DOM arena + pre-sized, inline node lists sized to their content, no copy of text before whitespace + collapsing). +- Spreadsheets make a third fewer allocations per cell (XLSX values are formatted without an + intermediate copy and trimmed in place). - **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, diff --git a/README.md b/README.md index 323a715..1f58e93 100644 --- a/README.md +++ b/README.md @@ -252,37 +252,36 @@ Other options: `include_hidden_sheets`, `allow_private_networks`, `allow_host`. ## Performance Measured with `just bench` (in-process, release build, 20 iterations after 3 warmup) on a -4-CPU Linux x86_64 VM, markkit 0.1.1. Allocations and peak heap are per single extraction. -Full report: [`bench-vm-linux-x86_64-1793058767.md`](https://github.com/everruns/markkit/blob/main/crates/markkit-bench/results/bench-vm-linux-x86_64-1793058767.md); history in +4-CPU Linux x86_64 VM. Allocations and peak heap are per single extraction; the last column +is the 0.1.1 baseline. Full report: [`bench-vm-linux-x86_64-1793070312.md`](https://github.com/everruns/markkit/blob/main/crates/markkit-bench/results/bench-vm-linux-x86_64-1793070312.md); history in [`crates/markkit-bench/results/`](https://github.com/everruns/markkit/blob/main/crates/markkit-bench/results/). -| Case | Input | Mean | Ops/s | Throughput | Allocs/op | Peak heap | -|------|------:|-----:|------:|-----------:|----------:|----------:| -| HTML article (fixture) | 881 B | 49 µs | 20,365 | 17 MiB/s | 213 | 28 KiB | -| HTML blog post with nav/scripts | 15 KiB | 641 µs | 1,561 | 23 MiB/s | 2,427 | 304 KiB | -| HTML long docs page | 300 KiB | 13.0 ms | 77 | 22 MiB/s | 47,578 | 7.5 MiB | -| HTML 50 tables | 257 KiB | 15.0 ms | 67 | 17 MiB/s | 50,349 | 8.6 MiB | -| HTML mostly boilerplate | 45 KiB | 706 µs | 1,417 | 63 MiB/s | 81 | 305 KiB | -| HTML 1000 nested divs | 11 KiB | 3.6 ms | 282 | 3 MiB/s | 442 | 147 KiB | -| DOCX memo | 1.4 KiB | 87 µs | 11,508 | 16 MiB/s | 733 | 93 KiB | -| DOCX 400-section report | 9 KiB (262 KiB out) | 35.7 ms | 28 | 0.2 MiB/s | 265,791 | 18.5 MiB | -| XLSX 50×5 | 2.8 KiB | 291 µs | 3,435 | 9 MiB/s | 1,668 | 129 KiB | -| XLSX 3 sheets, 150k cells | 766 KiB | 172 ms | 6 | 4 MiB/s | 782,800 | 24.9 MiB | -| ODS 2000×8 | 70 KiB | 23.2 ms | 43 | 3 MiB/s | 92,515 | 3.0 MiB | -| Format sniffing (DOCX) | 9 KiB | 0.9 µs | 1,141,227 | n/a | 0 | 0 B | +| Case | Input | Mean | Ops/s | Throughput | Allocs/op | Peak heap | Allocs / peak in 0.1.1 | +|------|------:|-----:|------:|-----------:|----------:|----------:|------:| +| HTML article (fixture) | 881 B | 54 µs | 18,375 | 15 MiB/s | 158 | 22 KiB | 213 / 28 KiB | +| HTML blog post with nav/scripts | 15 KiB | 690 µs | 1,449 | 22 MiB/s | 2,505 | 241 KiB | 2,427 / 304 KiB | +| HTML long docs page | 300 KiB | 15.1 ms | 66 | 19 MiB/s | 49,342 | 4.7 MiB | 47,578 / 7.5 MiB | +| HTML 50 tables | 257 KiB | 18.3 ms | 55 | 14 MiB/s | 33,635 | 5.1 MiB | 50,349 / 8.6 MiB | +| HTML mostly boilerplate | 45 KiB | 704 µs | 1,420 | 63 MiB/s | 66 | 252 KiB | 81 / 305 KiB | +| HTML 1000 nested divs | 11 KiB | 512 µs | 1,953 | 20 MiB/s | 551 | 52 KiB | 442 / 147 KiB | +| DOCX memo | 1.4 KiB | 75 µs | 13,401 | 18 MiB/s | 499 | 59 KiB | 733 / 93 KiB | +| DOCX 400-section report | 9 KiB (262 KiB out) | 22.3 ms | 45 | 0.4 MiB/s | 136,866 | 3.9 MiB | 265,791 / 18.5 MiB | +| XLSX 50×5 | 2.8 KiB | 301 µs | 3,321 | 9 MiB/s | 1,165 | 129 KiB | 1,668 / 129 KiB | +| XLSX 3 sheets, 150k cells | 766 KiB | 172 ms | 6 | 4 MiB/s | 512,786 | 21.5 MiB | 782,800 / 24.9 MiB | +| ODS 2000×8 | 70 KiB | 24.5 ms | 41 | 3 MiB/s | 72,498 | 3.0 MiB | 92,515 / 3.0 MiB | +| Format sniffing (DOCX) | 9 KiB | 0.9 µs | 1,076,774 | n/a | 0 | 0 B | 0 / 0 B | Takeaways: -- **HTML** extracts at 17–25 MiB/s; peak heap is ~25x the input size. +- **HTML** extracts at 14–22 MiB/s with peak heap 16–20x the input (was ~25x). +- **Deep nesting is linear**: 1000 nested `
`s take 0.5 ms (was 3.6 ms), and 64k nested + `
`s take 2.4 ms (was 12.6 s). - **Boilerplate-heavy pages are cheap**: scripts, styles and nav are dropped at ~60 MiB/s. -- **Deep nesting is the slow outlier**: 1000 nested `
`s run at 3 MiB/s with few - allocations, so the cost is CPU in tree building, not memory. -- **Office formats allocate most**: a 150k-cell XLSX takes 172 ms and ~5 allocations per cell; - a 9 KiB compressed DOCX peaks at 18.5 MiB of heap (~2000x its compressed size). -- **Format detection** is effectively free (under 1 µs, no allocations). - -Timings on a shared VM are noisy (stddev up to ~70% on the smallest cases); allocation counts -are deterministic and the better regression signal. Compare runs from the same machine only. +- **DOCX** streams the document body: a 400-section report peaks at 3.9 MiB (was 18.5 MiB) with + half the allocations. +- **Spreadsheets** cost about 3.4 allocations per cell (was 5.2); one is calamine's and two are + the cell's text and its table node, so peak heap now tracks the size of the output table. +- **Format detection** is effectively free (microseconds, no allocations). ## Security diff --git a/crates/markkit-bench/README.md b/crates/markkit-bench/README.md index 2429931..7fea9c9 100644 --- a/crates/markkit-bench/README.md +++ b/crates/markkit-bench/README.md @@ -27,22 +27,23 @@ same moniker only. Commit a fresh run with performance-relevant PRs. Design and metric definitions: `knowledge/operations/performance-results.md`. -## Latest baseline (vm, linux x86_64, 4 CPUs, 2026-10-26) +## Latest baseline (vm, linux x86_64, 4 CPUs, 2026-10-27) 20 iterations, 3 warmup, release. Full report: -[`bench-vm-linux-x86_64-1793058767.md`](results/bench-vm-linux-x86_64-1793058767.md). - -| Case | Input | Mean | Ops/s | Throughput | Allocs/op | Peak heap | -|------|------:|-----:|------:|-----------:|----------:|----------:| -| HTML article (fixture) | 881 B | 49 µs | 20,365 | 17 MiB/s | 213 | 28 KiB | -| HTML blog post with nav/scripts | 15 KiB | 641 µs | 1,561 | 23 MiB/s | 2,427 | 304 KiB | -| HTML long docs page | 300 KiB | 13.0 ms | 77 | 22 MiB/s | 47,578 | 7.5 MiB | -| HTML 50 tables | 257 KiB | 15.0 ms | 67 | 17 MiB/s | 50,349 | 8.6 MiB | -| HTML mostly boilerplate | 45 KiB | 706 µs | 1,417 | 63 MiB/s | 81 | 305 KiB | -| HTML 1000 nested divs | 11 KiB | 3.6 ms | 282 | 3 MiB/s | 442 | 147 KiB | -| DOCX memo | 1.4 KiB | 87 µs | 11,508 | 16 MiB/s | 733 | 93 KiB | -| DOCX 400-section report | 9 KiB (262 KiB out) | 35.7 ms | 28 | 0.2 MiB/s | 265,791 | 18.5 MiB | -| XLSX 50×5 | 2.8 KiB | 291 µs | 3,435 | 9 MiB/s | 1,668 | 129 KiB | -| XLSX 3 sheets, 150k cells | 766 KiB | 172 ms | 6 | 4 MiB/s | 782,800 | 24.9 MiB | -| ODS 2000×8 | 70 KiB | 23.2 ms | 43 | 3 MiB/s | 92,515 | 3.0 MiB | -| Format sniffing (DOCX) | 9 KiB | 0.9 µs | 1,141,227 | n/a | 0 | 0 B | +[`bench-vm-linux-x86_64-1793070312.md`](results/bench-vm-linux-x86_64-1793070312.md). +Last column: the 0.1.1 baseline ([`bench-vm-linux-x86_64-1793058767.md`](results/bench-vm-linux-x86_64-1793058767.md)). + +| Case | Input | Mean | Ops/s | Throughput | Allocs/op | Peak heap | Allocs / peak in 0.1.1 | +|------|------:|-----:|------:|-----------:|----------:|----------:|------:| +| HTML article (fixture) | 881 B | 54 µs | 18,375 | 15 MiB/s | 158 | 22 KiB | 213 / 28 KiB | +| HTML blog post with nav/scripts | 15 KiB | 690 µs | 1,449 | 22 MiB/s | 2,505 | 241 KiB | 2,427 / 304 KiB | +| HTML long docs page | 300 KiB | 15.1 ms | 66 | 19 MiB/s | 49,342 | 4.7 MiB | 47,578 / 7.5 MiB | +| HTML 50 tables | 257 KiB | 18.3 ms | 55 | 14 MiB/s | 33,635 | 5.1 MiB | 50,349 / 8.6 MiB | +| HTML mostly boilerplate | 45 KiB | 704 µs | 1,420 | 63 MiB/s | 66 | 252 KiB | 81 / 305 KiB | +| HTML 1000 nested divs | 11 KiB | 512 µs | 1,953 | 20 MiB/s | 551 | 52 KiB | 442 / 147 KiB | +| DOCX memo | 1.4 KiB | 75 µs | 13,401 | 18 MiB/s | 499 | 59 KiB | 733 / 93 KiB | +| DOCX 400-section report | 9 KiB (262 KiB out) | 22.3 ms | 45 | 0.4 MiB/s | 136,866 | 3.9 MiB | 265,791 / 18.5 MiB | +| XLSX 50×5 | 2.8 KiB | 301 µs | 3,321 | 9 MiB/s | 1,165 | 129 KiB | 1,668 / 129 KiB | +| XLSX 3 sheets, 150k cells | 766 KiB | 172 ms | 6 | 4 MiB/s | 512,786 | 21.5 MiB | 782,800 / 24.9 MiB | +| ODS 2000×8 | 70 KiB | 24.5 ms | 41 | 3 MiB/s | 72,498 | 3.0 MiB | 92,515 / 3.0 MiB | +| Format sniffing (DOCX) | 9 KiB | 0.9 µs | 1,076,774 | n/a | 0 | 0 B | 0 / 0 B | diff --git a/crates/markkit-bench/results/bench-vm-linux-x86_64-1793070312.json b/crates/markkit-bench/results/bench-vm-linux-x86_64-1793070312.json new file mode 100644 index 0000000..0e9a2c6 --- /dev/null +++ b/crates/markkit-bench/results/bench-vm-linux-x86_64-1793070312.json @@ -0,0 +1,522 @@ +{ + "timestamp": 1793070312, + "system": { + "moniker": "vm-linux-x86_64", + "hostname": "vm", + "os": "linux", + "arch": "x86_64", + "cpus": 4, + "markkit_version": "0.1.1", + "git_commit": "50076076e78a", + "profile": "release" + }, + "iterations": 20, + "warmup": 3, + "peak_rss_bytes": 37728256, + "results": [ + { + "name": "html_fixture_article", + "category": "html", + "description": "Golden fixture full_article.html", + "iterations": 20, + "input_bytes": 881, + "output_bytes": 315, + "times_ns": [ + 56391, + 53838, + 52929, + 51967, + 51485, + 51612, + 51576, + 51281, + 51323, + 51687, + 51400, + 51509, + 58255, + 51618, + 53024, + 53052, + 79138, + 61426, + 52955, + 51956 + ], + "mean_ns": 54421.1, + "median_ns": 51961.5, + "stddev_ns": 6229.844884906846, + "min_ns": 51281, + "max_ns": 79138, + "ops_per_sec": 18375.22578558684, + "throughput_mib_s": 15.438627163984306, + "allocations": 158, + "allocated_bytes": 35167, + "peak_heap_bytes": 22099, + "errors": 0, + "error_messages": [] + }, + { + "name": "html_blog_post", + "category": "html", + "description": "Article with nav/footer/scripts, 20 sections", + "iterations": 20, + "input_bytes": 15685, + "output_bytes": 12030, + "times_ns": [ + 678159, + 635396, + 957858, + 682138, + 635035, + 701317, + 657555, + 632264, + 941074, + 695342, + 700757, + 646748, + 653105, + 652368, + 630161, + 665557, + 652438, + 672512, + 661595, + 652285 + ], + "mean_ns": 690183.2, + "median_ns": 659575.0, + "stddev_ns": 89000.25739996486, + "min_ns": 630161, + "max_ns": 957858, + "ops_per_sec": 1448.8906713463903, + "throughput_mib_s": 21.67305963522733, + "allocations": 2505, + "allocated_bytes": 345159, + "peak_heap_bytes": 247097, + "errors": 0, + "error_messages": [] + }, + { + "name": "html_docs_page", + "category": "html", + "description": "Long docs page, 400 sections, no chrome", + "iterations": 20, + "input_bytes": 306808, + "output_bytes": 240691, + "times_ns": [ + 14608732, + 14494204, + 17473183, + 14701334, + 15692695, + 13684670, + 16405998, + 15737469, + 15483821, + 14030462, + 13941592, + 14491759, + 14236963, + 14448286, + 14252603, + 14258596, + 14371443, + 13950370, + 14075038, + 21460969 + ], + "mean_ns": 15090009.35, + "median_ns": 14470022.5, + "stddev_ns": 1729556.7873620188, + "min_ns": 13684670, + "max_ns": 21460969, + "ops_per_sec": 66.26901129123556, + "throughput_mib_s": 19.38997537254467, + "allocations": 49342, + "allocated_bytes": 6969184, + "peak_heap_bytes": 4961744, + "errors": 0, + "error_messages": [] + }, + { + "name": "html_tables", + "category": "html", + "description": "50 tables x 40 rows x 6 cols", + "iterations": 20, + "input_bytes": 262766, + "output_bytes": 174139, + "times_ns": [ + 17871537, + 17942627, + 17245498, + 18195817, + 18666446, + 19009430, + 18142024, + 18572852, + 18053392, + 18418790, + 18228648, + 18450218, + 18631919, + 17924290, + 18421024, + 17943492, + 17724502, + 19561773, + 18513291, + 17865419 + ], + "mean_ns": 18269149.45, + "median_ns": 18212232.5, + "stddev_ns": 490170.7115306335, + "min_ns": 17245498, + "max_ns": 19561773, + "ops_per_sec": 54.73708574867452, + "throughput_mib_s": 13.716740678631028, + "allocations": 33635, + "allocated_bytes": 7111395, + "peak_heap_bytes": 5315858, + "errors": 0, + "error_messages": [] + }, + { + "name": "html_boilerplate", + "category": "html", + "description": "Mostly script/style/nav to strip, little content", + "iterations": 20, + "input_bytes": 46312, + "output_bytes": 191, + "times_ns": [ + 688856, + 706404, + 693015, + 678654, + 753383, + 695389, + 687722, + 670547, + 699625, + 708990, + 709207, + 690169, + 692903, + 674445, + 686865, + 882798, + 700335, + 672732, + 694837, + 697011 + ], + "mean_ns": 704194.35, + "median_ns": 693926.0, + "stddev_ns": 44463.30638546238, + "min_ns": 670547, + "max_ns": 882798, + "ops_per_sec": 1420.062515412116, + "throughput_mib_s": 62.71928330780593, + "allocations": 66, + "allocated_bytes": 264364, + "peak_heap_bytes": 257958, + "errors": 0, + "error_messages": [] + }, + { + "name": "html_deep_nesting", + "category": "html", + "description": "Divs nested past the depth limit (1000 levels)", + "iterations": 20, + "input_bytes": 11011, + "output_bytes": 5, + "times_ns": [ + 505734, + 479007, + 496965, + 481487, + 581373, + 517735, + 498138, + 484429, + 500713, + 473105, + 499351, + 478692, + 592757, + 541053, + 502114, + 479752, + 631782, + 475017, + 505177, + 515173 + ], + "mean_ns": 511977.7, + "median_ns": 500032.0, + "stddev_ns": 41992.77829591654, + "min_ns": 473105, + "max_ns": 631782, + "ops_per_sec": 1953.2100714542842, + "throughput_mib_s": 20.51047906568825, + "allocations": 551, + "allocated_bytes": 93075, + "peak_heap_bytes": 53331, + "errors": 0, + "error_messages": [] + }, + { + "name": "docx_memo", + "category": "docx", + "description": "Short memo: heading, 10 paragraphs, 1 list", + "iterations": 20, + "input_bytes": 1429, + "output_bytes": 636, + "times_ns": [ + 78486, + 75616, + 72508, + 105005, + 75727, + 71660, + 70384, + 70504, + 70063, + 69006, + 69191, + 75537, + 69611, + 69807, + 69503, + 69097, + 93274, + 74533, + 72542, + 70321 + ], + "mean_ns": 74618.75, + "median_ns": 71082.0, + "stddev_ns": 8794.424721805288, + "min_ns": 69006, + "max_ns": 105005, + "ops_per_sec": 13401.457408493176, + "throughput_mib_s": 18.26351417230296, + "allocations": 499, + "allocated_bytes": 208772, + "peak_heap_bytes": 60705, + "errors": 0, + "error_messages": [] + }, + { + "name": "docx_report", + "category": "docx", + "description": "Report: 400 sections with lists and tables", + "iterations": 20, + "input_bytes": 9299, + "output_bytes": 268569, + "times_ns": [ + 19723549, + 19264029, + 19663660, + 28980688, + 35737368, + 28498262, + 19866797, + 19612692, + 20332627, + 29235752, + 20082434, + 19560911, + 23568320, + 20981933, + 19931665, + 20027242, + 20757657, + 21117495, + 19290237, + 19337913 + ], + "mean_ns": 22278561.55, + "median_ns": 20054838.0, + "stddev_ns": 4470329.341242985, + "min_ns": 19264029, + "max_ns": 35737368, + "ops_per_sec": 44.886201371470506, + "throughput_mib_s": 0.39806059508638786, + "allocations": 136866, + "allocated_bytes": 12535863, + "peak_heap_bytes": 4096657, + "errors": 0, + "error_messages": [] + }, + { + "name": "xlsx_small", + "category": "spreadsheet", + "description": "1 sheet, 50 rows x 5 cols", + "iterations": 20, + "input_bytes": 2865, + "output_bytes": 2723, + "times_ns": [ + 335953, + 297172, + 328670, + 283776, + 286168, + 312018, + 288700, + 287661, + 291075, + 314857, + 291192, + 283531, + 310292, + 332756, + 287941, + 302848, + 284210, + 285175, + 289087, + 329330 + ], + "mean_ns": 301120.6, + "median_ns": 291133.5, + "stddev_ns": 17897.54023434506, + "min_ns": 283531, + "max_ns": 335953, + "ops_per_sec": 3320.928558192299, + "throughput_mib_s": 9.073696440907419, + "allocations": 1165, + "allocated_bytes": 666307, + "peak_heap_bytes": 132231, + "errors": 0, + "error_messages": [] + }, + { + "name": "xlsx_large", + "category": "spreadsheet", + "description": "3 sheets, 5000 rows x 10 cols each (150k cells)", + "iterations": 20, + "input_bytes": 784216, + "output_bytes": 1844036, + "times_ns": [ + 188676257, + 187679344, + 162135681, + 162019860, + 163153416, + 162607970, + 167529742, + 175896649, + 192586659, + 165677263, + 188200749, + 174994360, + 164653608, + 165030395, + 164757980, + 164647239, + 162814779, + 165140734, + 202335115, + 162396829 + ], + "mean_ns": 172146731.45, + "median_ns": 165085564.5, + "stddev_ns": 12268224.876868717, + "min_ns": 162019860, + "max_ns": 202335115, + "ops_per_sec": 5.808997891374139, + "throughput_mib_s": 4.34447201765238, + "allocations": 512786, + "allocated_bytes": 39383548, + "peak_heap_bytes": 22510370, + "errors": 0, + "error_messages": [] + }, + { + "name": "ods_medium", + "category": "spreadsheet", + "description": "1 sheet, 2000 rows x 8 cols", + "iterations": 20, + "input_bytes": 71489, + "output_bytes": 166354, + "times_ns": [ + 20929141, + 23527772, + 21070397, + 37198268, + 38767485, + 24290114, + 21781735, + 20785127, + 21101834, + 22763499, + 21865093, + 21887328, + 24175539, + 22247397, + 23571586, + 28480754, + 22005846, + 29140881, + 22079017, + 22346256 + ], + "mean_ns": 24500753.45, + "median_ns": 22296826.5, + "stddev_ns": 4999884.1172604645, + "min_ns": 20785127, + "max_ns": 38767485, + "ops_per_sec": 40.81507134222438, + "throughput_mib_s": 2.7826582290499484, + "allocations": 72498, + "allocated_bytes": 9342742, + "peak_heap_bytes": 3180660, + "errors": 0, + "error_messages": [] + }, + { + "name": "detect_docx_magic", + "category": "detect", + "description": "Sniff DOCX from bytes without a hint", + "iterations": 20, + "input_bytes": 9299, + "output_bytes": 0, + "times_ns": [ + 978, + 939, + 935, + 925, + 918, + 909, + 908, + 943, + 936, + 911, + 910, + 929, + 921, + 924, + 924, + 929, + 926, + 928, + 939, + 942 + ], + "mean_ns": 928.7, + "median_ns": 927.0, + "stddev_ns": 15.550241155686301, + "min_ns": 908, + "max_ns": 978, + "ops_per_sec": 1076773.985140519, + "throughput_mib_s": 9549.065864392935, + "allocations": 0, + "allocated_bytes": 0, + "peak_heap_bytes": 0, + "errors": 0, + "error_messages": [] + } + ] +} diff --git a/crates/markkit-bench/results/bench-vm-linux-x86_64-1793070312.md b/crates/markkit-bench/results/bench-vm-linux-x86_64-1793070312.md new file mode 100644 index 0000000..eea0522 --- /dev/null +++ b/crates/markkit-bench/results/bench-vm-linux-x86_64-1793070312.md @@ -0,0 +1,50 @@ +# Markkit Benchmark Report + +## System Information + +- **Moniker**: `vm-linux-x86_64` +- **OS / Arch**: linux / x86_64 +- **CPUs**: 4 +- **markkit**: 0.1.1 (`50076076e78a`) +- **Profile**: release +- **Timestamp**: 1793070312 +- **Iterations**: 20 (warmup 3) +- **Peak RSS**: 35.98 MiB + +## Summary + +12 cases, 254.623 ms total mean time per pass, 0 errors. + +Allocations, allocated and peak heap are per single operation. Throughput is input bytes over mean time. + +### HTML + +| Benchmark | Input | Output | Mean | Median | StdDev | Ops/s | MiB/s | Allocs | Allocated | Peak heap | Errors | +|-----------|------:|-------:|-----:|-------:|-------:|------:|------:|-------:|----------:|----------:|-------:| +| html_fixture_article | 881 B | 315 B | 54.42 µs | 51.96 µs | ±6.23 µs | 18375 | 15.4 | 158 | 34.3 KiB | 21.6 KiB | - | +| html_blog_post | 15.3 KiB | 11.7 KiB | 690.18 µs | 659.58 µs | ±89.00 µs | 1449 | 21.7 | 2505 | 337.1 KiB | 241.3 KiB | - | +| html_docs_page | 299.6 KiB | 235.0 KiB | 15.09 ms | 14.47 ms | ±1.73 ms | 66 | 19.4 | 49342 | 6.65 MiB | 4.73 MiB | - | +| html_tables | 256.6 KiB | 170.1 KiB | 18.27 ms | 18.21 ms | ±490.17 µs | 55 | 13.7 | 33635 | 6.78 MiB | 5.07 MiB | - | +| html_boilerplate | 45.2 KiB | 191 B | 704.19 µs | 693.93 µs | ±44.46 µs | 1420 | 62.7 | 66 | 258.2 KiB | 251.9 KiB | - | +| html_deep_nesting | 10.8 KiB | 5 B | 511.98 µs | 500.03 µs | ±41.99 µs | 1953 | 20.5 | 551 | 90.9 KiB | 52.1 KiB | - | + +### DOCX + +| Benchmark | Input | Output | Mean | Median | StdDev | Ops/s | MiB/s | Allocs | Allocated | Peak heap | Errors | +|-----------|------:|-------:|-----:|-------:|-------:|------:|------:|-------:|----------:|----------:|-------:| +| docx_memo | 1.4 KiB | 636 B | 74.62 µs | 71.08 µs | ±8.79 µs | 13401 | 18.3 | 499 | 203.9 KiB | 59.3 KiB | - | +| docx_report | 9.1 KiB | 262.3 KiB | 22.28 ms | 20.05 ms | ±4.47 ms | 45 | 0.4 | 136866 | 11.96 MiB | 3.91 MiB | - | + +### Spreadsheet + +| Benchmark | Input | Output | Mean | Median | StdDev | Ops/s | MiB/s | Allocs | Allocated | Peak heap | Errors | +|-----------|------:|-------:|-----:|-------:|-------:|------:|------:|-------:|----------:|----------:|-------:| +| xlsx_small | 2.8 KiB | 2.7 KiB | 301.12 µs | 291.13 µs | ±17.90 µs | 3321 | 9.1 | 1165 | 650.7 KiB | 129.1 KiB | - | +| xlsx_large | 765.8 KiB | 1.76 MiB | 172.15 ms | 165.09 ms | ±12.27 ms | 6 | 4.3 | 512786 | 37.56 MiB | 21.47 MiB | - | +| ods_medium | 69.8 KiB | 162.5 KiB | 24.50 ms | 22.30 ms | ±5.00 ms | 41 | 2.8 | 72498 | 8.91 MiB | 3.03 MiB | - | + +### Detect + +| Benchmark | Input | Output | Mean | Median | StdDev | Ops/s | MiB/s | Allocs | Allocated | Peak heap | Errors | +|-----------|------:|-------:|-----:|-------:|-------:|------:|------:|-------:|----------:|----------:|-------:| +| detect_docx_magic | 9.1 KiB | 0 B | 929 ns | 927 ns | ±16 ns | 1076774 | 9549.1 | 0 | 0 B | 0 B | - | diff --git a/crates/markkit/src/archive.rs b/crates/markkit/src/archive.rs index 3166a02..10616fb 100644 --- a/crates/markkit/src/archive.rs +++ b/crates/markkit/src/archive.rs @@ -97,10 +97,16 @@ impl<'a> ZipGuard<'a> { } } }; - let mut buf = Vec::new(); - let n = { + let (n, buf) = { let mut entry = self.archive.by_index(index)?; - copy_capped(&mut entry, self.max_entry, name, &mut buf)? + // The declared size is only a capacity hint (saves growth copies and + // slack); it is clamped, and the real limit is still `copy_capped`. + let hint = entry.size().min(self.max_entry).min(MAX_SIZE_HINT); + let mut buf = Vec::with_capacity(usize::try_from(hint).unwrap_or(0)); + ( + copy_capped(&mut entry, self.max_entry, name, &mut buf)?, + buf, + ) }; self.add_total(n)?; Ok(Some(buf)) @@ -118,6 +124,9 @@ impl<'a> ZipGuard<'a> { } } +/// Largest capacity reserved up front from a (lying) declared entry size. +const MAX_SIZE_HINT: u64 = 16 * 1024 * 1024; + fn copy_capped( reader: &mut R, cap: u64, diff --git a/crates/markkit/src/docx.rs b/crates/markkit/src/docx.rs index d68189c..9a56220 100644 --- a/crates/markkit/src/docx.rs +++ b/crates/markkit/src/docx.rs @@ -18,7 +18,9 @@ //! - Skipped: deleted text, field instructions, drawings/text boxes, comments, //! headers/footers, footnotes. Macros (`vbaProject.bin`) are never read. +use std::borrow::Cow; use std::collections::HashMap; +use std::rc::Rc; use crate::archive::{ArchiveError, ZipGuard}; use crate::config::MarkkitBuilder; @@ -72,6 +74,22 @@ impl Parts<'_, '_> { None => Ok(None), } } + + /// Read a part, passing each child of the `split` element path to `sink` + /// as soon as it is complete (see [`xml::parse_split`]). + fn read_xml_split( + &mut self, + path: &str, + split: &[&str], + sink: &mut dyn FnMut(Element), + ) -> Result, Error> { + match self.zip.read(path).map_err(archive_err)? { + Some(bytes) => xml::parse_split(&bytes, self.max_depth, self.budget, split, sink) + .map(Some) + .map_err(|e| xml_err(path, e)), + None => Ok(None), + } + } } // ─── Styles ───────────────────────────────────────────────────────── @@ -309,13 +327,13 @@ impl Ctx<'_> { .map(|a| format!("#{}", a.replace(' ', "%20"))) } - fn collect_segs(&self, el: &Element, link: Option<&str>, segs: &mut Vec) { + fn collect_segs<'e>(&self, el: &'e Element, link: Option<&Rc>, segs: &mut Vec>) { for child in el.elements() { match child.name.as_str() { "r" => self.run_segs(child, link, segs), "hyperlink" => { - let url = self.hyperlink_url(child); - self.collect_segs(child, url.as_deref().or(link), segs); + let url = self.hyperlink_url(child).map(Rc::from); + self.collect_segs(child, url.as_ref().or(link), segs); } "sdt" => { if let Some(content) = child.child("sdtContent") { @@ -330,15 +348,15 @@ impl Ctx<'_> { } } - fn run_segs(&self, run: &Element, link: Option<&str>, segs: &mut Vec) { + fn run_segs<'e>(&self, run: &'e Element, link: Option<&Rc>, segs: &mut Vec>) { let rpr = run.child("rPr"); if !self.include_hidden && toggle_on(rpr.and_then(|r| r.child("vanish"))) { self.hidden_runs.set(self.hidden_runs.get() + 1); return; } let fmt = run_fmt(rpr); - let link = link.map(str::to_string); - let push_text = |segs: &mut Vec, t: String| { + let link = link.cloned(); + let push_text = |segs: &mut Vec>, t: Cow<'e, str>| { segs.push(Seg::Text { text: t, fmt, @@ -348,8 +366,8 @@ impl Ctx<'_> { for c in run.elements() { match c.name.as_str() { "t" => push_text(segs, c.text()), - "tab" | "ptab" => push_text(segs, " ".to_string()), - "noBreakHyphen" => push_text(segs, "-".to_string()), + "tab" | "ptab" => push_text(segs, Cow::Borrowed(" ")), + "noBreakHyphen" => push_text(segs, Cow::Borrowed("-")), "br" | "cr" => match c.attr("type") { Some("page") | Some("column") => {} _ => segs.push(Seg::Break { link: link.clone() }), @@ -411,29 +429,33 @@ impl Ctx<'_> { if self.budget.expired() { return; } - match child.name.as_str() { - "p" => { - let index = body.next_index; - body.next_index += 1; - self.paragraph(child, body, index); - } - "tbl" => { - let index = body.next_index; - body.next_index += 1; - body.flush_list(); - if let Some(t) = self.table(child) { - body.tables += 1; - body.push(t, index); - } + self.block(child, body); + } + } + + fn block(&self, child: &Element, body: &mut Body) { + match child.name.as_str() { + "p" => { + let index = body.next_index; + body.next_index += 1; + self.paragraph(child, body, index); + } + "tbl" => { + let index = body.next_index; + body.next_index += 1; + body.flush_list(); + if let Some(t) = self.table(child) { + body.tables += 1; + body.push(t, index); } - "sdt" => { - if let Some(content) = child.child("sdtContent") { - self.blocks(content, body); - } + } + "sdt" => { + if let Some(content) = child.child("sdtContent") { + self.blocks(content, body); } - "ins" | "moveTo" | "customXml" | "smartTag" => self.blocks(child, body), - _ => {} } + "ins" | "moveTo" | "customXml" | "smartTag" => self.blocks(child, body), + _ => {} } } @@ -617,9 +639,6 @@ pub(crate) fn parse( .map(|t| resolve_target("", t)) .unwrap_or_else(|| "docProps/core.xml".to_string()); - let main = zip - .read_xml(&main_path)? - .ok_or_else(|| Error::DocxExtract("missing main document part".into()))?; let (dir, file) = split_dir(&main_path); let rels = parse_rels(zip.read_xml(&format!("{dir}/_rels/{file}.rels"))?); @@ -638,13 +657,6 @@ pub(crate) fn parse( .and_then(|c| c.find("title").map(|t| t.text().trim().to_string())) .filter(|t| !t.is_empty()); - let document = main - .child("document") - .ok_or_else(|| Error::DocxExtract("main part is not a w:document".into()))?; - let body_el = document - .child("body") - .ok_or_else(|| Error::DocxExtract("document has no body".into()))?; - // Hyperlinks only resolve through external relationships or explicit targets. let ctx = Ctx { rels: &rels, @@ -658,7 +670,19 @@ pub(crate) fn parse( track: config.track_sources, ..Body::default() }; - ctx.blocks(body_el, &mut body); + // Body blocks are converted as they are parsed and freed right after, so + // the whole document tree is never held at once. + let main = zip + .read_xml_split(&main_path, &["document", "body"], &mut |el| { + if !budget.expired() { + ctx.block(&el, &mut body); + } + })? + .ok_or_else(|| Error::DocxExtract("missing main document part".into()))?; + main.child("document") + .ok_or_else(|| Error::DocxExtract("main part is not a w:document".into()))? + .child("body") + .ok_or_else(|| Error::DocxExtract("document has no body".into()))?; budget.check_time()?; body.flush_list(); diff --git a/crates/markkit/src/html/parser.rs b/crates/markkit/src/html/parser.rs index 2ffd2e8..58c42e2 100644 --- a/crates/markkit/src/html/parser.rs +++ b/crates/markkit/src/html/parser.rs @@ -92,6 +92,28 @@ enum RuleHit { } /// HTML-spec clamps for table spans. +/// Most nodes a DOM arena is pre-sized for; larger documents grow it as before. +const MAX_NODE_ESTIMATE: usize = 1 << 18; + +/// Rough DOM node count: start tags (elements) plus text runs (a `>` not +/// directly followed by `<`). Only a capacity hint, so it is clamped: at most one +/// node per 4 input bytes (`b` is 4 bytes per node; pages run 15+) and +/// `MAX_NODE_ESTIMATE` (32 MiB of arena), so hostile ` usize { + let (mut tags, mut texts) = (0usize, 0usize); + for w in html.as_bytes().windows(2) { + match w { + [b'<', c] if c.is_ascii_alphabetic() => tags += 1, + [b'>', c] if *c != b'<' => texts += 1, + _ => {} + } + } + (tags + texts + 8) + .min(html.len() / 4) + .min(MAX_NODE_ESTIMATE) +} + const MAX_COLSPAN: u32 = 1000; const MAX_ROWSPAN: u32 = 65534; @@ -184,11 +206,19 @@ impl<'a> HtmlParser<'a> { /// html5ever parse of `html` with nesting capped first (see `nesting`). fn parse_html5ever(&self, html: &str) -> Html { - Html::parse_document(&nesting::cap_nesting( + use html5ever::tendril::TendrilSink; + let html = nesting::cap_nesting( html, self.max_depth + NESTING_SLACK, !self.config.include_hidden_content, - )) + ); + // Same as `Html::parse_document`, but the node arena starts near its final + // size: growing it by doubling briefly holds old and new arrays, which + // set the HTML peak heap. + let mut doc = Html::new_document(); + doc.tree = ego_tree::Tree::with_capacity(scraper::Node::Document, node_estimate(&html)); + html5ever::driver::parse_document(scraper::HtmlTreeSink::new(doc), Default::default()) + .one(html.as_ref()) } /// Parse an HTML string into a [`Document`] plus page title and metadata. @@ -351,8 +381,8 @@ impl<'a> HtmlParser<'a> { for child in parent.children() { match child.value() { HtmlNode::Text(text) => { - let t = text.to_string(); - let collapsed = collapse_whitespace(&t); + let t: &str = text; + let collapsed = collapse_whitespace(t); if !collapsed.is_empty() { out.push(Node::Text(collapsed)); } else if t.contains(char::is_whitespace) && !out.is_empty() { @@ -959,10 +989,12 @@ impl<'a> HtmlParser<'a> { exclude: &[Selector], depth: usize, ) -> Vec { - let mut nodes = Vec::new(); if self.budget().expired() { - return nodes; + return Vec::new(); } + // Each child yields at most one node: size once (most cells and + // paragraphs hold one or two) instead of the default growth to four. + let mut nodes = Vec::with_capacity(el.children().count().max(1)); if depth >= self.max_depth { let text = flat_text(el, !self.config.include_hidden_content); let trimmed = text.trim(); @@ -975,8 +1007,8 @@ impl<'a> HtmlParser<'a> { for child in el.children() { match child.value() { HtmlNode::Text(text) => { - let t = text.to_string(); - let collapsed = collapse_whitespace(&t); + let t: &str = text; + let collapsed = collapse_whitespace(t); if !collapsed.is_empty() { nodes.push(Node::Text(collapsed)); } else if t.contains(char::is_whitespace) && !nodes.is_empty() { @@ -1334,6 +1366,10 @@ fn extract_language_from_class(class: &str) -> Option { /// Collapse runs of whitespace into single spaces. /// Preserves a leading/trailing space if the original had one (important for inline context). fn collapse_whitespace(s: &str) -> String { + // Whitespace-only text nodes are common between tags: no allocation. + if s.trim().is_empty() { + return String::new(); + } let has_leading_space = s.starts_with(char::is_whitespace); let has_trailing_space = s.ends_with(char::is_whitespace); @@ -1435,6 +1471,22 @@ fn is_transparent_container(tag: &str) -> bool { mod tests { use super::*; + #[test] + fn node_estimate_tracks_tags_and_text_but_is_clamped() { + let page = "

a b c

".repeat(100); + let real = Html::parse_document(&page).tree.nodes().count(); + let est = node_estimate(&page); + assert!( + est >= real * 9 / 10 && est <= real * 3 / 2, + "{est} vs {real}" + ); + // Hostile: thousands of `x".repeat(1_000_000)) <= MAX_NODE_ESTIMATE); + } + #[test] fn test_collapse_whitespace() { assert_eq!(collapse_whitespace(" hello world "), " hello world "); diff --git a/crates/markkit/src/ooxml.rs b/crates/markkit/src/ooxml.rs index 1139713..c5a25eb 100644 --- a/crates/markkit/src/ooxml.rs +++ b/crates/markkit/src/ooxml.rs @@ -6,7 +6,9 @@ //! only internal targets are followed for parts; external targets are kept as //! hyperlink URLs only. +use std::borrow::Cow; use std::collections::HashMap; +use std::rc::Rc; use crate::ir::{ListItem, Node}; use crate::xml::Element; @@ -85,27 +87,29 @@ pub(crate) struct Fmt { pub strike: bool, } -pub(crate) enum Seg { +/// Inline segment. Text borrows from the XML tree until it becomes an IR node; +/// the link is shared by every segment of a hyperlink. +pub(crate) enum Seg<'e> { Text { - text: String, + text: Cow<'e, str>, fmt: Fmt, - link: Option, + link: Option>, }, Break { - link: Option, + link: Option>, }, } -impl Seg { - pub(crate) fn link(&self) -> Option<&str> { +impl Seg<'_> { + pub(crate) fn link(&self) -> Option<&Rc> { match self { - Seg::Text { link, .. } | Seg::Break { link } => link.as_deref(), + Seg::Text { link, .. } | Seg::Break { link } => link.as_ref(), } } } /// Merge segments and turn them into IR inline nodes. -pub(crate) fn build_inline(segs: Vec) -> Vec { +pub(crate) fn build_inline(segs: Vec>) -> Vec { // 1. Merge adjacent text with identical formatting and link. let mut merged: Vec = Vec::with_capacity(segs.len()); for seg in segs { @@ -120,81 +124,77 @@ pub(crate) fn build_inline(segs: Vec) -> Vec { && pf == fmt && pl == link { - prev.push_str(text); + prev.to_mut().push_str(text); continue; } merged.push(seg); } // 2. Group by link. - let mut out: Vec = Vec::new(); - let mut i = 0; - while i < merged.len() { - let link = merged[i].link().map(str::to_string); - let mut j = i; - while j < merged.len() && merged[j].link() == link.as_deref() { - j += 1; + let mut out: Vec = Vec::with_capacity(merged.len()); + let mut segs = merged.into_iter().peekable(); + while let Some(first) = segs.next() { + let Some(url) = first.link().cloned() else { + format_seg(first, &mut out); + continue; + }; + let mut nodes = Vec::new(); + format_seg(first, &mut nodes); + while let Some(next) = segs.next_if(|s| s.link() == Some(&url)) { + format_seg(next, &mut nodes); } - let nodes = format_nodes(&merged[i..j]); - match link { - Some(url) if !is_blank(&nodes) => { - let (lead, core, trail) = split_ws_nodes(nodes); - out.extend(lead); - out.push(Node::Link { - url, - title: None, - children: core, - }); - out.extend(trail); - } - _ => out.extend(nodes), + if is_blank(&nodes) { + out.extend(nodes); + } else { + let (lead, core, trail) = split_ws_nodes(nodes); + out.extend(lead); + out.push(Node::Link { + url: url.to_string(), + title: None, + children: core, + }); + out.extend(trail); } - i = j; } tidy_inline(out) } -pub(crate) fn format_nodes(segs: &[Seg]) -> Vec { - let mut out = Vec::new(); - for seg in segs { - match seg { - Seg::Break { .. } => out.push(Node::LineBreak), - Seg::Text { text, fmt, .. } => { - let core = text.trim(); - if core.is_empty() || *fmt == Fmt::default() { - out.push(Node::Text(text.clone())); - continue; - } - let start = text.len() - text.trim_start().len(); - let lead = &text[..start]; - let trail = &text[start + core.len()..]; - if !lead.is_empty() { - out.push(Node::Text(lead.to_string())); - } - let mut node = Node::Text(core.to_string()); - if fmt.strike { - node = Node::Strikethrough { - children: vec![node], - }; - } - if fmt.italic { - node = Node::Italic { - children: vec![node], - }; - } - if fmt.bold { - node = Node::Bold { - children: vec![node], - }; - } - out.push(node); - if !trail.is_empty() { - out.push(Node::Text(trail.to_string())); - } - } - } +fn format_seg(seg: Seg<'_>, out: &mut Vec) { + let (text, fmt) = match seg { + Seg::Break { .. } => return out.push(Node::LineBreak), + Seg::Text { text, fmt, .. } => (text, fmt), + }; + let core = text.trim(); + if core.is_empty() || fmt == Fmt::default() { + out.push(Node::Text(text.into_owned())); + return; + } + let start = text.len() - text.trim_start().len(); + let lead = &text[..start]; + let trail = &text[start + core.len()..]; + if !lead.is_empty() { + out.push(Node::Text(lead.to_string())); + } + let mut node = Node::Text(core.to_string()); + if fmt.strike { + node = Node::Strikethrough { + children: vec![node], + }; + } + if fmt.italic { + node = Node::Italic { + children: vec![node], + }; + } + if fmt.bold { + node = Node::Bold { + children: vec![node], + }; + } + out.push(node); + if !trail.is_empty() { + out.push(Node::Text(trail.to_string())); } - out } /// Split leading/trailing whitespace text out of a node list (for link text). @@ -220,22 +220,33 @@ pub(crate) fn split_ws_nodes(mut nodes: Vec) -> (Vec, Vec, Vec } /// Merge adjacent text nodes and strip leading/trailing line breaks. -pub(crate) fn tidy_inline(nodes: Vec) -> Vec { - let mut out: Vec = Vec::with_capacity(nodes.len()); - for n in nodes { - match (out.last_mut(), n) { - (Some(Node::Text(prev)), Node::Text(t)) => prev.push_str(&t), - (_, Node::Text(t)) if t.is_empty() => {} - (_, n) => out.push(n), +pub(crate) fn tidy_inline(mut nodes: Vec) -> Vec { + // In place: `w` is the write cursor, read slots are taken by swap. + let mut w = 0; + for r in 0..nodes.len() { + match std::mem::replace(&mut nodes[r], Node::LineBreak) { + Node::Text(t) if t.is_empty() => {} + Node::Text(t) if w > 0 && matches!(nodes[w - 1], Node::Text(_)) => { + if let Node::Text(prev) = &mut nodes[w - 1] { + prev.push_str(&t); + } + } + n => { + nodes[w] = n; + w += 1; + } } } - while matches!(out.first(), Some(Node::LineBreak)) { - out.remove(0); - } - while matches!(out.last(), Some(Node::LineBreak)) { - out.pop(); + nodes.truncate(w); + while matches!(nodes.last(), Some(Node::LineBreak)) { + nodes.pop(); } - out + let lead = nodes + .iter() + .take_while(|n| matches!(n, Node::LineBreak)) + .count(); + nodes.drain(..lead); + nodes } pub(crate) fn is_blank(nodes: &[Node]) -> bool { diff --git a/crates/markkit/src/pptx.rs b/crates/markkit/src/pptx.rs index 214fb14..8a4d7cb 100644 --- a/crates/markkit/src/pptx.rs +++ b/crates/markkit/src/pptx.rs @@ -20,6 +20,7 @@ //! layouts (template text), and macros (`vbaProject.bin` is never read). use std::collections::HashMap; +use std::rc::Rc; use crate::archive::{ArchiveError, ZipGuard}; use crate::config::MarkkitBuilder; @@ -171,7 +172,7 @@ impl SlideCtx<'_> { segs.push(Seg::Text { text: t.text(), fmt: run_fmt(rpr), - link: self.link(rpr), + link: self.link(rpr).map(Rc::from), }); } } diff --git a/crates/markkit/src/render/escape.rs b/crates/markkit/src/render/escape.rs index 3621b59..09568a0 100644 --- a/crates/markkit/src/render/escape.rs +++ b/crates/markkit/src/render/escape.rs @@ -27,6 +27,47 @@ pub(crate) enum Position { /// Escape document text for inline Markdown. Returns whether the output still sits at /// a line start (the run was empty or whitespace only). pub(crate) fn push_text(buf: &mut String, s: &str, pos: Position) -> bool { + // Fast path: most runs hold nothing escapable, so copy them without the + // per-char buffer. Must agree with `push_text_slow` (tested). + let inert = !s.bytes().any(|b| { + matches!( + b, + b'`' | b'*' + | b'[' + | b']' + | b'<' + | b'~' + | b'|' + | b'\\' + | b'!' + | b'_' + | b'&' + | b'\n' + | b'\r' + ) + }); + if inert { + let lead = s.bytes().find(|b| !matches!(b, b' ' | b'\t')); + if pos == Position::Inline { + buf.push_str(s); + return false; + } + match lead { + None => { + buf.push_str(s); + return true; + } + Some(b) if !matches!(b, b'#' | b'>' | b'-' | b'+' | b'=' | b'0'..=b'9') => { + buf.push_str(s); + return false; + } + Some(_) => {} + } + } + push_text_slow(buf, s, pos) +} + +fn push_text_slow(buf: &mut String, s: &str, pos: Position) -> bool { let chars: Vec = s .chars() .map(|c| if c == '\n' || c == '\r' { ' ' } else { c }) @@ -302,6 +343,45 @@ mod tests { b } + #[test] + fn fast_path_matches_slow_path() { + let samples = [ + "", + " ", + "\t ", + "plain words", + " lead", + "# h", + " # h", + "- x", + "+x", + "=", + ">q", + "1. one", + "12) x", + "1x", + "a1. b", + "é # ü", + "x\ny", + "\r", + "a_b", + "C:\\dir", + "!", + "&", + "trail ", + " 9", + "\u{a0}-", + ]; + for s in samples { + for pos in [Position::LineStart, Position::Inline] { + let (mut fast, mut slow) = (String::new(), String::new()); + let a = push_text(&mut fast, s, pos); + let b = push_text_slow(&mut slow, s, pos); + assert_eq!((fast, a), (slow, b), "{s:?}"); + } + } + } + #[test] fn escapes_inline_syntax() { assert_eq!( diff --git a/crates/markkit/src/render/markdown.rs b/crates/markkit/src/render/markdown.rs index 5e9074a..1847558 100644 --- a/crates/markkit/src/render/markdown.rs +++ b/crates/markkit/src/render/markdown.rs @@ -106,6 +106,8 @@ impl MarkdownRenderer { /// Nested renderer sharing this renderer's limits. fn sub(&self) -> Self { Self { + // Nested blocks are usually short: no up-front 4 KiB per list item. + buf: String::new(), max_output: self.max_output, deadline: self.deadline, ..Self::new() @@ -641,7 +643,7 @@ fn layout_table<'a>(rows: &[&'a [TableCell]], spans: bool) -> Option = Vec::new(); + let mut out: GridRow<'a> = Vec::with_capacity(row.len()); let mut col = 0usize; for cell in row.iter() { while busy.get(col).copied().unwrap_or(false) { @@ -758,34 +760,30 @@ fn normalize_output(s: &str) -> String { for line in s.lines() { // Preserve markdown hard break (line ending in two+ spaces with content) let core = line.trim_end(); - let is_hard_break = line.ends_with(" ") && !core.is_empty(); - let trimmed = if is_hard_break { - core.to_string() + " " - } else { - core.to_string() - }; - - if trimmed.is_empty() || trimmed == " " { + if core.is_empty() { blank_count += 1; if blank_count <= 1 { result.push('\n'); } } else { blank_count = 0; - result.push_str(&trimmed); + result.push_str(core); + if line.ends_with(" ") { + result.push_str(" "); + } result.push('\n'); } } // Trim leading/trailing blank lines - let trimmed = result.trim_matches('\n'); - if trimmed.is_empty() { - String::new() - } else { - let mut out = trimmed.to_string(); - out.push('\n'); - out + // Trim in place: drop leading newlines, keep exactly one trailing. + let lead = result.len() - result.trim_start_matches('\n').len(); + result.drain(..lead); + result.truncate(result.trim_end_matches('\n').len()); + if !result.is_empty() { + result.push('\n'); } + result } #[cfg(test)] diff --git a/crates/markkit/src/spreadsheet.rs b/crates/markkit/src/spreadsheet.rs index 2d0a436..21fc03b 100644 --- a/crates/markkit/src/spreadsheet.rs +++ b/crates/markkit/src/spreadsheet.rs @@ -330,13 +330,29 @@ fn fmt_xlsb(v: &xlsb::Value, is_1904: bool) -> String { } } +/// Same output as [`fmt_data`], without first cloning into an owned `Data`. fn fmt_ref(d: &DataRef<'_>) -> String { match d { + DataRef::Int(i) => i.to_string(), + DataRef::Float(f) => fmt_float(*f), + DataRef::String(s) | DataRef::DurationIso(s) => s.clone(), DataRef::SharedString(s) => (*s).to_string(), - other => fmt_data(&Data::from(other.clone())), + DataRef::DateTimeIso(s) => s.replacen('T', " ", 1), + DataRef::Bool(b) => b.to_string(), + DataRef::DateTime(dt) => fmt_datetime(dt), + DataRef::Error(e) => e.to_string(), + DataRef::Empty => String::new(), } } +/// Trim in place, keeping the allocation. +fn trim_owned(mut s: String) -> String { + s.truncate(s.trim_end().len()); + let lead = s.len() - s.trim_start().len(); + s.drain(..lead); + s +} + // ─── Grid collection (bounded) ────────────────────────────────────── struct SheetTable { @@ -384,7 +400,8 @@ impl Grid { } fn add(&mut self, r: u32, c: u32, value: String) { - if value.trim().is_empty() { + let value = trim_owned(value); + if value.is_empty() { return; } let keep = self.max_rows.saturating_add(1); @@ -440,7 +457,7 @@ impl Grid { for (c, v) in cells { let idx = (c - min_col) as usize; if idx < width { - row[idx] = v.trim().to_string(); + row[idx] = v; } } row @@ -1173,6 +1190,35 @@ fn prescan_ods(content: &[u8], max_cells: u64) -> Result<(), Error> { mod tests { use super::*; + #[test] + fn fmt_ref_matches_fmt_data() { + let dt = + calamine::ExcelDateTime::new(45_000.5, calamine::ExcelDateTimeType::DateTime, false); + let refs = [ + DataRef::Int(-3), + DataRef::Float(2.0), + DataRef::Float(0.25), + DataRef::String("s".into()), + DataRef::SharedString("shared"), + DataRef::Bool(true), + DataRef::DateTime(dt), + DataRef::DateTimeIso("2024-01-02T03:04:05".into()), + DataRef::DurationIso("PT1H".into()), + DataRef::Error(calamine::CellErrorType::Div0), + DataRef::Empty, + ]; + for r in refs { + assert_eq!(fmt_ref(&r), fmt_data(&Data::from(r.clone())), "{r:?}"); + } + } + + #[test] + fn trim_owned_trims_unicode_whitespace() { + assert_eq!(trim_owned("\u{a0} a b \t\n".into()), "a b"); + assert_eq!(trim_owned(" \u{3000} ".into()), ""); + assert_eq!(trim_owned("x".into()), "x"); + } + #[test] fn float_formatting() { assert_eq!(fmt_float(3.0), "3"); diff --git a/crates/markkit/src/xml.rs b/crates/markkit/src/xml.rs index 7e8d433..489389c 100644 --- a/crates/markkit/src/xml.rs +++ b/crates/markkit/src/xml.rs @@ -8,6 +8,16 @@ //! Tree depth (`limits.max_nesting_depth`), node count (`limits.max_nodes`, //! shared across parts) and the deadline are enforced while building; the tree //! is built iteratively (no recursion on input). +//! +//! Decision: memory. Names are interned per parse, attribute and child lists are +//! exact-size boxed slices, and single-text elements hand out borrowed text. +//! [`parse_split`] streams the children of one element path (a DOCX body's +//! paragraphs and tables) to a callback and frees each one, so peak memory follows +//! the largest block, not the whole part. + +use std::borrow::Cow; +use std::collections::{HashMap, HashSet}; +use std::rc::Rc; use quick_xml::events::{BytesStart, Event}; use quick_xml::{Reader, XmlVersion}; @@ -21,18 +31,54 @@ pub(crate) enum XmlError { Invalid(String), } +/// Interned string (local names and attribute values): one allocation per +/// distinct value per parse, shared by every element that uses it. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub(crate) struct Name(Rc); + +impl Name { + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl std::ops::Deref for Name { + type Target = str; + fn deref(&self) -> &str { + &self.0 + } +} + +impl PartialEq for Name { + fn eq(&self, other: &str) -> bool { + &*self.0 == other + } +} + +impl PartialEq<&str> for Name { + fn eq(&self, other: &&str) -> bool { + &*self.0 == *other + } +} + +/// Element node. Attribute and child lists are exact-size boxed slices (built +/// from a shared scratch stack), so leaves allocate nothing for them and no +/// list carries growth slack. #[derive(Debug, Default)] pub(crate) struct Element { - pub name: String, + pub name: Name, /// `(qualified name, value)`; looked up by local name. - pub attrs: Vec<(String, String)>, - pub children: Vec, + pub attrs: Attrs, + pub children: Box<[XNode]>, } +/// Attribute list: (local name, value). +pub(crate) type Attrs = Box<[(Name, Name)]>; + #[derive(Debug)] pub(crate) enum XNode { Element(Element), - Text(String), + Text(Box), } impl Element { @@ -41,7 +87,7 @@ impl Element { self.attrs .iter() .find(|(k, _)| local_of(k) == local_name) - .map(|(_, v)| v.as_str()) + .map(|(_, v)| &**v) } /// Attribute value by local name, prefixed attributes only. Tells @@ -50,7 +96,7 @@ impl Element { self.attrs .iter() .find(|(k, _)| k.contains(':') && local_of(k) == local_name) - .map(|(_, v)| v.as_str()) + .map(|(_, v)| &**v) } /// Child elements. @@ -72,7 +118,13 @@ impl Element { } /// Concatenated text of this element and its descendants. - pub fn text(&self) -> String { + pub fn text(&self) -> Cow<'_, str> { + // Common case: one text child, borrowed as is. + match &*self.children { + [] => return Cow::Borrowed(""), + [XNode::Text(t)] => return Cow::Borrowed(t), + _ => {} + } let mut out = String::new(); // Iterative depth-first walk in document order. let mut pending: Vec> = vec![self.children.iter()]; @@ -85,7 +137,7 @@ impl Element { } } } - out + Cow::Owned(out) } /// Depth-first search for the first descendant with the given local name. @@ -107,35 +159,98 @@ fn local_of(name: &str) -> &str { name.rsplit_once(':').map_or(name, |(_, l)| l) } -fn local(name: &str) -> String { - local_of(name).to_string() +/// Builder state: interned names plus scratch stacks for attributes and +/// children, so every finished list is collected once at its exact size. +#[derive(Default)] +struct Builder<'s> { + names: HashMap, Name>, + /// Attribute values repeat heavily in OOXML (style ids, `val="0"`), so they + /// are interned too. + values: HashSet>, + /// Open elements: name, attributes, start index of their children in `nodes`. + open: Vec<(Name, Attrs, usize)>, + nodes: Vec, + attrs: Vec<(Name, Name)>, + /// Elements whose open ancestors match `split` go to `sink` instead of the tree. + split: &'s [&'s str], + sink: Option<&'s mut dyn FnMut(Element)>, } -fn element_from(start: &BytesStart<'_>) -> Element { - let mut attrs = Vec::new(); - for attr in start.attributes().with_checks(false).flatten() { - let key = attr.key.as_ref().to_string(); - let value = match attr.normalized_value(XmlVersion::Implicit1_0) { - Ok(v) => v.into_owned(), - Err(_) => attr.value.into_owned(), - }; - attrs.push((key, value)); +impl Builder<'_> { + fn name(&mut self, qname: &str) -> Name { + if let Some(n) = self.names.get(qname) { + return n.clone(); + } + let n = Name(Rc::from(local_of(qname))); + self.names.insert(qname.into(), n.clone()); + n } - Element { - name: local(start.name().as_ref()), - attrs, - children: Vec::new(), + + fn value(&mut self, v: &str) -> Name { + if let Some(k) = self.values.get(v) { + return Name(k.clone()); + } + let k: Rc = Rc::from(v); + self.values.insert(k.clone()); + Name(k) } -} -fn push_text(parent: &mut Element, text: &str) { - if text.is_empty() { - return; + fn start(&mut self, start: &BytesStart<'_>) -> (Name, Attrs) { + for attr in start.attributes().with_checks(false).flatten() { + // Attributes keep their qualified name (`r:id` vs `id`). + let key = self.value(attr.key.as_ref()); + let value = match attr.normalized_value(XmlVersion::Implicit1_0) { + Ok(v) => self.value(&v), + Err(_) => self.value(&attr.value), + }; + self.attrs.push((key, value)); + } + let attrs = self.attrs.drain(..).collect(); + (self.name(start.name().as_ref()), attrs) } - if let Some(XNode::Text(prev)) = parent.children.last_mut() { - prev.push_str(text); - } else { - parent.children.push(XNode::Text(text.to_string())); + + /// Start of the innermost open element's children in `nodes`. + fn base(&self) -> usize { + self.open.last().map_or(0, |o| o.2) + } + + fn text(&mut self, text: &str) { + if text.is_empty() { + return; + } + let base = self.base(); + if self.nodes.len() > base + && let Some(XNode::Text(prev)) = self.nodes.last_mut() + { + let mut joined = String::with_capacity(prev.len() + text.len()); + joined.push_str(prev); + joined.push_str(text); + *prev = joined.into_boxed_str(); + } else { + self.nodes.push(XNode::Text(text.into())); + } + } + + fn close(&mut self) { + if let Some((name, attrs, base)) = self.open.pop() { + let children = self.nodes.drain(base..).collect(); + self.finish(Element { + name, + attrs, + children, + }); + } + } + + fn finish(&mut self, el: Element) { + if let Some(sink) = self.sink.as_mut() + && self.open.len() == self.split.len() + && self.open.iter().zip(self.split).all(|(o, s)| o.0 == *s) + { + sink(el); + } else { + self.nodes.push(XNode::Element(el)); + } } } @@ -149,9 +264,35 @@ fn limit(e: Error) -> XmlError { /// Parse an XML document into a tree. Returns a synthetic root whose children /// are the document's top-level nodes. pub(crate) fn parse(bytes: &[u8], max_depth: usize, budget: &Budget) -> Result { + build(bytes, max_depth, budget, Builder::default()) +} + +/// Like [`parse`], but every element whose ancestors' local names are exactly +/// `split` (outermost first) is passed to `sink` in document order and left out +/// of the returned tree. +pub(crate) fn parse_split( + bytes: &[u8], + max_depth: usize, + budget: &Budget, + split: &[&str], + sink: &mut dyn FnMut(Element), +) -> Result { + let b = Builder { + split, + sink: Some(sink), + ..Builder::default() + }; + build(bytes, max_depth, budget, b) +} + +fn build( + bytes: &[u8], + max_depth: usize, + budget: &Budget, + mut b: Builder<'_>, +) -> Result { let mut reader = Reader::from_reader(bytes); reader.config_mut().check_end_names = false; - let mut stack: Vec = vec![Element::default()]; let mut buf = Vec::new(); loop { let event = reader @@ -168,37 +309,32 @@ pub(crate) fn parse(bytes: &[u8], max_depth: usize, budget: &Budget) -> Result { - if stack.len() > max_depth { + if b.open.len() >= max_depth { return Err(XmlError::Limit(format!( "XML nesting deeper than max_nesting_depth {max_depth}" ))); } - stack.push(element_from(&s)); + let (name, attrs) = b.start(&s); + let base = b.nodes.len(); + b.open.push((name, attrs, base)); } Event::Empty(s) => { - let el = element_from(&s); - if let Some(top) = stack.last_mut() { - top.children.push(XNode::Element(el)); - } + let (name, attrs) = b.start(&s); + b.finish(Element { + name, + attrs, + children: Box::default(), + }); } Event::End(_) => { - if stack.len() > 1 { - let el = stack.pop().unwrap_or_default(); - if let Some(top) = stack.last_mut() { - top.children.push(XNode::Element(el)); - } - } + b.close(); } Event::Text(t) => { let text = t.xml_content(XmlVersion::Implicit1_0); - if let Some(top) = stack.last_mut() { - push_text(top, &text); - } + b.text(&text); } Event::CData(t) => { - if let Some(top) = stack.last_mut() { - push_text(top, &t); - } + b.text(&t); } Event::GeneralRef(r) => { let resolved: Option = match r.resolve_char_ref() { @@ -209,8 +345,8 @@ pub(crate) fn parse(bytes: &[u8], max_depth: usize, budget: &Budget) -> Result None, }; // Unknown (DTD-declared / external) entities are dropped, never expanded. - if let (Some(text), Some(top)) = (resolved, stack.last_mut()) { - push_text(top, &text); + if let Some(text) = resolved { + b.text(&text); } } Event::Eof => break, @@ -219,13 +355,13 @@ pub(crate) fn parse(bytes: &[u8], max_depth: usize, budget: &Budget) -> Result 1 { - let el = stack.pop().unwrap_or_default(); - if let Some(top) = stack.last_mut() { - top.children.push(XNode::Element(el)); - } + while !b.open.is_empty() { + b.close(); } - Ok(stack.pop().unwrap_or_default()) + Ok(Element { + children: b.nodes.into_boxed_slice(), + ..Element::default() + }) } #[cfg(test)] @@ -248,6 +384,36 @@ mod tests { assert_eq!(root.find("c").map(|e| e.name.as_str()), Some("c")); } + #[test] + fn split_streams_matching_children_in_order() { + let xml = br#"aintout"#; + let mut seen = Vec::new(); + let rest = parse_split( + xml, + 256, + &Budget::new(&Limits::default()), + &["document", "body"], + &mut |el| seen.push(format!("{}:{}", el.name.as_str(), el.text())), + ) + .unwrap(); + assert_eq!(seen, ["p:a", "tbl:in", "p:", "sectPr:"]); + let doc = rest.child("document").unwrap(); + // Streamed elements are gone; text and non-matching paths stay. + assert_eq!(doc.child("body").unwrap().text(), "t"); + assert_eq!(doc.child("p").unwrap().text(), "out"); + } + + #[test] + fn names_and_values_are_shared_not_copied() { + let root = parse(br#"tu"#).unwrap(); + let xs: Vec<_> = root.child("r").unwrap().elements().collect(); + assert!(Rc::ptr_eq(&xs[0].name.0, &xs[1].name.0)); + assert!(Rc::ptr_eq(&xs[0].attrs[0].1.0, &xs[1].attrs[0].1.0)); + assert_eq!(xs[2].attr("v"), Some("2")); + assert!(matches!(xs[0].text(), Cow::Borrowed(""))); + assert!(matches!(xs[2].text(), Cow::Owned(t) if t == "tu")); + } + #[test] fn prefixed_attribute_lookup() { let root = parse(br#""#).unwrap(); diff --git a/crates/markkit/tests/document_security.rs b/crates/markkit/tests/document_security.rs index ba68be3..5f5c279 100644 --- a/crates/markkit/tests/document_security.rs +++ b/crates/markkit/tests/document_security.rs @@ -589,6 +589,35 @@ fn docx_missing_document_part_is_error() { assert!(matches!(err, Error::DocxExtract(_)), "{err:?}"); } +#[cfg(feature = "docx")] +#[test] +fn docx_without_document_or_body_is_error() { + // The body is streamed while parsing; structure errors must still surface. + let w = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main""#; + for doc in [ + format!("x"), + format!("x"), + format!(""), + ] { + let bytes = zip_bytes(&[("word/document.xml", doc.as_bytes())]); + let err = run(&Markkit::builder().build(), &bytes, DOCX_CT).unwrap_err(); + assert!(matches!(err, Error::DocxExtract(_)), "{doc}: {err:?}"); + } +} + +#[cfg(feature = "docx")] +#[test] +fn docx_streamed_body_keeps_node_and_depth_limits() { + let body = para("x").repeat(2_000); + let err = run( + &Markkit::builder().max_nodes(500).build(), + &docx(&body), + DOCX_CT, + ) + .unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); +} + #[cfg(feature = "spreadsheet")] #[test] fn spreadsheet_garbage_is_error() { diff --git a/knowledge/formats/format-support.md b/knowledge/formats/format-support.md index 9ef70f4..6327e5c 100644 --- a/knowledge/formats/format-support.md +++ b/knowledge/formats/format-support.md @@ -145,7 +145,10 @@ is public for them. | Runs with direct `w:vanish` (hidden text) | Skipped with `hidden_content_skipped` unless `include_hidden_content(true)` | The main part is located through `_rels/.rels` (fallback `word/document.xml`); styles and -numbering through the main part's relationships. +numbering through the main part's relationships. Body blocks (`w:p`, `w:tbl`, ...) are +converted while the main part is parsed and each block's XML subtree is freed right after +(`xml::parse_split`), so peak memory follows the largest block, not the document. XML +trees intern element names and attribute values per part. ## PPTX Mapping diff --git a/knowledge/formats/html-extraction.md b/knowledge/formats/html-extraction.md index 8033ace..9ad11ab 100644 --- a/knowledge/formats/html-extraction.md +++ b/knowledge/formats/html-extraction.md @@ -157,6 +157,13 @@ tables stay in the nested table. Prevents stack overflow and super-linear time on adversarial nesting. Tested with 64 000 nested `
`s, 20 000 unclosed nested tables, and script/style past the limit. +## Memory + +The scraper DOM (about 128 bytes per node) dominates peak heap. `parse_html5ever` pre-sizes +the node arena from a byte scan (start tags plus text runs, clamped to `len / 4` and 2^18 +nodes) instead of letting it double, inline node lists are sized from the child count, and +whitespace-only text nodes allocate nothing. Bench pages peak at 16-20x input (was 25x). + ## Exclude Selector Defaults Default `exclude_selectors` in `MarkkitBuilder::default()`: diff --git a/knowledge/log.md b/knowledge/log.md index bd7c9f9..f3a6a46 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -12,6 +12,9 @@ * **Plain text and Markdown input**: text is always on and goes through the IR; Markdown (`markdown` feature) is re-parsed with pulldown-cmark rather than passed through, so raw HTML never reaches output. Neither is sniffed from bytes. Shares the call's `Limits` budget (timeout, nodes, nesting depth). New threat T7 and invariant S18. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md). * **Eval harness on mira**: `markkit-eval` is now a mira study (modeled on bashkit-eval). `markkit_extract` scores markkit's HTML output on ten realistic pages with typed deterministic checks and one scorer per metric; a `mode` axis compares defaults with `main_content` (73.0% vs 86.8% mean, noise 34% vs 82%); dated results committed in `crates/markkit-eval/results/`; gaps recorded (`main_content` drops the forum question, `4.6` over-escaped as `4\.6`, `tfoot`/`caption` dropped, edit links in headings). See [Evaluation Harness](operations/eval.md). * **CSV / TSV input**: always on, hand-written RFC 4180 reader into one IR table; delimiter detected for CSV; bounded by the sheet row/column caps and the call's `timeout` while reading; cells stay raw in the IR (renderer escapes). T7 and S18 now cover CSV. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md). +* **DOCX memory**: `docx_report` peaked at 18.5 MiB for 1.5 MB of XML with 265k allocations. The main part's body is now streamed block by block, XML names and attribute values are interned, child/attribute lists are exact-size, run text is borrowed until it becomes IR, and the escaper and output normalizer stop allocating per run/line. Result: 3.9 MiB peak, about 137k allocations. See [Format Support](formats/format-support.md). +* **Spreadsheet allocations**: `xlsx_large` (150k cells) made about 5 allocations per cell: XLSX values were cloned into `Data` and formatted again, and every cell was trimmed into a new string. Cells are now formatted straight from calamine's `DataRef` and trimmed in place; about 3.4 per cell remain (one is calamine's, two are the cell's text and its IR `TableCell` node list). +* **HTML memory**: peak heap was about 25x input, mostly scraper's node arena grown by doubling plus four-slot inline node lists for one-node cells. The arena is now pre-sized from a clamped byte scan, inline lists are sized from the child count, and text nodes are no longer copied before whitespace collapsing. See [HTML Extraction](formats/html-extraction.md). ## 2026-10-26 diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 2027041..02baaaf 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -247,7 +247,8 @@ process, or smuggling content past a human reviewer. #### T6a: Zip bombs - Limits apply to bytes actually produced by the decompressor (`Read::take`), never to - attacker-controlled header sizes. + attacker-controlled header sizes. A declared size is only a buffer capacity hint, + clamped to `max_archive_entry_size` and 16 MiB. - `max_archive_entries` (10 000), `max_archive_entry_size` (64 MiB), `max_archive_total_size` (256 MiB) → `Error::ResourceLimit`. - DOCX reads only the parts it needs, each through the bounded reader.