diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d68d5..81fa83e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/). `
`/`
` and table spans/alignment. Row-header `` cells in table bodies are no longer dropped. - `Document.ir_version` (`markkit::ir::IR_VERSION = 2`) is serialized with the IR JSON. +- PDF extraction (`pdf` feature, on in the CLI `all_formats` build and the Python/Node bindings): + the text layer becomes paragraphs, font-size headings and bullet lists, with `/Title` as + `Extraction::title` and `stats.pages`. Pure Rust (hayro-syntax), no OCR. Fonts map through + `ToUnicode` CMaps, simple-font encodings and `/Differences`. New warnings `truncated_pdf`, + `pdf_unmapped_font`, `pdf_no_text`. Encrypted PDFs return `UnsupportedFormat`. +- `Limits` fields `max_pdf_pages` (10 000), `max_pdf_stream_size` (64 MiB) and + `max_pdf_decoded_size` (256 MiB), also on the builder, as CLI flags (`--max-pdf-pages`, ...), + Python `Limits(...)` arguments and Node options (`maxPdfPages`, ...). PDF extraction also + honours `timeout` and `max_output_size`. - Relative URL resolution: links, images and metadata URLs resolve against the new `MarkkitBuilder::base_url` (CLI `--base-url`) and the page's ``; `extract_url` uses the final URL after redirects. Absolute and fragment-only URLs are left unchanged. @@ -73,6 +82,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/). thousands of rows); `max_nodes` (5 000 000) caps parsed HTML DOM / DOCX XML nodes; `max_nesting_depth` (256) is configurable, with a hard ceiling of 512 that keeps deep input off the host stack. Violations return `Error::ResourceLimit` naming the limit. +- PDF decompression bombs: every stream in the file is decoded under the PDF size limits before + the parser runs (object streams included), interpreted content is budgeted so reused form + XObjects cannot multiply work, and stream filters that cannot be verified are refused. + - `extract_url` (CLI, Python and Node `extract_url` too) now refuses non-public destinations by default: loopback, private, link-local / cloud metadata and other reserved addresses, checked on the URL, after DNS resolution and on every redirect. Only `http`/`https` are fetched. Redirects are capped diff --git a/Cargo.lock b/Cargo.lock index 1c3af4c..ed873b7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,6 +2,12 @@ # It is not intended for manual editing. version = 4 +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + [[package]] name = "aho-corasick" version = "1.1.5" @@ -810,6 +816,40 @@ version = "0.17.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" +[[package]] +name = "hayro-ccitt" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2866f0d976e76497480b853d5e258dab6e4b89ecfe991124e08f15dfa8fd03ad" + +[[package]] +name = "hayro-jbig2" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "000e025850f4592ddcaee85640db70704151f2bafd43b6f431aa5b6a2c6a35a9" +dependencies = [ + "hayro-ccitt", +] + +[[package]] +name = "hayro-jpeg2000" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "122c5f834c37a2260b9c851bb8be685cf539b2a8b94b704a931ef7d2525d5835" + +[[package]] +name = "hayro-syntax" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6cdc818d8d265dcb42081d488f7599df4f73fb79e9d347ecedcd3c23809d2026" +dependencies = [ + "hayro-jbig2", + "hayro-jpeg2000", + "rustc-hash", + "smallvec", + "zune-jpeg", +] + [[package]] name = "heck" version = "0.5.0" @@ -1195,7 +1235,9 @@ dependencies = [ "criterion", "ego-tree", "futures-core", + "hayro-syntax", "html5ever", + "miniz_oxide", "proptest", "pulldown-cmark", "quick-xml", @@ -1296,6 +1338,15 @@ version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" +[[package]] +name = "miniz_oxide" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c" +dependencies = [ + "adler2", +] + [[package]] name = "mio" version = "1.2.4" @@ -3106,3 +3157,18 @@ dependencies = [ "log", "simd-adler32", ] + +[[package]] +name = "zune-core" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d56377fd46368984a170bc5aac5567e52ca5da874caa60bea39fcbca78fb658b" + +[[package]] +name = "zune-jpeg" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27bc9d5b815bc103f142aa054f561d9187d191692ec7c2d1e2b4737f8dbd7296" +dependencies = [ + "zune-core", +] diff --git a/Cargo.toml b/Cargo.toml index 17be211..df8fff0 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -52,6 +52,8 @@ calamine = "0.37" # Excel/spreadsheet (XLSX/XLS/ODS) zip = { version = "9", default-features = false, features = ["deflate"] } # OOXML/ODF containers; matches calamine's zip quick-xml = "0.42" # DOCX/ODS XML parsing cfb = "0.15" # XLS (OLE2) container pre-scan +hayro-syntax = { version = "0.8", default-features = false, features = ["std"] } # PDF object/content parsing (no image decoders) +miniz_oxide = "0.9" # Bounded FlateDecode for PDF streams pulldown-cmark = { version = "0.13", default-features = false } # Markdown input; output-safety test oracle # Logging diff --git a/README.md b/README.md index 2a1826f..94dffb6 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ package for browsers and edge runtimes. - **Configurable** – CSS include/exclude selectors, link/image preservation, resource limits - **Safe by default** – No code execution (scripts, macros, formulas), bounded decompression, depth and size limits, [threat model](knowledge/security/threat-model.md) with a test per mitigation -- **PDF** – Planned (currently an explicit `UnsupportedFormat` error, never an empty success) +- **PDF** – Text layer to paragraphs, headings and bullet lists (pure Rust, no OCR; encrypted PDFs rejected) | Format | Status | Feature | |---|---|---| @@ -35,7 +35,7 @@ package for browsers and edge runtimes. | XLSX, XLSM, XLS, ODS | Supported | `spreadsheet` | | Plain text | Supported | always on | | Markdown | Supported | `markdown` | -| PDF | Planned | `pdf` | +| PDF (text layer) | Supported | `pdf` | | DOC, PPTX, XLSB | Detected, rejected explicitly | – | ## Quick Start @@ -188,7 +188,7 @@ let markkit = Markkit::builder() | `http_client` | No | URL fetching via reqwest (`extract_url`) | | `docx` | No | DOCX conversion (zip + quick-xml) | | `spreadsheet` | No | XLSX/XLSM, XLS, ODS conversion (calamine) | -| `pdf` | No | Placeholder; PDF currently returns `UnsupportedFormat` | +| `pdf` | No | PDF text-layer extraction (hayro-syntax, pure Rust) | | `markdown` | No | Markdown normalization (pulldown-cmark) | | `all_formats` | No | `html`, `pdf`, `docx`, `spreadsheet`, `markdown` | | `tool` | No | `markkit::tool::MarkkitTool` LLM agent tool (Everruns toolkit contract) | @@ -221,6 +221,9 @@ generous; violations return `Error::ResourceLimit` naming the limit. | `max_archive_total_size` | 256 MiB | Decompressed bytes per ZIP | | `max_spreadsheet_cells` | 4 000 000 | XLS/ODS grid before parsing | | `max_sheet_rows` / `max_sheet_cols` | 10 000 / 256 | Rendered per sheet (truncated with a warning) | +| `max_pdf_pages` | 10 000 | PDF pages rendered (truncated with a warning) | +| `max_pdf_stream_size` | 64 MiB | Decoded bytes per PDF stream | +| `max_pdf_decoded_size` | 256 MiB | Decoded PDF stream bytes in total, and PDF content interpreted | | `max_redirects` | 5 | `extract_url` redirects | ```rust @@ -277,7 +280,8 @@ are deterministic and the better regression signal. Compare runs from the same m Markkit treats every input as untrusted. HTML scripts, styles, forms, iframes and SVG are stripped; office containers are decompressed under measured byte limits; spreadsheet grids are pre-scanned so sparse cells cannot force huge allocations; -XML entities are never expanded; macros and formulas are never executed; hidden +XML entities are never expanded; every PDF stream is decode-checked against size +limits before parsing; macros and formulas are never executed; hidden sheets are skipped by default. `extract_url` accepts only http/https, refuses loopback, private, link-local and cloud-metadata addresses (also after DNS resolution and on every redirect), caps redirects, and bounds the body by diff --git a/crates/markkit-cli/src/main.rs b/crates/markkit-cli/src/main.rs index 2757cba..753cf5d 100644 --- a/crates/markkit-cli/src/main.rs +++ b/crates/markkit-cli/src/main.rs @@ -121,6 +121,15 @@ struct LimitArgs { /// Maximum redirects followed when fetching a URL [default: 5] #[arg(long, value_name = "N")] max_redirects: Option, + /// Maximum PDF pages rendered [default: 10000] + #[arg(long, value_name = "N")] + max_pdf_pages: Option, + /// Maximum decoded size of one PDF stream, bytes [default: 67108864] + #[arg(long, value_name = "BYTES")] + max_pdf_stream_size: Option, + /// Maximum total decoded PDF content, bytes [default: 268435456] + #[arg(long, value_name = "BYTES")] + max_pdf_decoded_size: Option, } impl LimitArgs { @@ -140,7 +149,10 @@ impl LimitArgs { max_sheet_rows, max_sheet_cols, max_spreadsheet_cells, - max_redirects + max_redirects, + max_pdf_pages, + max_pdf_stream_size, + max_pdf_decoded_size ); l } @@ -282,6 +294,9 @@ mod tests { "--max-sheet-cols=9", "--max-spreadsheet-cells=10", "--max-redirects=11", + "--max-pdf-pages=12", + "--max-pdf-stream-size=13", + "--max-pdf-decoded-size=14", ]); let want = Limits::default() .max_input_size(1) @@ -295,7 +310,10 @@ mod tests { .max_sheet_rows(8) .max_sheet_cols(9) .max_spreadsheet_cells(10) - .max_redirects(11); + .max_redirects(11) + .max_pdf_pages(12) + .max_pdf_stream_size(13) + .max_pdf_decoded_size(14); assert_eq!(l, want); } diff --git a/crates/markkit-js/README.md b/crates/markkit-js/README.md index 0f194a3..3789f8b 100644 --- a/crates/markkit-js/README.md +++ b/crates/markkit-js/README.md @@ -39,6 +39,6 @@ rendering; `maxProcessingSecs` is the whole-second form), `maxNodes`, `maxNestin `includeSelectors`, `excludeSelectors` (added to the defaults), `maxArchiveEntries`, `maxArchiveEntrySize`, `maxArchiveTotalSize`, `maxSheetRows`, `maxSheetCols`, `maxSpreadsheetCells`, `includeHiddenSheets`, `allowPrivateNetworks` (trusted URLs only), -`allowedHosts`, `maxRedirects`. +`allowedHosts`, `maxRedirects`, `maxPdfPages`, `maxPdfStreamSize`, `maxPdfDecodedSize`. For browsers and edge runtimes use [`@everruns/markkit-wasm`](https://www.npmjs.com/package/@everruns/markkit-wasm). diff --git a/crates/markkit-js/__test__/options.test.cjs b/crates/markkit-js/__test__/options.test.cjs index aa60a0c..e469028 100644 --- a/crates/markkit-js/__test__/options.test.cjs +++ b/crates/markkit-js/__test__/options.test.cjs @@ -45,6 +45,7 @@ test("baseUrl and mainContent options", () => { test("invalid numeric options are rejected", () => { assert.throws(() => m.extractHtmlSync("

x

", { maxArchiveTotalSize: -1 }), /invalid options/); assert.throws(() => m.extractHtmlSync("

x

", { maxSpreadsheetCells: 1.5 }), /invalid options/); + assert.throws(() => m.extractHtmlSync("

x

", { maxPdfDecodedSize: -1 }), /invalid options/); assert.throws(() => m.extractHtmlSync("

x

", { timeout: -1 }), /invalid options/); assert.throws(() => m.extractHtmlSync("

x

", { timeout: NaN }), /invalid options/); }); @@ -55,7 +56,7 @@ test("extractDocument returns metadata", async () => { assert.equal(out.format, "html"); assert.deepEqual(out.warnings, []); assert.equal(out.title ?? null, null); - assert.throws(() => m.extractDocumentSync(Buffer.from("%PDF-1.7"), { fileName: "a.pdf" }), /Unsupported/); + assert.throws(() => m.extractDocumentSync(Buffer.from("PK"), { fileName: "a.pptx" }), /Unsupported/); }); test("extractDocument exposes page metadata", () => { diff --git a/crates/markkit-js/src/lib.rs b/crates/markkit-js/src/lib.rs index 84483c9..8b05e1c 100644 --- a/crates/markkit-js/src/lib.rs +++ b/crates/markkit-js/src/lib.rs @@ -52,6 +52,9 @@ pub struct MarkkitOptions { pub allowed_hosts: Option>, /// Maximum redirects `extractUrl` follows (default 5). pub max_redirects: Option, + pub max_pdf_pages: Option, + pub max_pdf_stream_size: Option, + pub max_pdf_decoded_size: Option, } /// Format hints for `extractDocument`. @@ -194,6 +197,15 @@ fn markkit(options: Option) -> napi::Result { if let Some(v) = o.max_redirects { b = b.max_redirects(v as usize); } + if let Some(v) = o.max_pdf_pages { + b = b.max_pdf_pages(v as usize); + } + if let Some(v) = o.max_pdf_stream_size { + b = b.max_pdf_stream_size(to_u64(v, "maxPdfStreamSize")? as usize); + } + if let Some(v) = o.max_pdf_decoded_size { + b = b.max_pdf_decoded_size(to_u64(v, "maxPdfDecodedSize")? as usize); + } Ok(b.build()) } diff --git a/crates/markkit-python/README.md b/crates/markkit-python/README.md index 437da4c..38e2621 100644 --- a/crates/markkit-python/README.md +++ b/crates/markkit-python/README.md @@ -26,7 +26,8 @@ markkit.extract_html_sync("

Hello

", limits=limits) `Limits` fields: `max_input_size`, `max_output_size`, `timeout` (seconds), `max_nesting_depth`, `max_nodes`, `max_archive_entries`, `max_archive_entry_size`, `max_archive_total_size`, `max_sheet_rows`, `max_sheet_cols`, -`max_spreadsheet_cells`, `max_redirects`. Every function accepts `limits=`. +`max_spreadsheet_cells`, `max_redirects`, `max_pdf_pages`, `max_pdf_stream_size`, +`max_pdf_decoded_size`. Every function accepts `limits=`. Publication verifies wheel metadata, installs the actual wheel into a clean virtual environment and exercises native extraction before upload. diff --git a/crates/markkit-python/src/lib.rs b/crates/markkit-python/src/lib.rs index d28c0f6..71b28a9 100644 --- a/crates/markkit-python/src/lib.rs +++ b/crates/markkit-python/src/lib.rs @@ -10,7 +10,8 @@ use pyo3::prelude::*; /// Keyword arguments: max_input_size, max_output_size, timeout (seconds, /// float), max_nesting_depth, max_nodes, max_archive_entries, /// max_archive_entry_size, max_archive_total_size, max_sheet_rows, -/// max_sheet_cols, max_spreadsheet_cells, max_redirects. +/// max_sheet_cols, max_spreadsheet_cells, max_redirects, max_pdf_pages, +/// max_pdf_stream_size, max_pdf_decoded_size. #[pyclass(name = "Limits", module = "markkit", frozen, get_all)] #[derive(Clone)] struct Limits { @@ -26,6 +27,9 @@ struct Limits { max_sheet_cols: usize, max_spreadsheet_cells: u64, max_redirects: usize, + max_pdf_pages: usize, + max_pdf_stream_size: usize, + max_pdf_decoded_size: usize, } #[pymethods] @@ -45,6 +49,9 @@ impl Limits { max_sheet_cols=None, max_spreadsheet_cells=None, max_redirects=None, + max_pdf_pages=None, + max_pdf_stream_size=None, + max_pdf_decoded_size=None, ))] #[allow(clippy::too_many_arguments)] fn new( @@ -60,6 +67,9 @@ impl Limits { max_sheet_cols: Option, max_spreadsheet_cells: Option, max_redirects: Option, + max_pdf_pages: Option, + max_pdf_stream_size: Option, + max_pdf_decoded_size: Option, ) -> PyResult { let d = markkit::Limits::default(); if let Some(t) = timeout { @@ -79,6 +89,9 @@ impl Limits { max_sheet_cols: max_sheet_cols.unwrap_or(d.max_sheet_cols), max_spreadsheet_cells: max_spreadsheet_cells.unwrap_or(d.max_spreadsheet_cells), max_redirects: max_redirects.unwrap_or(d.max_redirects), + max_pdf_pages: max_pdf_pages.unwrap_or(d.max_pdf_pages), + max_pdf_stream_size: max_pdf_stream_size.unwrap_or(d.max_pdf_stream_size), + max_pdf_decoded_size: max_pdf_decoded_size.unwrap_or(d.max_pdf_decoded_size), }) } @@ -87,7 +100,8 @@ impl Limits { "Limits(max_input_size={}, max_output_size={}, timeout={}, max_nesting_depth={}, \ max_nodes={}, max_archive_entries={}, max_archive_entry_size={}, \ max_archive_total_size={}, max_sheet_rows={}, max_sheet_cols={}, \ - max_spreadsheet_cells={}, max_redirects={})", + max_spreadsheet_cells={}, max_redirects={}, max_pdf_pages={}, \ + max_pdf_stream_size={}, max_pdf_decoded_size={})", self.max_input_size, self.max_output_size, self.timeout, @@ -99,7 +113,10 @@ impl Limits { self.max_sheet_rows, self.max_sheet_cols, self.max_spreadsheet_cells, - self.max_redirects + self.max_redirects, + self.max_pdf_pages, + self.max_pdf_stream_size, + self.max_pdf_decoded_size ) } } @@ -120,6 +137,9 @@ impl Limits { .max_sheet_cols(self.max_sheet_cols) .max_spreadsheet_cells(self.max_spreadsheet_cells) .max_redirects(self.max_redirects) + .max_pdf_pages(self.max_pdf_pages) + .max_pdf_stream_size(self.max_pdf_stream_size) + .max_pdf_decoded_size(self.max_pdf_decoded_size) } } diff --git a/crates/markkit/Cargo.toml b/crates/markkit/Cargo.toml index 34f30e1..ba1edfd 100644 --- a/crates/markkit/Cargo.toml +++ b/crates/markkit/Cargo.toml @@ -16,7 +16,7 @@ readme = "../../README.md" [features] default = ["html"] html = ["dep:scraper", "dep:html5ever", "dep:ego-tree", "dep:url"] -pdf = [] +pdf = ["dep:hayro-syntax", "dep:miniz_oxide"] docx = ["dep:zip", "dep:quick-xml"] spreadsheet = ["dep:calamine", "dep:zip", "dep:quick-xml", "dep:cfb"] markdown = ["dep:pulldown-cmark"] @@ -55,6 +55,10 @@ quick-xml = { workspace = true, optional = true } calamine = { workspace = true, optional = true } cfb = { workspace = true, optional = true } +# PDF (optional) +hayro-syntax = { workspace = true, optional = true } +miniz_oxide = { workspace = true, optional = true } + # Markdown (optional) pulldown-cmark = { workspace = true, optional = true } @@ -79,6 +83,7 @@ criterion.workspace = true tower = { workspace = true } zip = { workspace = true } cfb = { workspace = true } +miniz_oxide = { workspace = true } [[example]] name = "basic" diff --git a/crates/markkit/src/config.rs b/crates/markkit/src/config.rs index ab0e7f0..f6c66d3 100644 --- a/crates/markkit/src/config.rs +++ b/crates/markkit/src/config.rs @@ -195,6 +195,21 @@ impl MarkkitBuilder { self } + pub fn max_pdf_pages(mut self, n: usize) -> Self { + self.limits.max_pdf_pages = n; + self + } + + pub fn max_pdf_stream_size(mut self, bytes: usize) -> Self { + self.limits.max_pdf_stream_size = bytes; + self + } + + pub fn max_pdf_decoded_size(mut self, bytes: usize) -> Self { + self.limits.max_pdf_decoded_size = bytes; + self + } + /// Resolve relative HTML URLs against this absolute URL. Ignored when it /// does not parse or cannot be a base (`data:`, `javascript:`, ...). pub fn base_url(mut self, url: impl Into) -> Self { diff --git a/crates/markkit/src/extract.rs b/crates/markkit/src/extract.rs index ae904ee..f2f9608 100644 --- a/crates/markkit/src/extract.rs +++ b/crates/markkit/src/extract.rs @@ -15,14 +15,14 @@ pub struct Extraction { pub markdown: String, /// Document title when known: HTML `og:title` / JSON-LD / ``; DOCX /// core properties or `Title` style; XLSX `docProps/core.xml`, ODS - /// `meta.xml`, XLS `SummaryInformation`; + /// `meta.xml`, XLS `SummaryInformation`; PDF `/Info /Title`; /// Markdown front matter `title:` or first `# ` heading. pub title: Option<String>, /// Format the document was parsed as. pub format: Format, /// Non-fatal issues, each prefixed with a stable code /// (`truncated_sheet:`, `hidden_sheet_skipped:`, `sheet_error:`, - /// `invalid_utf8:`). + /// `truncated_pdf:`, `pdf_unmapped_font:`, `pdf_no_text:`, `invalid_utf8:`). pub warnings: Vec<String>, /// Format-specific counters. pub stats: ExtractionStats, @@ -111,7 +111,13 @@ impl ParsedDocument { } /// [`ParsedDocument::render`] with `max_output_size` and the deadline enforced. - #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + #[cfg(any( + feature = "html", + feature = "docx", + feature = "spreadsheet", + feature = "pdf", + feature = "markdown" + ))] fn render_bounded(self, max_output: usize, budget: &Budget) -> crate::Result<Extraction> { let markdown = crate::render::MarkdownRenderer::new().render_bounded( &self.document, @@ -136,7 +142,7 @@ impl ParsedDocument { /// Run a parser, converting a panic into an error (defence in depth for /// third-party parsers; only effective when built with `panic = "unwind"`, which /// markkit's own release profile uses). -#[cfg(any(feature = "docx", feature = "spreadsheet"))] +#[cfg(any(feature = "docx", feature = "spreadsheet", feature = "pdf"))] fn guarded<T>( what: &str, make_err: fn(String) -> Error, @@ -224,7 +230,7 @@ impl Markkit { .map(|e| e.markdown) } - /// Convert a document (HTML, DOCX, XLSX/XLS/ODS, plain text, Markdown) to Markdown with metadata. + /// Convert a document (HTML, PDF, DOCX, XLSX/XLS/ODS, plain text, Markdown) to Markdown with metadata. /// /// The format is detected from `hint.content_type`, then the extension of /// `hint.file_name`, then magic bytes. Parsing is CPU-bound and runs inline; @@ -257,11 +263,23 @@ impl Markkit { budget: &Budget, ) -> crate::Result<Extraction> { let parsed = self.parse_document_with(bytes, hint, budget)?; - #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + #[cfg(any( + feature = "html", + feature = "docx", + feature = "spreadsheet", + feature = "pdf", + feature = "markdown" + ))] { parsed.render_bounded(self.config.limits.max_output_size, budget) } - #[cfg(not(any(feature = "html", feature = "docx", feature = "spreadsheet")))] + #[cfg(not(any( + feature = "html", + feature = "docx", + feature = "spreadsheet", + feature = "pdf", + feature = "markdown" + )))] Ok(parsed.render()) } @@ -294,7 +312,7 @@ impl Markkit { match Format::detect(bytes, hint) { Format::Html => self.parse_html_doc(bytes, budget), - Format::Pdf => self.parse_pdf(bytes), + Format::Pdf => self.parse_pdf(bytes, budget), Format::Docx => self.parse_docx(bytes, budget), Format::Xlsx | Format::Xlsb | Format::Xls | Format::Ods => { self.parse_spreadsheet(bytes, budget) @@ -445,14 +463,23 @@ impl Markkit { ) } - /// PDF extraction is not implemented yet; report it explicitly rather than - /// returning an empty (misleading) success. - fn parse_pdf(&self, _bytes: &[u8]) -> crate::Result<ParsedDocument> { - #[cfg(feature = "pdf")] - let msg = "PDF extraction is not implemented yet"; - #[cfg(not(feature = "pdf"))] - let msg = "PDF support requires the 'pdf' feature"; - Err(Error::UnsupportedFormat(msg.to_string())) + #[cfg(feature = "pdf")] + fn parse_pdf(&self, bytes: &[u8], budget: &Budget) -> crate::Result<ParsedDocument> { + let out = guarded("PDF", Error::PdfExtract, || { + crate::pdf::parse(bytes, &self.config, budget) + })?; + let mut p = ParsedDocument::new(out.doc, Format::Pdf); + p.title = out.title; + p.warnings = out.warnings; + p.stats.pages = Some(out.pages); + Ok(p) + } + + #[cfg(not(feature = "pdf"))] + fn parse_pdf(&self, _bytes: &[u8], _budget: &Budget) -> crate::Result<ParsedDocument> { + Err(Error::UnsupportedFormat( + "PDF support requires the 'pdf' feature".to_string(), + )) } /// Access the current configuration. diff --git a/crates/markkit/src/lib.rs b/crates/markkit/src/lib.rs index afb874d..696d56a 100644 --- a/crates/markkit/src/lib.rs +++ b/crates/markkit/src/lib.rs @@ -36,7 +36,8 @@ //! - **Spreadsheet** (`spreadsheet`): Convert XLSX/XLSM, XLS and ODS workbooks to tables //! - **Plain text** (always on): `text/plain`, `.txt` //! - **Markdown** (`markdown`): `text/markdown`, `.md`, normalized through the IR -//! - **PDF** (`pdf`): Planned; currently returns [`Error::UnsupportedFormat`] +//! - **PDF** (`pdf`): Text layer of PDF documents (no OCR); encrypted PDFs are +//! rejected with [`Error::UnsupportedFormat`] //! - **URL fetching** (`http_client`, opt-in): Fetch and extract from URLs via reqwest //! - **Agent tool** (`tool`): `markkit::tool::MarkkitTool`, an LLM tool following the //! Everruns toolkit-library contract (schemas, OpenAI definition, `tower::Service`) @@ -62,6 +63,8 @@ pub mod ir; pub mod limits; #[cfg(feature = "markdown")] mod markdown; +#[cfg(feature = "pdf")] +mod pdf; pub mod render; #[cfg(feature = "spreadsheet")] mod spreadsheet; diff --git a/crates/markkit/src/limits.rs b/crates/markkit/src/limits.rs index 9477ee9..7ebe40e 100644 --- a/crates/markkit/src/limits.rs +++ b/crates/markkit/src/limits.rs @@ -69,6 +69,13 @@ pub struct Limits { pub max_spreadsheet_cells: u64, /// Maximum redirects `extract_url` follows. Default: 5. pub max_redirects: usize, + /// Maximum PDF pages rendered (rest skipped with a warning). Default: 10 000. + pub max_pdf_pages: usize, + /// Maximum decoded size of a single PDF stream in bytes. Default: 64 MiB. + pub max_pdf_stream_size: usize, + /// Maximum total decoded size of all PDF streams, and of content + /// interpreted (form XObjects count once per use), in bytes. Default: 256 MiB. + pub max_pdf_decoded_size: usize, } impl Default for Limits { @@ -86,6 +93,9 @@ impl Default for Limits { max_sheet_cols: 256, max_spreadsheet_cells: 4_000_000, max_redirects: 5, + max_pdf_pages: 10_000, + max_pdf_stream_size: 64 * 1024 * 1024, + max_pdf_decoded_size: 256 * 1024 * 1024, } } } @@ -161,6 +171,21 @@ impl Limits { self } + pub fn max_pdf_pages(mut self, n: usize) -> Self { + self.max_pdf_pages = n; + self + } + + pub fn max_pdf_stream_size(mut self, bytes: usize) -> Self { + self.max_pdf_stream_size = bytes; + self + } + + pub fn max_pdf_decoded_size(mut self, bytes: usize) -> Self { + self.max_pdf_decoded_size = bytes; + self + } + /// Reject inputs larger than `max_input_size`. pub(crate) fn check_input(&self, len: usize) -> crate::Result<()> { if len > self.max_input_size { @@ -176,7 +201,12 @@ impl Limits { // Budget is only driven by the feature-gated parsers. /// How many [`Budget::expired`] calls share one clock read. #[cfg_attr( - not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + not(any( + feature = "html", + feature = "docx", + feature = "spreadsheet", + feature = "pdf" + )), allow(dead_code) )] const CLOCK_EVERY: u32 = 256; @@ -185,7 +215,12 @@ const CLOCK_EVERY: u32 = 256; /// the renderer (interior mutability so `&self` walkers can charge it). #[derive(Debug)] #[cfg_attr( - not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + not(any( + feature = "html", + feature = "docx", + feature = "spreadsheet", + feature = "pdf" + )), allow(dead_code) )] pub(crate) struct Budget { @@ -200,7 +235,12 @@ pub(crate) struct Budget { } #[cfg_attr( - not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + not(any( + feature = "html", + feature = "docx", + feature = "spreadsheet", + feature = "pdf" + )), allow(dead_code) )] impl Budget { diff --git a/crates/markkit/src/pdf/font.rs b/crates/markkit/src/pdf/font.rs new file mode 100644 index 0000000..7af7dff --- /dev/null +++ b/crates/markkit/src/pdf/font.rs @@ -0,0 +1,907 @@ +//! PDF font → Unicode mapping and glyph widths for the text layer. +//! +//! Decision: Unicode comes from, in order, the font's `ToUnicode` CMap, the +//! simple-font encoding (`/Encoding` base + `/Differences` glyph names resolved +//! through an Adobe Glyph List subset), and finally WinAnsi for simple fonts +//! with no usable encoding. Composite (Type0) fonts without `ToUnicode` carry +//! no reliable Unicode mapping; their text is dropped and reported via the +//! `pdf_unmapped_font` warning instead of emitting CID garbage. +//! +//! CMap `bfrange`s are kept as ranges (never expanded) and looked up by binary +//! search, so a `<0000> <FFFFFFFF>` range costs nothing. + +use std::collections::HashMap; + +use hayro_syntax::object::{Array, Dict, Name, Object, Stream}; + +use super::guard::{self, Kind}; + +/// One decoded character code. +pub(crate) struct Glyph { + pub text: Option<String>, + /// Advance in text-space units (glyph width / 1000). + pub width: f32, + /// Single-byte code 32 (word spacing applies). + pub is_space: bool, +} + +pub(crate) struct Font { + cmap: Option<CMap>, + simple: Option<Vec<Option<String>>>, + composite: bool, + first_char: u32, + widths: Vec<f32>, + cid_widths: Vec<(u32, u32, f32)>, + default_width: f32, + /// Composite font with no ToUnicode: text cannot be mapped. + pub unmapped: bool, + pub name: String, +} + +impl Font { + pub(crate) fn load(dict: &Dict<'_>, cap: usize) -> Self { + let subtype = name_of(dict.get::<Name<'_>>(b"Subtype")); + let composite = subtype == "Type0"; + let cmap = dict + .get::<Stream<'_>>(b"ToUnicode") + .and_then(|s| decode_stream(&s, cap)) + .map(|d| CMap::parse(&d)); + let name = name_of(dict.get::<Name<'_>>(b"BaseFont")); + + let mut font = Font { + cmap, + simple: None, + composite, + first_char: 0, + widths: Vec::new(), + cid_widths: Vec::new(), + default_width: 0.5, + unmapped: false, + name, + }; + + if composite { + let desc = dict + .get::<Array<'_>>(b"DescendantFonts") + .and_then(|a| a.iter::<Dict<'_>>().next()); + if let Some(desc) = desc { + font.default_width = desc.get::<f32>(b"DW").unwrap_or(1000.0) / 1000.0; + if let Some(w) = desc.get::<Array<'_>>(b"W") { + font.cid_widths = parse_cid_widths(&w); + } + } else { + font.default_width = 1.0; + } + font.unmapped = font.cmap.as_ref().is_none_or(|c| c.is_empty()); + } else { + font.simple = Some(simple_encoding(dict)); + font.first_char = dict.get::<u32>(b"FirstChar").unwrap_or(0); + if let Some(w) = dict.get::<Array<'_>>(b"Widths") { + font.widths = w + .iter::<Object<'_>>() + .take(256) + .map(|o| o.into_f32().unwrap_or(0.0) / 1000.0) + .collect(); + } + if let Some(mw) = dict + .get::<Dict<'_>>(b"FontDescriptor") + .and_then(|d| d.get::<f32>(b"MissingWidth")) + .filter(|w| *w > 0.0) + { + font.default_width = mw / 1000.0; + } + } + font + } + + /// Fallback for a missing / unloadable font: single-byte WinAnsi. + pub(crate) fn fallback() -> Self { + Font { + cmap: None, + simple: Some(to_strings(base_table(BaseEncoding::WinAnsi))), + composite: false, + first_char: 0, + widths: Vec::new(), + cid_widths: Vec::new(), + default_width: 0.5, + unmapped: false, + name: String::new(), + } + } + + pub(crate) fn decode(&self, bytes: &[u8], out: &mut Vec<Glyph>) { + let mut i = 0; + while i < bytes.len() { + let len = self.code_len(&bytes[i..]); + let code = bytes[i..i + len] + .iter() + .fold(0u32, |a, &b| (a << 8) | b as u32); + i += len; + let text = match self.cmap.as_ref().and_then(|c| c.lookup(code)) { + Some(s) => Some(s), + None if self.composite => None, + None => self + .simple + .as_ref() + .and_then(|t| t[(code & 0xFF) as usize].clone()), + }; + out.push(Glyph { + text, + width: self.width(code), + is_space: len == 1 && code == 32, + }); + } + } + + fn code_len(&self, rest: &[u8]) -> usize { + if let Some(cmap) = &self.cmap + && !cmap.codespace.is_empty() + { + for len in 1..=4.min(rest.len()) { + let v = rest[..len].iter().fold(0u32, |a, &b| (a << 8) | b as u32); + if cmap + .codespace + .iter() + .any(|&(l, lo, hi)| l as usize == len && (lo..=hi).contains(&v)) + { + return len; + } + } + } + if self.composite { 2.min(rest.len()) } else { 1 } + } + + fn width(&self, code: u32) -> f32 { + if self.composite { + let i = self.cid_widths.partition_point(|&(lo, _, _)| lo <= code); + if i > 0 { + let (lo, hi, w) = self.cid_widths[i - 1]; + if (lo..=hi).contains(&code) { + return w; + } + } + return self.default_width; + } + code.checked_sub(self.first_char) + .and_then(|i| self.widths.get(i as usize)) + .copied() + .filter(|w| *w > 0.0) + .unwrap_or(self.default_width) + } +} + +pub(crate) fn name_of(n: Option<Name<'_>>) -> String { + n.map(|n| n.as_str().to_string()).unwrap_or_default() +} + +/// Decode a stream through the bounded decoders (never hayro's `decoded()`). +pub(crate) fn decode_stream(s: &Stream<'_>, cap: usize) -> Option<Vec<u8>> { + let chain: Vec<Kind> = s.filters().into_iter().map(Kind::from_filter).collect(); + if chain.contains(&Kind::Image) { + return None; + } + guard::decode(&chain, &s.raw_data(), cap).ok() +} + +/// `/W` array: `c [w1 w2 ...]` and `c_first c_last w` forms, sorted by start. +fn parse_cid_widths(w: &Array<'_>) -> Vec<(u32, u32, f32)> { + const MAX: usize = 65_536; + let items: Vec<Object<'_>> = w.iter::<Object<'_>>().take(MAX * 3).collect(); + let mut out = Vec::new(); + let mut i = 0; + while i < items.len() && out.len() < MAX { + let Some(start) = items[i].clone().into_f32().map(|v| v as u32) else { + break; + }; + match items.get(i + 1) { + Some(Object::Array(arr)) => { + for (k, o) in arr.iter::<Object<'_>>().take(MAX).enumerate() { + let wv = o.into_f32().unwrap_or(0.0) / 1000.0; + let c = start.saturating_add(k as u32); + out.push((c, c, wv)); + } + i += 2; + } + Some(Object::Number(end)) => { + let wv = items + .get(i + 2) + .and_then(|o| o.clone().into_f32()) + .unwrap_or(0.0) + / 1000.0; + out.push((start, end.as_f64().max(0.0) as u32, wv)); + i += 3; + } + _ => break, + } + } + out.sort_by_key(|&(lo, _, _)| lo); + out +} + +// ─── ToUnicode CMap ───────────────────────────────────────────────── + +#[derive(Default)] +struct CMap { + codespace: Vec<(u8, u32, u32)>, + chars: HashMap<u32, String>, + /// (lo, hi, dst): dst is the UTF-16 for `lo` (last unit incremented), or an + /// explicit per-code array. + ranges: Vec<(u32, u32, RangeDst)>, +} + +enum RangeDst { + Start(Vec<u16>), + List(Vec<String>), +} + +#[derive(Debug, PartialEq)] +enum Tok<'a> { + Hex(Vec<u8>), + Word(&'a [u8]), + Open, + Close, +} + +fn tokens(data: &[u8]) -> Vec<Tok<'_>> { + let mut out = Vec::new(); + let mut i = 0; + while i < data.len() { + match data[i] { + b'%' => { + while i < data.len() && data[i] != b'\n' && data[i] != b'\r' { + i += 1; + } + } + b'<' if data.get(i + 1) == Some(&b'<') => i += 2, + b'>' if data.get(i + 1) == Some(&b'>') => i += 2, + b'<' => { + let end = data[i..].iter().position(|&b| b == b'>').map(|p| i + p); + let end = end.unwrap_or(data.len()); + let digits: Vec<u8> = data[i + 1..end] + .iter() + .filter_map(|b| (*b as char).to_digit(16).map(|d| d as u8)) + .collect(); + let mut bytes: Vec<u8> = digits + .chunks(2) + .map(|c| c[0] << 4 | c.get(1).copied().unwrap_or(0)) + .collect(); + if digits.is_empty() { + bytes.clear(); + } + out.push(Tok::Hex(bytes)); + i = end + 1; + } + b'[' => { + out.push(Tok::Open); + i += 1; + } + b']' => { + out.push(Tok::Close); + i += 1; + } + b if b.is_ascii_whitespace() => i += 1, + _ => { + let start = i; + while i < data.len() + && !data[i].is_ascii_whitespace() + && !matches!(data[i], b'<' | b'>' | b'[' | b']' | b'%') + { + i += 1; + } + if i == start { + i += 1; + } else { + out.push(Tok::Word(&data[start..i])); + } + } + } + } + out +} + +fn be(bytes: &[u8]) -> u32 { + bytes.iter().take(4).fold(0u32, |a, &b| (a << 8) | b as u32) +} + +fn utf16(bytes: &[u8]) -> Vec<u16> { + bytes + .chunks(2) + .map(|c| u16::from_be_bytes([c[0], c.get(1).copied().unwrap_or(0)])) + .collect() +} + +fn utf16_string(units: &[u16]) -> String { + char::decode_utf16(units.iter().copied()) + .map(|r| r.unwrap_or('\u{FFFD}')) + .collect() +} + +impl CMap { + fn is_empty(&self) -> bool { + self.chars.is_empty() && self.ranges.is_empty() + } + + fn parse(data: &[u8]) -> Self { + let toks = tokens(data); + let mut cmap = CMap::default(); + let mut i = 0; + let hex = |t: Option<&Tok<'_>>| match t { + Some(Tok::Hex(b)) => Some(b.clone()), + _ => None, + }; + while i < toks.len() { + match &toks[i] { + Tok::Word(b"begincodespacerange") => { + i += 1; + while let (Some(lo), Some(hi)) = (hex(toks.get(i)), hex(toks.get(i + 1))) { + if (1..=4).contains(&lo.len()) { + cmap.codespace.push((lo.len() as u8, be(&lo), be(&hi))); + } + i += 2; + } + } + Tok::Word(b"beginbfchar") => { + i += 1; + while let (Some(src), Some(dst)) = (hex(toks.get(i)), hex(toks.get(i + 1))) { + cmap.chars.insert(be(&src), utf16_string(&utf16(&dst))); + i += 2; + } + } + Tok::Word(b"beginbfrange") => { + i += 1; + while let (Some(lo), Some(hi)) = (hex(toks.get(i)), hex(toks.get(i + 1))) { + let (lo, hi) = (be(&lo), be(&hi)); + match toks.get(i + 2) { + Some(Tok::Hex(dst)) => { + if hi >= lo && !dst.is_empty() { + cmap.ranges.push((lo, hi, RangeDst::Start(utf16(dst)))); + } + i += 3; + } + Some(Tok::Open) => { + let mut list = Vec::new(); + i += 3; + while let Some(Tok::Hex(d)) = toks.get(i) { + list.push(utf16_string(&utf16(d))); + i += 1; + } + if toks.get(i) == Some(&Tok::Close) { + i += 1; + } + if hi >= lo { + cmap.ranges.push((lo, hi, RangeDst::List(list))); + } + } + _ => break, + } + } + } + _ => i += 1, + } + } + cmap.ranges.sort_by_key(|r| r.0); + cmap + } + + fn lookup(&self, code: u32) -> Option<String> { + if let Some(s) = self.chars.get(&code) { + return Some(s.clone()); + } + let i = self.ranges.partition_point(|r| r.0 <= code); + // Ranges may overlap; check a few preceding candidates. + for (lo, hi, dst) in self.ranges[..i].iter().rev().take(8) { + if code > *hi { + continue; + } + let off = code - lo; + return match dst { + RangeDst::List(l) => l.get(off as usize).cloned(), + RangeDst::Start(units) => { + let mut u = units.clone(); + let last = u.last_mut()?; + *last = last.wrapping_add(off as u16); + Some(utf16_string(&u)) + } + }; + } + None + } +} + +// ─── Simple-font encodings ────────────────────────────────────────── + +#[derive(Clone, Copy)] +enum BaseEncoding { + Standard, + WinAnsi, + MacRoman, +} + +fn simple_encoding(dict: &Dict<'_>) -> Vec<Option<String>> { + let base_of = |n: &str| match n { + "WinAnsiEncoding" => Some(BaseEncoding::WinAnsi), + "MacRomanEncoding" => Some(BaseEncoding::MacRoman), + "StandardEncoding" => Some(BaseEncoding::Standard), + _ => None, + }; + // Unknown / absent base: WinAnsi agrees with Standard on printable ASCII and + // covers the high range that symbolic TrueType fonts usually use. + let mut base = BaseEncoding::WinAnsi; + let mut diffs = None; + if let Some(n) = dict.get::<Name<'_>>(b"Encoding") { + base = base_of(n.as_str()).unwrap_or(base); + } else if let Some(enc) = dict.get::<Dict<'_>>(b"Encoding") { + if let Some(b) = enc + .get::<Name<'_>>(b"BaseEncoding") + .and_then(|n| base_of(n.as_str())) + { + base = b; + } + diffs = enc.get::<Array<'_>>(b"Differences"); + } + let mut table = to_strings(base_table(base)); + if let Some(diffs) = diffs { + let mut code: usize = 0; + for o in diffs.iter::<Object<'_>>().take(1024) { + match o { + Object::Number(n) => code = n.as_f64().clamp(0.0, 255.0) as usize, + Object::Name(n) => { + if code < 256 { + table[code] = glyph_name_text(n.as_str()); + } + code += 1; + } + _ => {} + } + } + } + table +} + +const WIN_ANSI_80: [char; 32] = [ + '€', '\0', '‚', 'ƒ', '„', '…', '†', '‡', 'ˆ', '‰', 'Š', '‹', 'Œ', '\0', 'Ž', '\0', '\0', '‘', + '’', '“', '”', '•', '–', '—', '˜', '™', 'š', '›', 'œ', '\0', 'ž', 'Ÿ', +]; + +const MAC_ROMAN_80: &str = "ÄÅÇÉÑÖÜáàâäãåçéèêëíìîïñóòôöõúùûü†°¢£§•¶ß®©™´¨≠ÆØ∞±≤≥¥µ∂∑∏π∫ªºΩæø¿¡¬√ƒ≈∆«»…\u{A0}ÀÃÕŒœ–—“”‘’÷◊ÿŸ⁄€‹›fifl‡·‚„‰ÂÊÁËÈÍÎÏÌÓÔ\u{F8FF}ÒÚÛÙıˆ˜¯˘˙˚¸˝˛ˇ"; + +const STANDARD_HIGH: &[(u8, char)] = &[ + (0xA1, '¡'), + (0xA2, '¢'), + (0xA3, '£'), + (0xA4, '⁄'), + (0xA5, '¥'), + (0xA6, 'ƒ'), + (0xA7, '§'), + (0xA8, '¤'), + (0xA9, '\''), + (0xAA, '“'), + (0xAB, '«'), + (0xAC, '‹'), + (0xAD, '›'), + (0xAE, 'fi'), + (0xAF, 'fl'), + (0xB1, '–'), + (0xB2, '†'), + (0xB3, '‡'), + (0xB4, '·'), + (0xB6, '¶'), + (0xB7, '•'), + (0xB8, '‚'), + (0xB9, '„'), + (0xBA, '”'), + (0xBB, '»'), + (0xBC, '…'), + (0xBD, '‰'), + (0xBF, '¿'), + (0xC1, '`'), + (0xC2, '´'), + (0xC3, 'ˆ'), + (0xC4, '˜'), + (0xC5, '¯'), + (0xC6, '˘'), + (0xC7, '˙'), + (0xC8, '¨'), + (0xCA, '˚'), + (0xCB, '¸'), + (0xCD, '˝'), + (0xCE, '˛'), + (0xCF, 'ˇ'), + (0xD0, '—'), + (0xE1, 'Æ'), + (0xE3, 'ª'), + (0xE8, 'Ł'), + (0xE9, 'Ø'), + (0xEA, 'Œ'), + (0xEB, 'º'), + (0xF1, 'æ'), + (0xF5, 'ı'), + (0xF8, 'ł'), + (0xF9, 'ø'), + (0xFA, 'œ'), + (0xFB, 'ß'), +]; + +fn base_table(base: BaseEncoding) -> [Option<char>; 256] { + let mut t = [None; 256]; + for (c, slot) in t.iter_mut().enumerate().take(0x7F).skip(0x20) { + *slot = Some(c as u8 as char); + } + match base { + BaseEncoding::WinAnsi => { + for (i, c) in WIN_ANSI_80.iter().enumerate() { + t[0x80 + i] = (*c != '\0').then_some(*c); + } + for (c, slot) in t.iter_mut().enumerate().skip(0xA0) { + *slot = char::from_u32(c as u32); + } + } + BaseEncoding::MacRoman => { + for (i, c) in MAC_ROMAN_80.chars().enumerate() { + t[0x80 + i] = Some(c); + } + } + BaseEncoding::Standard => { + t[0x27] = Some('’'); + t[0x60] = Some('‘'); + for &(code, c) in STANDARD_HIGH { + t[code as usize] = Some(c); + } + } + } + t +} + +/// Glyph names for U+0020..U+007E, then U+00A0..U+00FF. +const ASCII_NAMES: [&str; 95] = [ + "space", + "exclam", + "quotedbl", + "numbersign", + "dollar", + "percent", + "ampersand", + "quotesingle", + "parenleft", + "parenright", + "asterisk", + "plus", + "comma", + "hyphen", + "period", + "slash", + "zero", + "one", + "two", + "three", + "four", + "five", + "six", + "seven", + "eight", + "nine", + "colon", + "semicolon", + "less", + "equal", + "greater", + "question", + "at", + "A", + "B", + "C", + "D", + "E", + "F", + "G", + "H", + "I", + "J", + "K", + "L", + "M", + "N", + "O", + "P", + "Q", + "R", + "S", + "T", + "U", + "V", + "W", + "X", + "Y", + "Z", + "bracketleft", + "backslash", + "bracketright", + "asciicircum", + "underscore", + "grave", + "a", + "b", + "c", + "d", + "e", + "f", + "g", + "h", + "i", + "j", + "k", + "l", + "m", + "n", + "o", + "p", + "q", + "r", + "s", + "t", + "u", + "v", + "w", + "x", + "y", + "z", + "braceleft", + "bar", + "braceright", + "asciitilde", +]; + +const LATIN1_NAMES: [&str; 96] = [ + "nbspace", + "exclamdown", + "cent", + "sterling", + "currency", + "yen", + "brokenbar", + "section", + "dieresis", + "copyright", + "ordfeminine", + "guillemotleft", + "logicalnot", + "sfthyphen", + "registered", + "macron", + "degree", + "plusminus", + "twosuperior", + "threesuperior", + "acute", + "mu", + "paragraph", + "periodcentered", + "cedilla", + "onesuperior", + "ordmasculine", + "guillemotright", + "onequarter", + "onehalf", + "threequarters", + "questiondown", + "Agrave", + "Aacute", + "Acircumflex", + "Atilde", + "Adieresis", + "Aring", + "AE", + "Ccedilla", + "Egrave", + "Eacute", + "Ecircumflex", + "Edieresis", + "Igrave", + "Iacute", + "Icircumflex", + "Idieresis", + "Eth", + "Ntilde", + "Ograve", + "Oacute", + "Ocircumflex", + "Otilde", + "Odieresis", + "multiply", + "Oslash", + "Ugrave", + "Uacute", + "Ucircumflex", + "Udieresis", + "Yacute", + "Thorn", + "germandbls", + "agrave", + "aacute", + "acircumflex", + "atilde", + "adieresis", + "aring", + "ae", + "ccedilla", + "egrave", + "eacute", + "ecircumflex", + "edieresis", + "igrave", + "iacute", + "icircumflex", + "idieresis", + "eth", + "ntilde", + "ograve", + "oacute", + "ocircumflex", + "otilde", + "odieresis", + "divide", + "oslash", + "ugrave", + "uacute", + "ucircumflex", + "udieresis", + "yacute", + "thorn", + "ydieresis", +]; + +const EXTRA_NAMES: &[(&str, &str)] = &[ + ("quoteleft", "‘"), + ("quoteright", "’"), + ("quotedblleft", "“"), + ("quotedblright", "”"), + ("quotesinglbase", "‚"), + ("quotedblbase", "„"), + ("endash", "–"), + ("emdash", "—"), + ("bullet", "•"), + ("ellipsis", "…"), + ("dagger", "†"), + ("daggerdbl", "‡"), + ("fi", "fi"), + ("fl", "fl"), + ("ff", "ff"), + ("ffi", "ffi"), + ("ffl", "ffl"), + ("trademark", "™"), + ("minus", "−"), + ("Euro", "€"), + ("florin", "ƒ"), + ("perthousand", "‰"), + ("guilsinglleft", "‹"), + ("guilsinglright", "›"), + ("circumflex", "ˆ"), + ("tilde", "˜"), + ("OE", "Œ"), + ("oe", "œ"), + ("Scaron", "Š"), + ("scaron", "š"), + ("Zcaron", "Ž"), + ("zcaron", "ž"), + ("Ydieresis", "Ÿ"), + ("dotlessi", "ı"), + ("Lslash", "Ł"), + ("lslash", "ł"), + ("fraction", "⁄"), + ("nonbreakingspace", "\u{A0}"), + ("middot", "·"), + ("hyphenminus", "-"), +]; + +/// Map a glyph name to text (AGL subset + `uniXXXX` / `uXXXX` conventions). +fn glyph_name_text(name: &str) -> Option<String> { + let base = name.split('.').next().unwrap_or(name); + if base.is_empty() { + return None; + } + if base.contains('_') { + let parts: Option<String> = base.split('_').map(glyph_name_text).collect(); + return parts; + } + if let Some(i) = ASCII_NAMES.iter().position(|n| *n == base) { + return Some(((0x20 + i) as u8 as char).to_string()); + } + if let Some(i) = LATIN1_NAMES.iter().position(|n| *n == base) { + return char::from_u32(0xA0 + i as u32).map(String::from); + } + if let Some((_, s)) = EXTRA_NAMES.iter().find(|(n, _)| *n == base) { + return Some(s.to_string()); + } + if let Some(hex) = base.strip_prefix("uni") + && hex.len() >= 4 + && hex.len() % 4 == 0 + && hex.bytes().all(|b| b.is_ascii_hexdigit()) + { + let units: Vec<u16> = (0..hex.len()) + .step_by(4) + .filter_map(|i| u16::from_str_radix(&hex[i..i + 4], 16).ok()) + .collect(); + return Some(utf16_string(&units)); + } + if let Some(hex) = base.strip_prefix('u') + && (4..=6).contains(&hex.len()) + && hex.bytes().all(|b| b.is_ascii_hexdigit()) + { + return u32::from_str_radix(hex, 16) + .ok() + .and_then(char::from_u32) + .map(String::from); + } + None +} + +fn to_strings(t: [Option<char>; 256]) -> Vec<Option<String>> { + t.iter().map(|c| c.map(String::from)).collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn mac_roman_table_is_complete() { + assert_eq!(MAC_ROMAN_80.chars().count(), 128); + } + + #[test] + fn glyph_names() { + let g = |n| glyph_name_text(n); + assert_eq!(g("A").as_deref(), Some("A")); + assert_eq!(g("eacute").as_deref(), Some("é")); + assert_eq!(g("quoteright").as_deref(), Some("’")); + assert_eq!(g("uni00E9").as_deref(), Some("é")); + assert_eq!(g("uni00410042").as_deref(), Some("AB")); + assert_eq!(g("u1F600").as_deref(), Some("😀")); + assert_eq!(g("a.sc").as_deref(), Some("a")); + assert_eq!(g("fi").as_deref(), Some("fi")); + assert_eq!(g("g123"), None); + assert_eq!(glyph_name_text("f_f_i").as_deref(), Some("ffi")); + } + + #[test] + fn cmap_bfchar_bfrange_and_codespace() { + let cmap = CMap::parse( + b"1 begincodespacerange <0000> <FFFF> endcodespacerange\n\ + 2 beginbfchar <0003> <0020> <0024> <00410042> endbfchar\n\ + 2 beginbfrange <0010> <0012> <0061> <0020> <0021> [<0058> <D83DDE00>] endbfrange", + ); + assert_eq!(cmap.codespace, vec![(2, 0, 0xFFFF)]); + assert_eq!(cmap.lookup(3).as_deref(), Some(" ")); + assert_eq!(cmap.lookup(0x24).as_deref(), Some("AB")); + assert_eq!(cmap.lookup(0x11).as_deref(), Some("b")); + assert_eq!(cmap.lookup(0x21).as_deref(), Some("😀")); + assert_eq!(cmap.lookup(0x13), None); + } + + #[test] + fn huge_bfrange_is_not_expanded() { + let cmap = CMap::parse(b"beginbfrange <00000000> <FFFFFFFF> <0041> endbfrange"); + assert_eq!(cmap.ranges.len(), 1); + assert_eq!(cmap.lookup(1).as_deref(), Some("B")); + } + + #[test] + fn malformed_cmap_never_panics() { + for s in [ + &b"beginbfchar <"[..], + b"beginbfrange <00> <01> [", + b"<<>>[[]]%", + b"begincodespacerange <> <>", + ] { + let _ = CMap::parse(s); + } + } + + #[test] + fn win_ansi_high_range() { + let t = base_table(BaseEncoding::WinAnsi); + assert_eq!(t[0x93], Some('“')); + assert_eq!(t[0xE9], Some('é')); + assert_eq!(t[0x81], None); + } +} diff --git a/crates/markkit/src/pdf/guard.rs b/crates/markkit/src/pdf/guard.rs new file mode 100644 index 0000000..012d83a --- /dev/null +++ b/crates/markkit/src/pdf/guard.rs @@ -0,0 +1,575 @@ +//! Bounded PDF stream decoding (threat model T8a, decompression bombs). +//! +//! Decision: hayro-syntax decodes object streams and xref streams itself while +//! resolving objects, with no output limit, so a bomb disguised as an object +//! stream could exhaust memory before markkit sees a single page. Before the +//! document is handed to hayro, [`prescan`] finds every `stream` keyword in the +//! raw bytes (streams cannot live inside object streams, so all are visible at +//! top level), reads the filter chain from the stream dictionary, and decodes +//! it with our own bounded decoders, discarding the output. Any stream whose +//! decoded size exceeds `max_pdf_stream_size`, or a total above +//! `max_pdf_decoded_size`, rejects the document. A chain the scan cannot see +//! (indirect `/Filter`, several `/Filter` keys) or an encrypted document (the +//! scan would only see ciphertext) is refused instead of trusted. +//! +//! Content streams, form XObjects and ToUnicode CMaps are decoded through +//! [`decode`] (same bounded decoders), never hayro's unbounded `decoded()`. +//! Image filters (DCT, JPX, JBIG2, CCITT) end a chain: markkit never decodes +//! images, and hayro is built without its `images` feature. + +use hayro_syntax::Filter; + +/// Raised when a stream would decode past a limit. +#[derive(Debug)] +pub(crate) struct Limit(pub String); + +/// Filters the scan understands. `Image` stops a chain. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Kind { + AsciiHex, + Ascii85, + Lzw, + Flate, + RunLength, + Image, +} + +impl Kind { + pub(crate) fn from_filter(f: Filter) -> Self { + match f { + Filter::AsciiHexDecode => Kind::AsciiHex, + Filter::Ascii85Decode => Kind::Ascii85, + Filter::LzwDecode => Kind::Lzw, + Filter::FlateDecode => Kind::Flate, + Filter::RunLengthDecode => Kind::RunLength, + _ => Kind::Image, + } + } + + fn from_name(name: &[u8]) -> Option<Self> { + Some(match name { + b"ASCIIHexDecode" | b"AHx" => Kind::AsciiHex, + b"ASCII85Decode" | b"A85" => Kind::Ascii85, + b"LZWDecode" | b"LZW" => Kind::Lzw, + b"FlateDecode" | b"Fl" => Kind::Flate, + b"RunLengthDecode" | b"RL" => Kind::RunLength, + b"DCTDecode" | b"DCT" | b"JPXDecode" | b"JBIG2Decode" | b"CCITTFaxDecode" | b"CCF" + | b"Crypt" => Kind::Image, + _ => return None, + }) + } +} + +/// Decode `data` through `chain`, failing once any stage would produce more +/// than `cap` bytes. Corrupt data yields whatever decoded before the error, like +/// hayro's own lenient decoders. +pub(crate) fn decode(chain: &[Kind], data: &[u8], cap: usize) -> Result<Vec<u8>, Limit> { + let mut cur = data.to_vec(); + for kind in chain { + cur = match kind { + Kind::AsciiHex => ascii_hex(&cur), + Kind::Ascii85 => ascii85(&cur), + Kind::Flate => flate(&cur, cap)?, + Kind::Lzw => lzw(&cur, cap)?, + Kind::RunLength => run_length(&cur, cap)?, + Kind::Image => break, + }; + } + if cur.len() > cap { + return Err(over(cap)); + } + Ok(cur) +} + +fn over(cap: usize) -> Limit { + Limit(format!("PDF stream decodes to more than {cap} bytes")) +} + +/// Verify every stream in the file decodes within the limits. +pub(crate) fn prescan( + bytes: &[u8], + stream_cap: usize, + total_cap: usize, +) -> Result<(), PrescanError> { + if find_key(bytes, b"/Encrypt").is_some() { + return Err(PrescanError::Encrypted); + } + let mut total = 0usize; + let mut from = 0; + while let Some(rel) = find(&bytes[from..], b"stream") { + let kw = from + rel; + from = kw + 6; + // `endstream`, or a word that merely contains "stream". + if kw > 0 && is_regular(bytes[kw - 1]) { + continue; + } + let data_start = eol_after(bytes, from); + let dict = dict_before(bytes, kw); + let chain = match filter_chain(dict) { + Ok(c) => c, + Err(()) => return Err(PrescanError::Unverifiable), + }; + if chain.iter().all(|k| matches!(k, Kind::Image)) { + continue; + } + let end = find(&bytes[data_start..], b"endstream") + .map(|e| data_start + e) + .unwrap_or(bytes.len()); + let budget = stream_cap.min(total_cap.saturating_sub(total)); + let out = decode(&chain, &bytes[data_start..end], budget).map_err(|_| { + if budget < stream_cap { + PrescanError::Limit(format!( + "PDF streams decode to more than {total_cap} bytes in total" + )) + } else { + PrescanError::Limit(over(stream_cap).0) + } + })?; + total += out.len(); + } + Ok(()) +} + +#[derive(Debug)] +pub(crate) enum PrescanError { + Encrypted, + Unverifiable, + Limit(String), +} + +fn find(hay: &[u8], needle: &[u8]) -> Option<usize> { + hay.windows(needle.len()).position(|w| w == needle) +} + +/// PDF regular character (not whitespace, not a delimiter). +fn is_regular(b: u8) -> bool { + !matches!( + b, + b'\0' + | b'\t' + | b'\n' + | b'\x0C' + | b'\r' + | b' ' + | b'(' + | b')' + | b'<' + | b'>' + | b'[' + | b']' + | b'{' + | b'}' + | b'/' + | b'%' + ) +} + +/// `/Key` followed by a delimiter (so `/Encrypt` does not match `/EncryptMetadata`). +fn find_key(hay: &[u8], key: &[u8]) -> Option<usize> { + let mut from = 0; + while let Some(rel) = find(&hay[from..], key) { + let at = from + rel; + let next = hay.get(at + key.len()).copied().unwrap_or(b' '); + if !is_regular(next) { + return Some(at); + } + from = at + key.len(); + } + None +} + +/// Start of stream data after `stream`: CRLF, LF or CR; without an EOL the data +/// is assumed to start right away (malformed files must not evade the scan). +fn eol_after(bytes: &[u8], at: usize) -> usize { + let mut i = at; + while bytes.get(i) == Some(&b' ') { + i += 1; + } + match bytes.get(i) { + Some(b'\r') if bytes.get(i + 1) == Some(&b'\n') => i + 2, + Some(b'\r' | b'\n') => i + 1, + _ => i, + } +} + +/// Text between the nearest preceding `obj` keyword and `stream` (bounded window). +fn dict_before(bytes: &[u8], kw: usize) -> &[u8] { + const WINDOW: usize = 64 * 1024; + let lo = kw.saturating_sub(WINDOW); + let region = &bytes[lo..kw]; + let start = region + .windows(3) + .rposition(|w| w == b"obj") + .map(|p| p + 3) + .unwrap_or(0); + ®ion[start..] +} + +/// Parse `/Filter` from raw dictionary text. `Err` when it cannot be verified. +fn filter_chain(dict: &[u8]) -> Result<Vec<Kind>, ()> { + let Some(at) = find_key(dict, b"/Filter") else { + return Ok(Vec::new()); + }; + if find_key(&dict[at + 7..], b"/Filter").is_some() { + return Err(()); + } + let mut rest = skip_ws(&dict[at + 7..]); + let array = rest.first() == Some(&b'['); + if array { + rest = &rest[1..]; + } + let mut chain = Vec::new(); + loop { + rest = skip_ws(rest); + match rest.first() { + Some(b'/') => { + let len = rest[1..] + .iter() + .position(|&b| !is_regular(b)) + .unwrap_or(rest.len() - 1); + let name = unescape_name(&rest[1..1 + len]); + // Unknown names make hayro fail the stream, so they cannot expand. + chain.push(Kind::from_name(&name).unwrap_or(Kind::Image)); + rest = &rest[1 + len..]; + if !array { + return Ok(chain); + } + } + Some(b']') if array => return Ok(chain), + // Indirect reference, garbage, or truncated: cannot verify. + _ => return Err(()), + } + } +} + +fn skip_ws(mut s: &[u8]) -> &[u8] { + loop { + match s.first() { + Some(b) if !is_regular(*b) && !matches!(b, b'/' | b'[' | b']' | b'<' | b'(' | b'%') => { + s = &s[1..] + } + Some(b'%') => { + let n = s + .iter() + .position(|&b| b == b'\n' || b == b'\r') + .unwrap_or(s.len()); + s = &s[n..]; + } + _ => return s, + } + } +} + +/// Resolve `#xx` escapes in a name. +fn unescape_name(raw: &[u8]) -> Vec<u8> { + let mut out = Vec::with_capacity(raw.len()); + let mut i = 0; + while i < raw.len() { + if raw[i] == b'#' + && let (Some(h), Some(l)) = (raw.get(i + 1).and_then(hex), raw.get(i + 2).and_then(hex)) + { + out.push(h << 4 | l); + i += 3; + continue; + } + out.push(raw[i]); + i += 1; + } + out +} + +fn hex(b: &u8) -> Option<u8> { + (*b as char).to_digit(16).map(|d| d as u8) +} + +// ─── Decoders ─────────────────────────────────────────────────────── + +fn flate(data: &[u8], cap: usize) -> Result<Vec<u8>, Limit> { + use miniz_oxide::inflate::{self, TINFLStatus}; + // Limit + 1 so "exactly cap" is distinguishable from "more than cap". + let lim = cap.saturating_add(1); + let res = inflate::decompress_to_vec_zlib_with_limit(data, lim) + .or_else(|_| inflate::decompress_to_vec_with_limit(data, lim)); + let out = match res { + Ok(v) => v, + Err(e) if e.status == TINFLStatus::HasMoreOutput => return Err(over(cap)), + // Corrupt: keep the zlib attempt's partial output (raw attempt is + // usually junk when the zlib header was valid). + Err(_) => match inflate::decompress_to_vec_zlib_with_limit(data, lim) { + Err(e) if e.status == TINFLStatus::HasMoreOutput => return Err(over(cap)), + Err(e) => e.output, + Ok(v) => v, + }, + }; + if out.len() > cap { + return Err(over(cap)); + } + Ok(out) +} + +fn run_length(data: &[u8], cap: usize) -> Result<Vec<u8>, Limit> { + let mut out = Vec::new(); + let mut i = 0; + while i < data.len() { + let n = data[i] as usize; + i += 1; + match n { + 128 => break, + 0..=127 => { + let end = (i + n + 1).min(data.len()); + out.extend_from_slice(&data[i..end]); + i = end; + } + _ => { + let Some(&b) = data.get(i) else { break }; + out.resize(out.len() + 257 - n, b); + i += 1; + } + } + if out.len() > cap { + return Err(over(cap)); + } + } + Ok(out) +} + +/// LZW with EarlyChange = 1 (the PDF default). +fn lzw(data: &[u8], cap: usize) -> Result<Vec<u8>, Limit> { + const CLEAR: usize = 256; + const EOD: usize = 257; + let mut out = Vec::new(); + // Each table entry is (prefix index, last byte, length); strings are rebuilt + // on demand so the table stays O(4096). + let mut table: Vec<(usize, u8, usize)> = (0..256).map(|b| (usize::MAX, b as u8, 1)).collect(); + table.push((0, 0, 0)); + table.push((0, 0, 0)); + let mut width = 9; + let mut prev: Option<usize> = None; + let (mut buf, mut bits, mut pos) = (0u32, 0u32, 0usize); + let mut scratch = Vec::new(); + loop { + while bits < width && pos < data.len() { + buf = (buf << 8) | data[pos] as u32; + bits += 8; + pos += 1; + } + if bits < width { + break; + } + let code = ((buf >> (bits - width)) & ((1 << width) - 1)) as usize; + bits -= width; + if code == CLEAR { + table.truncate(258); + width = 9; + prev = None; + continue; + } + if code == EOD { + break; + } + let first_of = |table: &Vec<(usize, u8, usize)>, mut c: usize| { + while table[c].0 != usize::MAX { + c = table[c].0; + } + table[c].1 + }; + let entry = if code < table.len() && code != 256 && code != 257 { + if let Some(p) = prev + && table.len() < 4096 + { + let first = first_of(&table, code); + table.push((p, first, table[p].2 + 1)); + } + code + } else if code == table.len() { + let Some(p) = prev else { break }; + let first = first_of(&table, p); + table.push((p, first, table[p].2 + 1)); + code + } else { + break; + }; + // Emit the string for `entry`. + scratch.clear(); + let mut c = entry; + loop { + scratch.push(table[c].1); + if table[c].0 == usize::MAX { + break; + } + c = table[c].0; + } + out.extend(scratch.iter().rev()); + if out.len() > cap { + return Err(over(cap)); + } + prev = Some(entry); + if table.len() + 1 >= (1 << width) && width < 12 { + width += 1; + } + } + Ok(out) +} + +fn ascii_hex(data: &[u8]) -> Vec<u8> { + let mut out = Vec::with_capacity(data.len() / 2); + let mut hi: Option<u8> = None; + for b in data { + if *b == b'>' { + break; + } + let Some(v) = hex(b) else { continue }; + match hi.take() { + Some(h) => out.push(h << 4 | v), + None => hi = Some(v), + } + } + if let Some(h) = hi { + out.push(h << 4); + } + out +} + +fn ascii85(data: &[u8]) -> Vec<u8> { + let mut out = Vec::with_capacity(data.len() * 4 / 5); + let mut group = [0u8; 5]; + let mut n = 0; + let body = data.strip_prefix(b"<~").unwrap_or(data); + for &b in body { + match b { + b'~' => break, + b'z' if n == 0 => out.extend_from_slice(&[0; 4]), + b'!'..=b'u' => { + group[n] = b - b'!'; + n += 1; + if n == 5 { + let v = group.iter().fold(0u64, |a, &d| a * 85 + d as u64); + out.extend_from_slice(&(v as u32).to_be_bytes()); + n = 0; + } + } + _ => {} + } + } + if n > 1 { + for g in group.iter_mut().skip(n) { + *g = 84; + } + let v = group.iter().fold(0u64, |a, &d| a * 85 + d as u64); + out.extend_from_slice(&(v as u32).to_be_bytes()[..n - 1]); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn zlib(data: &[u8]) -> Vec<u8> { + miniz_oxide::deflate::compress_to_vec_zlib(data, 9) + } + + #[test] + fn flate_round_trip_and_cap() { + let z = zlib(&[b'a'; 10_000]); + assert_eq!(decode(&[Kind::Flate], &z, 10_000).unwrap().len(), 10_000); + assert!(decode(&[Kind::Flate], &z, 9_999).is_err()); + } + + #[test] + fn corrupt_flate_keeps_partial_output() { + let mut z = zlib(b"hello world, hello world, hello world"); + let n = z.len(); + z.truncate(n - 6); + assert!(decode(&[Kind::Flate], &z, 1000).is_ok()); + } + + #[test] + fn run_length_bounded() { + let bomb: Vec<u8> = std::iter::repeat_n([129u8, b'x'], 1000).flatten().collect(); + assert!(run_length(&bomb, 1000).is_err()); + assert_eq!( + run_length(&[2, b'a', b'b', b'c', 254, b'z', 128], 100).unwrap(), + b"abczzz" + ); + } + + #[test] + fn lzw_spec_example() { + // PDF 32000-1 §7.4.4.2 example: "-----A---B". + let enc = [0x80, 0x0B, 0x60, 0x50, 0x22, 0x0C, 0x0C, 0x85, 0x01]; + assert_eq!(lzw(&enc, 100).unwrap(), b"-----A---B"); + assert!(lzw(&enc, 5).is_err()); + } + + #[test] + fn ascii_filters() { + assert_eq!(ascii_hex(b"48 65 6C6C6F>"), b"Hello"); + assert_eq!(ascii_hex(b"7>"), [0x70]); + assert_eq!(ascii85(b"<~87cURDZ~>"), b"Hello"); + } + + #[test] + fn filter_chain_parsing() { + assert_eq!(filter_chain(b"<< /Length 5 >>"), Ok(vec![])); + assert_eq!( + filter_chain(b"<</Filter/FlateDecode>>"), + Ok(vec![Kind::Flate]) + ); + assert_eq!( + filter_chain(b"<< /Filter [ /A85 /Fl#61teDecode ] >>"), + Ok(vec![Kind::Ascii85, Kind::Flate]) + ); + assert_eq!( + filter_chain(b"<< /Filter /DCTDecode >>"), + Ok(vec![Kind::Image]) + ); + assert_eq!(filter_chain(b"<< /Filter 5 0 R >>"), Err(())); + assert_eq!( + filter_chain(b"<< /D << /Filter /AHx >> /Filter /Fl >>"), + Err(()) + ); + } + + #[test] + fn prescan_rejects_bomb_and_accepts_small() { + let mk = |payload: &[u8]| { + let z = zlib(payload); + let mut pdf = b"%PDF-1.7\n1 0 obj\n<< /Length ".to_vec(); + pdf.extend(format!("{} /Filter /FlateDecode >>\nstream\n", z.len()).bytes()); + pdf.extend(&z); + pdf.extend(b"\nendstream\nendobj\n"); + pdf + }; + assert!(prescan(&mk(&[0; 100]), 1000, 1000).is_ok()); + assert!(matches!( + prescan(&mk(&[0; 2000]), 1000, 10_000), + Err(PrescanError::Limit(_)) + )); + let mut two = mk(&[0; 600]); + two.extend(mk(&[0; 600])); + assert!(matches!( + prescan(&two, 1000, 1000), + Err(PrescanError::Limit(_)) + )); + } + + #[test] + fn prescan_flags_encryption_and_indirect_filters() { + assert!(matches!( + prescan(b"trailer << /Encrypt 5 0 R >>", 10, 10), + Err(PrescanError::Encrypted) + )); + assert!(prescan(b"<< /EncryptMetadata false >>", 10, 10).is_ok()); + assert!(matches!( + prescan( + b"1 0 obj << /Filter 2 0 R >>\nstream\nxx\nendstream", + 10, + 10 + ), + Err(PrescanError::Unverifiable) + )); + // Words containing "stream" are not stream keywords. + assert!(prescan(b"(upstream)\n", 10, 10).is_ok()); + } +} diff --git a/crates/markkit/src/pdf/mod.rs b/crates/markkit/src/pdf/mod.rs new file mode 100644 index 0000000..26c94e3 --- /dev/null +++ b/crates/markkit/src/pdf/mod.rs @@ -0,0 +1,845 @@ +//! PDF text layer → IR. +//! +//! Decision: parse with `hayro-syntax` (pure Rust, read-only, lazy object +//! resolution, xref repair, only `rustc-hash` + `smallvec` with +//! `default-features = false`). It has no decode limits, so every stream is +//! checked by [`guard::prescan`] before hayro sees the file, and all streams +//! markkit interprets go through [`guard::decode`] (see guard.rs). +//! +//! Text layer only: no OCR, no image extraction. Best-effort layout (see +//! knowledge/formats/format-support.md): +//! - Text-showing operators are interpreted with the text/graphics state +//! (`Tm`, `CTM`, `Tc`, `Tw`, `Tz`, `TL`, `Ts`, `TJ` kerning) to place each +//! run in device space. Form XObjects are followed (depth 8, cycle-safe). +//! - Runs join into lines in content-stream order (same baseline, moving right); +//! gaps wider than ~0.15 em become spaces. +//! - Lines join into paragraphs; a vertical gap above ~1.3× the page's typical +//! line pitch, a move upwards (new column), or a font-size change starts a new +//! one. `letter-` + lowercase continuation is de-hyphenated. +//! - A line starting with a bullet glyph (`•`, `▪`, `–`, …) plus a space starts a +//! list item; consecutive items form one unordered list. +//! - Headings: short paragraphs at ≥ 1.2× the dominant body size; the largest +//! sizes map to `#`, `##`, … (max 6). +//! +//! Limits: `max_pdf_pages` (truncate + `truncated_pdf` warning), +//! `max_pdf_stream_size` / `max_pdf_decoded_size` for decoding, and the same +//! total as an interpretation budget, so a form XObject reused on every page +//! cannot multiply CPU time without bound. + +mod font; +pub(crate) mod guard; + +use std::collections::HashMap; + +use hayro_syntax::Pdf; +use hayro_syntax::content::UntypedIter; +use hayro_syntax::object::{Array, Dict, Name, Object, ObjectIdentifier, Stream}; +use hayro_syntax::page::Resources; + +use crate::config::MarkkitBuilder; +use crate::error::Error; +use crate::ir::{Document, ListItem, Node, Source}; +use crate::limits::Budget; + +use font::{Font, Glyph}; +use guard::PrescanError; + +const MAX_FORM_DEPTH: usize = 8; + +/// Result of parsing a PDF. +pub(crate) struct PdfOutput { + pub doc: Document, + pub title: Option<String>, + pub pages: usize, + pub warnings: Vec<String>, +} + +pub(crate) fn parse( + bytes: &[u8], + cfg: &MarkkitBuilder, + clock: &Budget, +) -> Result<PdfOutput, Error> { + guard::prescan( + bytes, + cfg.limits.max_pdf_stream_size, + cfg.limits.max_pdf_decoded_size, + ) + .map_err(|e| match e { + PrescanError::Encrypted => { + Error::UnsupportedFormat("encrypted PDFs are not supported".into()) + } + PrescanError::Unverifiable => Error::PdfExtract( + "stream filter declared indirectly or ambiguously; refusing to decode".into(), + ), + PrescanError::Limit(m) => Error::ResourceLimit(m), + })?; + + clock.check_time()?; + + let pdf = Pdf::new(bytes.to_vec()).map_err(|e| match e { + hayro_syntax::LoadPdfError::Decryption(_) => { + Error::UnsupportedFormat("encrypted PDFs are not supported".into()) + } + _ => Error::PdfExtract("malformed PDF: no readable page tree".into()), + })?; + + let title = pdf + .metadata() + .title + .as_deref() + .map(pdf_text_string) + .map(|t| t.trim().to_string()) + .filter(|t| !t.is_empty()); + + let pages = pdf.pages(); + let mut warnings = Vec::new(); + let mut interp = Interpreter::new( + cfg.limits.max_pdf_stream_size, + cfg.limits.max_pdf_decoded_size, + clock, + ); + let mut page_lines = Vec::new(); + for page in pages.iter().take(cfg.limits.max_pdf_pages) { + clock.check_time()?; + interp.spans.clear(); + let res = page.resources().clone(); + if let Some(contents) = page.raw().get::<Object<'_>>(b"Contents") { + let streams: Vec<Stream<'_>> = match contents { + Object::Stream(s) => vec![s], + Object::Array(a) => a.iter::<Stream<'_>>().collect(), + _ => Vec::new(), + }; + // Contents arrays are one logical stream split anywhere. + let mut data = Vec::new(); + for s in &streams { + data.extend(interp.decode(s)?); + data.push(b'\n'); + } + interp.run(&data, &res, Matrix::IDENTITY, 0)?; + } + page_lines.push(build_lines(std::mem::take(&mut interp.spans))); + } + if pages.len() > cfg.limits.max_pdf_pages { + warnings.push(format!( + "truncated_pdf: rendered {} of {} pages (max_pdf_pages)", + cfg.limits.max_pdf_pages, + pages.len() + )); + } + let mut unmapped: Vec<&String> = interp.unmapped.iter().collect(); + unmapped.sort(); + for name in unmapped { + warnings.push(format!( + "pdf_unmapped_font: text in font {name} has no Unicode mapping and was skipped" + )); + } + + let nodes = build_nodes(page_lines, cfg.track_sources); + if nodes.is_empty() && !pages.is_empty() { + warnings + .push("pdf_no_text: no text layer found (scanned PDF? OCR is not supported)".into()); + } + Ok(PdfOutput { + doc: Document::new(nodes), + title, + pages: pages.len().min(cfg.limits.max_pdf_pages), + warnings, + }) +} + +/// PDF text string: UTF-16BE with BOM, UTF-8 with BOM, else PDFDocEncoding +/// (approximated as Latin-1). +fn pdf_text_string(b: &[u8]) -> String { + if let Some(rest) = b.strip_prefix(&[0xFE, 0xFF]) { + let units: Vec<u16> = rest + .chunks(2) + .map(|c| u16::from_be_bytes([c[0], c.get(1).copied().unwrap_or(0)])) + .collect(); + return char::decode_utf16(units) + .map(|r| r.unwrap_or('\u{FFFD}')) + .collect(); + } + if let Some(rest) = b.strip_prefix(&[0xEF, 0xBB, 0xBF]) { + return String::from_utf8_lossy(rest).into_owned(); + } + b.iter().map(|&c| c as char).collect() +} + +// ─── Geometry ─────────────────────────────────────────────────────── + +#[derive(Debug, Clone, Copy, PartialEq)] +struct Matrix([f32; 6]); + +impl Matrix { + const IDENTITY: Matrix = Matrix([1.0, 0.0, 0.0, 1.0, 0.0, 0.0]); + + /// `self × other` (apply `self` first, then `other`). + fn then(&self, o: &Matrix) -> Matrix { + let [a, b, c, d, e, f] = self.0; + let [oa, ob, oc, od, oe, of] = o.0; + Matrix([ + a * oa + b * oc, + a * ob + b * od, + c * oa + d * oc, + c * ob + d * od, + e * oa + f * oc + oe, + e * ob + f * od + of, + ]) + } + + fn translate(tx: f32, ty: f32) -> Matrix { + Matrix([1.0, 0.0, 0.0, 1.0, tx, ty]) + } + + fn apply(&self, x: f32, y: f32) -> (f32, f32) { + let [a, b, c, d, e, f] = self.0; + (a * x + c * y + e, b * x + d * y + f) + } + + fn is_finite(&self) -> bool { + self.0.iter().all(|v| v.is_finite()) + } +} + +// ─── Interpreter ──────────────────────────────────────────────────── + +#[derive(Clone)] +struct State { + ctm: Matrix, + font: Option<usize>, + size: f32, + char_spacing: f32, + word_spacing: f32, + scale: f32, + leading: f32, + rise: f32, +} + +impl State { + fn new(ctm: Matrix) -> Self { + State { + ctm, + font: None, + size: 0.0, + char_spacing: 0.0, + word_spacing: 0.0, + scale: 1.0, + leading: 0.0, + rise: 0.0, + } + } +} + +/// A run of text placed in device space. +#[derive(Debug, Clone)] +struct Span { + x0: f32, + x1: f32, + y: f32, + size: f32, + text: String, +} + +struct Interpreter<'c> { + fonts: Vec<Font>, + font_ids: HashMap<ObjectIdentifier, usize>, + unmapped: std::collections::BTreeSet<String>, + spans: Vec<Span>, + stream_cap: usize, + budget: usize, + form_stack: Vec<ObjectIdentifier>, + clock: &'c Budget, +} + +fn limit() -> Error { + Error::ResourceLimit("PDF content exceeds max_pdf_decoded_size".into()) +} + +fn num(o: Option<&Object<'_>>) -> f32 { + o.and_then(|o| o.clone().into_f32()) + .filter(|v| v.is_finite()) + .unwrap_or(0.0) +} + +impl<'c> Interpreter<'c> { + fn new(stream_cap: usize, budget: usize, clock: &'c Budget) -> Self { + Interpreter { + fonts: Vec::new(), + font_ids: HashMap::new(), + unmapped: Default::default(), + spans: Vec::new(), + stream_cap, + budget, + form_stack: Vec::new(), + clock, + } + } + + /// Bounded decode, charged against the interpretation budget. + fn decode(&mut self, s: &Stream<'_>) -> Result<Vec<u8>, Error> { + let cap = self.stream_cap.min(self.budget); + let chain: Vec<guard::Kind> = s + .filters() + .into_iter() + .map(guard::Kind::from_filter) + .collect(); + if chain.contains(&guard::Kind::Image) { + return Ok(Vec::new()); + } + let data = guard::decode(&chain, &s.raw_data(), cap).map_err(|_| limit())?; + self.charge(data.len())?; + Ok(data) + } + + fn charge(&mut self, n: usize) -> Result<(), Error> { + self.budget = self.budget.checked_sub(n).ok_or_else(limit)?; + Ok(()) + } + + fn font(&mut self, res: &Resources<'_>, name: &Name<'_>) -> Option<usize> { + let dict = res.get_font(name)?; + let id = dict.obj_id(); + if let Some(id) = id + && let Some(&i) = self.font_ids.get(&id) + { + return Some(i); + } + let f = Font::load(&dict, self.stream_cap.min(self.budget)); + self.fonts.push(f); + let i = self.fonts.len() - 1; + if let Some(id) = id { + self.font_ids.insert(id, i); + } + Some(i) + } + + fn run( + &mut self, + data: &[u8], + res: &Resources<'_>, + ctm: Matrix, + depth: usize, + ) -> Result<(), Error> { + let mut st = State::new(ctm); + let mut stack: Vec<State> = Vec::new(); + let mut tm = Matrix::IDENTITY; + let mut tlm = Matrix::IDENTITY; + let mut glyphs = Vec::new(); + let mut ops = UntypedIter::new(data); + let mut count = 0usize; + while let Some(ins) = ops.next() { + // Each operator costs at least one budget unit. + count += 1; + if count & 0xFFF == 0 { + self.charge(0x1000)?; + } + if self.clock.expired() { + return Err(self.clock.timeout_error()); + } + let args: Vec<&Object<'_>> = ins.operands().collect(); + let a = |i: usize| num(args.get(i).copied()); + let op: &[u8] = ins.operator; + match op { + b"q" => { + if stack.len() < 256 { + stack.push(st.clone()); + } + } + b"Q" => { + if let Some(s) = stack.pop() { + st = s; + } + } + b"cm" if args.len() == 6 => { + let m = Matrix([a(0), a(1), a(2), a(3), a(4), a(5)]); + let next = m.then(&st.ctm); + if next.is_finite() { + st.ctm = next; + } + } + b"BT" => { + tm = Matrix::IDENTITY; + tlm = Matrix::IDENTITY; + } + b"Tf" => { + st.font = match args.first() { + Some(Object::Name(n)) => self.font(res, n), + _ => None, + }; + st.size = a(1); + } + b"Tc" => st.char_spacing = a(0), + b"Tw" => st.word_spacing = a(0), + b"Tz" => st.scale = a(0) / 100.0, + b"TL" => st.leading = a(0), + b"Ts" => st.rise = a(0), + b"Td" => { + tlm = Matrix::translate(a(0), a(1)).then(&tlm); + tm = tlm; + } + b"TD" => { + st.leading = -a(1); + tlm = Matrix::translate(a(0), a(1)).then(&tlm); + tm = tlm; + } + b"Tm" if args.len() == 6 => { + let m = Matrix([a(0), a(1), a(2), a(3), a(4), a(5)]); + if m.is_finite() { + tlm = m; + tm = m; + } + } + b"T*" => { + tlm = Matrix::translate(0.0, -st.leading).then(&tlm); + tm = tlm; + } + b"Tj" => { + if let Some(Object::String(s)) = args.first() { + self.show(&st, &mut tm, s.as_bytes(), &mut glyphs); + } + } + b"'" | b"\"" => { + if op == b"\"" { + st.word_spacing = a(0); + st.char_spacing = a(1); + } + tlm = Matrix::translate(0.0, -st.leading).then(&tlm); + tm = tlm; + if let Some(Object::String(s)) = args.last() { + self.show(&st, &mut tm, s.as_bytes(), &mut glyphs); + } + } + b"TJ" => { + if let Some(Object::Array(arr)) = args.first() { + self.show_array(&st, &mut tm, arr, &mut glyphs); + } + } + b"Do" if depth < MAX_FORM_DEPTH => { + if let Some(Object::Name(n)) = args.first() { + self.form(res, n, &st, depth)?; + } + } + _ => {} + } + } + Ok(()) + } + + fn form( + &mut self, + res: &Resources<'_>, + name: &Name<'_>, + st: &State, + depth: usize, + ) -> Result<(), Error> { + let Some(stream) = res.get_x_object(name) else { + return Ok(()); + }; + let dict = stream.dict(); + if font::name_of(dict.get::<Name<'_>>(b"Subtype")) != "Form" { + return Ok(()); + } + let id = stream.obj_id(); + if self.form_stack.contains(&id) { + return Ok(()); + } + let m = dict + .get::<Array<'_>>(b"Matrix") + .map(|a| { + let v: Vec<f32> = a.iter::<Object<'_>>().map(|o| num(Some(&o))).collect(); + if v.len() == 6 { + Matrix([v[0], v[1], v[2], v[3], v[4], v[5]]) + } else { + Matrix::IDENTITY + } + }) + .unwrap_or(Matrix::IDENTITY); + let ctm = m.then(&st.ctm); + if !ctm.is_finite() { + return Ok(()); + } + let form_res = dict + .get::<Dict<'_>>(b"Resources") + .map(Resources::new) + .unwrap_or_else(|| res.clone()); + let data = self.decode(&stream)?; + self.form_stack.push(id); + let r = self.run(&data, &form_res, ctm, depth + 1); + self.form_stack.pop(); + r + } + + fn show_array( + &mut self, + st: &State, + tm: &mut Matrix, + arr: &Array<'_>, + glyphs: &mut Vec<Glyph>, + ) { + for item in arr.iter::<Object<'_>>() { + match item { + Object::String(s) => self.show(st, tm, s.as_bytes(), glyphs), + Object::Number(n) => { + let adj = n.as_f32(); + if !adj.is_finite() { + continue; + } + let tx = -adj / 1000.0 * st.size * st.scale; + *tm = Matrix::translate(tx, 0.0).then(tm); + // A large negative adjustment is a word gap. + if -adj > 200.0 + && let Some(last) = self.spans.last_mut() + && !last.text.ends_with(' ') + { + last.text.push(' '); + } + } + _ => {} + } + } + } + + fn show(&mut self, st: &State, tm: &mut Matrix, bytes: &[u8], glyphs: &mut Vec<Glyph>) { + glyphs.clear(); + match st.font { + Some(i) => { + let font = &self.fonts[i]; + font.decode(bytes, glyphs); + if font.unmapped { + let name = match font.name.as_str() { + "" => "(unnamed)".to_string(), + n => n.to_string(), + }; + self.unmapped.insert(name); + } + } + // No (resolvable) font selected: best effort as WinAnsi. + None => Font::fallback().decode(bytes, glyphs), + } + self.place(st, tm, glyphs); + } + + fn place(&mut self, st: &State, tm: &mut Matrix, glyphs: &[Glyph]) { + let trm = |tm: &Matrix| { + Matrix([st.size * st.scale, 0.0, 0.0, st.size, 0.0, st.rise]) + .then(tm) + .then(&st.ctm) + }; + let start = trm(tm); + let (x0, y) = start.apply(0.0, 0.0); + let size = (start.0[2].hypot(start.0[3])).abs(); + let mut text = String::new(); + for g in glyphs { + if let Some(t) = &g.text { + push_clean(&mut text, t); + } + let mut tx = g.width * st.size + st.char_spacing; + if g.is_space { + tx += st.word_spacing; + } + *tm = Matrix::translate(tx * st.scale, 0.0).then(tm); + } + let (x1, _) = trm(tm).apply(0.0, 0.0); + if text.is_empty() || !x0.is_finite() || !y.is_finite() || !size.is_finite() { + return; + } + self.spans.push(Span { + x0, + x1: x1.max(x0), + y, + size: size.max(0.1), + text, + }); + } +} + +/// Append glyph text: control chars dropped (tab → space), ligature +/// presentation forms expanded. +fn push_clean(out: &mut String, t: &str) { + for c in t.chars() { + match c { + '\t' => out.push(' '), + 'ff' => out.push_str("ff"), + 'fi' => out.push_str("fi"), + 'fl' => out.push_str("fl"), + 'ffi' => out.push_str("ffi"), + 'ffl' => out.push_str("ffl"), + 'ſt' | 'st' => out.push_str("st"), + c if c.is_control() => {} + c => out.push(c), + } + } +} + +// ─── Layout ───────────────────────────────────────────────────────── + +#[derive(Debug, Clone)] +struct Line { + y: f32, + size: f32, + x1: f32, + text: String, +} + +fn build_lines(spans: Vec<Span>) -> Vec<Line> { + let mut lines: Vec<Line> = Vec::new(); + for s in spans { + if let Some(line) = lines.last_mut() { + let tol = 0.5 * s.size.max(line.size); + let same_baseline = (s.y - line.y).abs() <= tol; + let gap = s.x0 - line.x1; + if same_baseline && gap > -0.5 * s.size { + if gap > 0.15 * s.size && !line.text.ends_with(' ') && !s.text.starts_with(' ') { + line.text.push(' '); + } + line.text.push_str(&s.text); + line.x1 = line.x1.max(s.x1); + // Superscripts / subscripts keep the line's baseline and size. + if s.size > line.size { + line.size = s.size; + } + continue; + } + } + lines.push(Line { + y: s.y, + size: s.size, + x1: s.x1, + text: s.text, + }); + } + for l in &mut lines { + l.text = l.text.split_whitespace().collect::<Vec<_>>().join(" "); + } + lines.retain(|l| !l.text.is_empty()); + lines +} + +struct Para { + /// 1-based page number. + page: u32, + size: f32, + lines: usize, + text: String, + bullet: bool, +} + +const BULLETS: &[char] = &['•', '●', '▪', '◦', '‣', '∙', '■', '–']; + +/// `• text` → `Some("text")`. +fn strip_bullet(text: &str) -> Option<&str> { + let mut it = text.chars(); + let c = it.next()?; + let rest = it.as_str(); + (BULLETS.contains(&c) && rest.starts_with(' ')).then(|| rest.trim_start()) +} + +fn round_size(s: f32) -> i32 { + (s * 2.0).round() as i32 +} + +/// Lines per page → blocks; `track` records `Source::page` on each block. +fn build_nodes(pages: Vec<Vec<Line>>, track: bool) -> Vec<Node> { + let mut paras: Vec<Para> = Vec::new(); + for (page_idx, lines) in pages.into_iter().enumerate() { + let page = u32::try_from(page_idx + 1).unwrap_or(u32::MAX); + // Typical line pitch: most common downward step between lines (ties → + // smaller), so paragraph gaps stand out even when frequent. + let mut steps: HashMap<i32, usize> = HashMap::new(); + for w in lines.windows(2) { + let d = w[0].y - w[1].y; + if d > 0.0 { + *steps.entry(round_size(d)).or_default() += 1; + } + } + let pitch = steps + .iter() + .max_by_key(|(d, n)| (**n, -**d)) + .map(|(d, _)| *d as f32 / 2.0); + + let mut prev: Option<&Line> = None; + for line in &lines { + let new_para = match prev { + None => true, + Some(p) => { + let step = p.y - line.y; + let size_change = (line.size - p.size).abs() > 0.15 * p.size.max(line.size); + let limit = pitch.unwrap_or(1.2 * p.size).max(p.size * 0.8) * 1.3; + size_change || step < -0.5 * line.size || step > limit + } + }; + let bullet = strip_bullet(&line.text); + if new_para || bullet.is_some() { + paras.push(Para { + page, + size: line.size, + lines: 1, + text: bullet.unwrap_or(&line.text).to_string(), + bullet: bullet.is_some(), + }); + } else if let Some(para) = paras.last_mut() { + join_line(&mut para.text, &line.text); + para.lines += 1; + } + prev = Some(line); + } + } + + // Dominant body size by character count. + let mut by_size: HashMap<i32, usize> = HashMap::new(); + for p in ¶s { + *by_size.entry(round_size(p.size)).or_default() += p.text.chars().count(); + } + let body = by_size + .iter() + .max_by_key(|(s, n)| (**n, -**s)) + .map(|(s, _)| *s as f32 / 2.0) + .unwrap_or(0.0); + let is_heading = |p: &Para| { + body > 0.0 + && p.size >= body * 1.2 + && p.lines <= 3 + && p.text.chars().count() <= 200 + && p.text.chars().any(char::is_alphabetic) + }; + let mut heading_sizes: Vec<i32> = paras + .iter() + .filter(|p| is_heading(p)) + .map(|p| round_size(p.size)) + .collect(); + heading_sizes.sort_unstable_by(|a, b| b.cmp(a)); + heading_sizes.dedup(); + + let mut nodes = Vec::new(); + for p in ¶s { + let children = vec![Node::Text(p.text.clone())]; + let source = track.then(|| Box::new(Source::default().with_page(p.page))); + if p.bullet { + let item = ListItem::new(vec![Node::Paragraph { + children, + source: None, + }]); + if let Some(Node::List { items, .. }) = nodes.last_mut() { + items.push(item); + } else { + nodes.push(Node::List { + ordered: false, + start: None, + items: vec![item], + source, + }); + } + } else if is_heading(p) { + let rank = heading_sizes + .iter() + .position(|s| *s == round_size(p.size)) + .unwrap_or(0); + nodes.push(Node::Heading { + level: (rank + 1).min(6) as u8, + children, + source, + }); + } else { + nodes.push(Node::Paragraph { children, source }); + } + } + nodes +} + +/// Append a line to a paragraph, undoing end-of-line hyphenation. +fn join_line(para: &mut String, line: &str) { + let mut tail = para.chars().rev(); + let dehyphenate = tail.next() == Some('-') + && tail.next().is_some_and(char::is_alphabetic) + && line.chars().next().is_some_and(char::is_lowercase); + if dehyphenate || para.ends_with('\u{AD}') { + para.pop(); + } else { + para.push(' '); + } + para.push_str(line); +} + +#[cfg(test)] +mod tests { + use super::*; + + fn line(y: f32, size: f32, text: &str) -> Line { + Line { + y, + size, + x1: 0.0, + text: text.into(), + } + } + + #[test] + fn spans_join_into_lines_with_spaces() { + let spans = vec![ + Span { + x0: 0.0, + x1: 20.0, + y: 100.0, + size: 10.0, + text: "Hello".into(), + }, + Span { + x0: 23.0, + x1: 50.0, + y: 100.0, + size: 10.0, + text: "world".into(), + }, + Span { + x0: 50.0, + x1: 60.0, + y: 100.0, + size: 10.0, + text: "!".into(), + }, + Span { + x0: 0.0, + x1: 20.0, + y: 88.0, + size: 10.0, + text: "Next".into(), + }, + ]; + let lines = build_lines(spans); + assert_eq!(lines.len(), 2); + assert_eq!(lines[0].text, "Hello world!"); + } + + #[test] + fn paragraphs_headings_and_hyphenation() { + let page = vec![ + line(700.0, 20.0, "Title"), + line(660.0, 10.0, "First para-"), + line(648.0, 10.0, "graph goes on."), + line(636.0, 10.0, "Still first."), + line(610.0, 10.0, "Second paragraph."), + ]; + let nodes = build_nodes(vec![page], false); + assert_eq!(nodes.len(), 3); + assert!(matches!(&nodes[0], Node::Heading { level: 1, .. })); + assert_eq!( + nodes[1], + Node::Paragraph { + children: vec![Node::Text("First paragraph goes on. Still first.".into())], + source: None, + } + ); + } + + #[test] + fn text_strings() { + assert_eq!(pdf_text_string(b"\xFE\xFF\x00H\x00i"), "Hi"); + assert_eq!(pdf_text_string(b"Caf\xE9"), "Café"); + } + + #[test] + fn matrix_compose() { + let m = Matrix::translate(10.0, 5.0).then(&Matrix([2.0, 0.0, 0.0, 2.0, 0.0, 0.0])); + assert_eq!(m.apply(1.0, 1.0), (22.0, 12.0)); + } +} diff --git a/crates/markkit/src/render/markdown.rs b/crates/markkit/src/render/markdown.rs index b80d4d5..6241535 100644 --- a/crates/markkit/src/render/markdown.rs +++ b/crates/markkit/src/render/markdown.rs @@ -77,7 +77,13 @@ impl MarkdownRenderer { } /// Render with `max_output` bytes and the budget's deadline enforced. - #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + #[cfg(any( + feature = "html", + feature = "docx", + feature = "spreadsheet", + feature = "pdf", + feature = "markdown" + ))] pub(crate) fn render_bounded( &mut self, doc: &Document, @@ -717,7 +723,13 @@ impl Default for MarkdownRenderer { } } -#[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] +#[cfg(any( + feature = "html", + feature = "docx", + feature = "spreadsheet", + feature = "pdf", + feature = "markdown" +))] fn output_error(max: usize) -> crate::Error { crate::Error::ResourceLimit(format!("output exceeds max_output_size {max}")) } diff --git a/crates/markkit/src/tool.rs b/crates/markkit/src/tool.rs index e5208b3..2b15e8f 100644 --- a/crates/markkit/src/tool.rs +++ b/crates/markkit/src/tool.rs @@ -1076,13 +1076,14 @@ mod tests { #[tokio::test] async fn unsupported_format_is_user_facing() { - let encoded = base64::engine::general_purpose::STANDARD.encode("%PDF-1.7"); + // PPTX is detected but never supported, whatever features are on. + let encoded = base64::engine::general_purpose::STANDARD.encode("not a deck"); let e = tool() - .execution(json!({"content_base64": encoded, "file_name": "a.pdf"})) + .execution(json!({"content_base64": encoded, "file_name": "a.pptx"})) .unwrap() .execute() .await - .expect_err("pdf unsupported"); + .expect_err("pptx unsupported"); assert!(e.is_user_facing()); } diff --git a/crates/markkit/tests/common/mod.rs b/crates/markkit/tests/common/mod.rs index 268d897..375b814 100644 --- a/crates/markkit/tests/common/mod.rs +++ b/crates/markkit/tests/common/mod.rs @@ -483,3 +483,80 @@ pub fn rezip(bytes: &[u8], drop: &[&str], add: &[(&str, &[u8])]) -> Vec<u8> { pub fn xls(sheets: &[XlsSheet]) -> Vec<u8> { xls_from_stream(&biff8_workbook(sheets)) } + +// ─── PDF ──────────────────────────────────────────────────────────── + +/// Zlib-compress (for `/FlateDecode` streams). +pub fn zlib(data: &[u8]) -> Vec<u8> { + miniz_oxide::deflate::compress_to_vec_zlib(data, 9) +} + +/// A stream object body: `<< dict /Length n >> stream … endstream`. +pub fn pdf_stream(dict: &str, data: &[u8]) -> Vec<u8> { + let mut out = format!("<< {dict} /Length {} >>\nstream\n", data.len()).into_bytes(); + out.extend_from_slice(data); + out.extend_from_slice(b"\nendstream"); + out +} + +/// Assemble a PDF from object bodies (object `i + 1` = `objects[i]`) with a +/// correct xref table. Object 1 must be the catalog. +pub fn pdf_bytes_with_trailer(objects: &[Vec<u8>], trailer_extra: &str) -> Vec<u8> { + let mut out = b"%PDF-1.7\n%\xE2\xE3\xCF\xD3\n".to_vec(); + let mut offsets = Vec::new(); + for (i, body) in objects.iter().enumerate() { + offsets.push(out.len()); + out.extend(format!("{} 0 obj\n", i + 1).bytes()); + out.extend_from_slice(body); + out.extend_from_slice(b"\nendobj\n"); + } + let xref = out.len(); + out.extend(format!("xref\n0 {}\n0000000000 65535 f \n", objects.len() + 1).bytes()); + for off in offsets { + out.extend(format!("{off:010} 00000 n \n").bytes()); + } + out.extend( + format!( + "trailer\n<< /Size {} /Root 1 0 R {trailer_extra} >>\nstartxref\n{xref}\n%%EOF\n", + objects.len() + 1 + ) + .bytes(), + ); + out +} + +pub fn pdf_bytes(objects: &[Vec<u8>]) -> Vec<u8> { + pdf_bytes_with_trailer(objects, "") +} + +/// Single- or multi-page PDF; each page's content uses `/F1` = Helvetica +/// (WinAnsi). Objects: 1 catalog, 2 pages, 3 font, then (page, content) pairs. +pub fn pdf_pages(contents: &[&str]) -> Vec<u8> { + pdf_pages_with(contents, "/F1 3 0 R", &[]) +} + +/// Like [`pdf_pages`] with a custom font resource dict body and extra objects +/// appended after the pages (numbered from `4 + 2 * contents.len()`). +pub fn pdf_pages_with(contents: &[&str], fonts: &str, extra: &[Vec<u8>]) -> Vec<u8> { + let n = contents.len(); + let kids: Vec<String> = (0..n).map(|i| format!("{} 0 R", 4 + 2 * i)).collect(); + let mut objs = vec![ + b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(), + format!("<< /Type /Pages /Kids [{}] /Count {n} >>", kids.join(" ")).into_bytes(), + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>" + .to_vec(), + ]; + for (i, c) in contents.iter().enumerate() { + objs.push( + format!( + "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << {fonts} >> /XObject << /X1 {} 0 R >> >> /Contents {} 0 R >>", + 4 + 2 * n, + 5 + 2 * i + ) + .into_bytes(), + ); + objs.push(pdf_stream("", c.as_bytes())); + } + objs.extend(extra.iter().cloned()); + pdf_bytes(&objs) +} diff --git a/crates/markkit/tests/document_detection.rs b/crates/markkit/tests/document_detection.rs index bd40e18..86d0dfb 100644 --- a/crates/markkit/tests/document_detection.rs +++ b/crates/markkit/tests/document_detection.rs @@ -200,6 +200,7 @@ fn extract_document_legacy_doc_and_pptx_unsupported() { } } +#[cfg(not(feature = "pdf"))] #[test] fn pdf_is_explicitly_unsupported_not_empty() { let m = Markkit::builder().build(); diff --git a/crates/markkit/tests/pdf_integration.rs b/crates/markkit/tests/pdf_integration.rs new file mode 100644 index 0000000..47236c3 --- /dev/null +++ b/crates/markkit/tests/pdf_integration.rs @@ -0,0 +1,287 @@ +//! PDF text-layer extraction: positive mapping tests on generated fixtures. + +#![cfg(feature = "pdf")] + +mod common; + +use common::*; +use markkit::{DocumentHint, Format, Markkit}; + +fn extract(bytes: &[u8]) -> markkit::Extraction { + Markkit::builder() + .build() + .extract_document_sync(bytes, DocumentHint::content_type("application/pdf")) + .unwrap() +} + +#[test] +fn paragraphs_and_heading_from_font_size() { + let pdf = pdf_pages(&[concat!( + "BT /F1 24 Tf 72 720 Td (Annual Report) Tj ET\n", + "BT /F1 11 Tf 13 TL 72 680 Td (First paragraph line one) Tj T* (continues here.) Tj ET\n", + "BT /F1 11 Tf 13 TL 72 640 Td (Second paragraph.) Tj T* (Also second.) Tj ET\n", + )]); + let e = extract(&pdf); + assert_eq!(e.format, Format::Pdf); + assert_eq!( + e.markdown.trim(), + "# Annual Report\n\nFirst paragraph line one continues here.\n\nSecond paragraph. Also second." + ); + assert_eq!(e.stats.pages, Some(1)); + assert!(e.warnings.is_empty(), "{:?}", e.warnings); +} + +#[test] +fn detected_by_magic_bytes_without_hint() { + let pdf = pdf_pages(&["BT /F1 12 Tf 72 700 Td (Sniffed) Tj ET"]); + let e = Markkit::builder() + .build() + .extract_document_sync(&pdf, DocumentHint::default()) + .unwrap(); + assert_eq!(e.format, Format::Pdf); + assert_eq!(e.markdown.trim(), "Sniffed"); +} + +#[test] +fn extract_bytes_returns_markdown() { + let pdf = pdf_pages(&["BT /F1 12 Tf 72 700 Td (Via bytes) Tj ET"]); + let rt = tokio::runtime::Runtime::new().unwrap(); + let md = rt + .block_on( + Markkit::builder() + .build() + .extract_bytes(&pdf, Some("application/pdf")), + ) + .unwrap(); + assert_eq!(md.trim(), "Via bytes"); +} + +#[test] +fn title_from_info_dictionary() { + let content = "BT /F1 12 Tf 72 700 Td (Body) Tj ET"; + let objs = vec![ + b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(), + b"<< /Type /Pages /Kids [4 0 R] /Count 1 >>".to_vec(), + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>".to_vec(), + b"<< /Type /Page /Parent 2 0 R /Resources << /Font << /F1 3 0 R >> >> /Contents 5 0 R >>" + .to_vec(), + pdf_stream("", content.as_bytes()), + b"<< /Title (\xFE\xFF\x00Q\x00 \x04\x1A) >>".to_vec(), + ]; + let e = extract(&pdf_bytes_with_trailer(&objs, "/Info 6 0 R")); + assert_eq!(e.title.as_deref(), Some("Q К")); +} + +#[test] +fn flate_compressed_content_and_win_ansi_high_bytes() { + let content = b"BT /F1 12 Tf 72 700 Td (Caf\xE9 \x93quoted\x94 \x97 done) Tj ET"; + let objs = vec![ + b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(), + b"<< /Type /Pages /Kids [4 0 R] /Count 1 >>".to_vec(), + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>" + .to_vec(), + b"<< /Type /Page /Parent 2 0 R /Resources << /Font << /F1 3 0 R >> >> /Contents 5 0 R >>" + .to_vec(), + pdf_stream("/Filter /FlateDecode", &zlib(content)), + ]; + assert_eq!( + extract(&pdf_bytes(&objs)).markdown.trim(), + "Café “quoted” — done" + ); +} + +#[test] +fn differences_encoding_and_glyph_names() { + let font = "<< /Type /Font /Subtype /Type1 /BaseFont /Custom /Encoding << /BaseEncoding /WinAnsiEncoding /Differences [65 /Eacute /uni0416 /f_i] >> >>"; + let pdf = pdf_pages_with( + &["BT /F2 12 Tf 72 700 Td (ABC x) Tj ET"], + "/F2 7 0 R", + &[b"<< >>".to_vec(), font.as_bytes().to_vec()], + ); + assert_eq!(extract(&pdf).markdown.trim(), "ÉЖfi x"); +} + +#[test] +fn type0_font_with_to_unicode_cmap() { + let cmap = b"/CIDInit /ProcSet findresource begin 12 dict begin begincmap\n\ +1 begincodespacerange <0000> <FFFF> endcodespacerange\n\ +2 beginbfchar <0001> <041F> <0002> <0440> endbfchar\n\ +1 beginbfrange <0003> <0005> <0438> endbfrange\n\ +endcmap end end"; + let font = "<< /Type /Font /Subtype /Type0 /BaseFont /Sub+Font /Encoding /Identity-H /DescendantFonts [9 0 R] /ToUnicode 8 0 R >>"; + let desc = "<< /Type /Font /Subtype /CIDFontType2 /BaseFont /Sub+Font /DW 600 /W [1 [700 500] 3 5 550] >>"; + let pdf = pdf_pages_with( + &["BT /F2 12 Tf 72 700 Td <00010002000300040005> Tj ET"], + "/F2 7 0 R", + &[ + b"<< >>".to_vec(), + font.as_bytes().to_vec(), + pdf_stream("", cmap), + desc.as_bytes().to_vec(), + ], + ); + assert_eq!(extract(&pdf).markdown.trim(), "Прийк"); +} + +#[test] +fn tj_kerning_gap_becomes_space() { + let pdf = pdf_pages(&["BT /F1 12 Tf 72 700 Td [(Hello) -600 (world) -20 (.)] TJ ET"]); + assert_eq!(extract(&pdf).markdown.trim(), "Hello world."); +} + +#[test] +fn separate_show_ops_on_one_baseline_join_with_spaces() { + let pdf = pdf_pages(&[concat!( + "BT /F1 12 Tf 72 700 Td (Left) Tj ET\n", + "BT /F1 12 Tf 200 700 Td (Right) Tj ET\n", + )]); + assert_eq!(extract(&pdf).markdown.trim(), "Left Right"); +} + +#[test] +fn text_matrix_and_ctm_scaling() { + // 2x CTM scale on 9pt text = 18pt rendered, so it is a heading over 9pt body. + let pdf = pdf_pages(&[concat!( + "q 2 0 0 2 0 0 cm BT /F1 9 Tf 36 360 Td (Scaled) Tj ET Q\n", + "BT /F1 9 Tf 1 0 0 1 72 600 Tm (body text here) Tj ET\n", + )]); + assert_eq!(extract(&pdf).markdown.trim(), "# Scaled\n\nbody text here"); +} + +#[test] +fn bullets_become_list() { + let pdf = pdf_pages(&[ + "BT /F1 12 Tf 14 TL 72 700 Td (Intro:) Tj T* (\\225 first) Tj T* (\\225 second) Tj ET", + ]); + assert_eq!(extract(&pdf).markdown.trim(), "Intro:\n\n- first\n- second"); +} + +#[test] +fn hyphenated_line_break_is_joined() { + let pdf = pdf_pages(&["BT /F1 12 Tf 14 TL 72 700 Td (extrac-) Tj T* (tion works) Tj ET"]); + assert_eq!(extract(&pdf).markdown.trim(), "extraction works"); +} + +#[test] +fn form_xobject_text_is_extracted() { + let form = pdf_stream( + "/Type /XObject /Subtype /Form /BBox [0 0 600 800] /Resources << /Font << /F1 3 0 R >> >>", + b"BT /F1 12 Tf 72 700 Td (From a form) Tj ET", + ); + let pdf = pdf_pages_with(&["/X1 Do"], "/F1 3 0 R", &[form]); + assert_eq!(extract(&pdf).markdown.trim(), "From a form"); +} + +#[test] +fn multiple_pages_and_page_cap() { + let pdf = pdf_pages(&[ + "BT /F1 12 Tf 72 700 Td (Page one) Tj ET", + "BT /F1 12 Tf 72 700 Td (Page two) Tj ET", + "BT /F1 12 Tf 72 700 Td (Page three) Tj ET", + ]); + let e = extract(&pdf); + assert_eq!(e.markdown.trim(), "Page one\n\nPage two\n\nPage three"); + assert_eq!(e.stats.pages, Some(3)); + + let e = Markkit::builder() + .max_pdf_pages(1) + .build() + .extract_document_sync(&pdf, DocumentHint::default()) + .unwrap(); + assert_eq!(e.markdown.trim(), "Page one"); + assert_eq!(e.stats.pages, Some(1)); + assert!( + e.warnings[0].starts_with("truncated_pdf:"), + "{:?}", + e.warnings + ); +} + +#[test] +fn contents_array_is_one_stream() { + let objs = vec![ + b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(), + b"<< /Type /Pages /Kids [4 0 R] /Count 1 >>".to_vec(), + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>".to_vec(), + b"<< /Type /Page /Parent 2 0 R /Resources << /Font << /F1 3 0 R >> >> /Contents [5 0 R 6 0 R] >>" + .to_vec(), + pdf_stream("", b"BT /F1 12 Tf 72 700 Td (Split"), + pdf_stream("", b") Tj ET"), + ]; + // A string split across content streams is malformed but must not panic. + let _ = extract(&pdf_bytes(&objs)); + + let objs2 = vec![ + objs[0].clone(), + objs[1].clone(), + objs[2].clone(), + objs[3].clone(), + pdf_stream("", b"BT /F1 12 Tf 72 700 Td"), + pdf_stream("", b"(Joined) Tj ET"), + ]; + assert_eq!(extract(&pdf_bytes(&objs2)).markdown.trim(), "Joined"); +} + +#[test] +fn unmapped_composite_font_is_skipped_with_warning() { + let font = "<< /Type /Font /Subtype /Type0 /BaseFont /NoMap /Encoding /Identity-H /DescendantFonts [8 0 R] >>"; + let desc = "<< /Type /Font /Subtype /CIDFontType2 /BaseFont /NoMap >>"; + let pdf = pdf_pages_with( + &["BT /F2 12 Tf 72 700 Td <00010002> Tj ET BT /F1 12 Tf 72 650 Td (Visible) Tj ET"], + "/F1 3 0 R /F2 7 0 R", + &[ + b"<< >>".to_vec(), + font.as_bytes().to_vec(), + desc.as_bytes().to_vec(), + ], + ); + let e = extract(&pdf); + assert_eq!(e.markdown.trim(), "Visible"); + assert_eq!(e.warnings.len(), 1, "{:?}", e.warnings); + assert!(e.warnings[0].starts_with("pdf_unmapped_font: ")); + assert!(e.warnings[0].contains("NoMap")); +} + +#[test] +fn page_without_text_warns() { + let e = extract(&pdf_pages(&["0 0 m 100 100 l S"])); + assert_eq!(e.markdown.trim(), ""); + assert!( + e.warnings[0].starts_with("pdf_no_text:"), + "{:?}", + e.warnings + ); +} + +#[test] +fn missing_font_resource_falls_back_to_win_ansi() { + let pdf = pdf_pages(&["BT /Nope 12 Tf 72 700 Td (Still here) Tj ET"]); + assert_eq!(extract(&pdf).markdown.trim(), "Still here"); +} + +#[test] +fn track_sources_records_page_numbers() { + use markkit::ir::Node; + let pdf = pdf_pages(&[ + "BT /F1 12 Tf 72 700 Td (One) Tj ET", + "BT /F1 12 Tf 72 700 Td (Two) Tj ET", + ]); + let parsed = Markkit::builder() + .track_sources(true) + .build() + .parse_document_sync(&pdf, DocumentHint::default()) + .unwrap(); + let pages: Vec<Option<u32>> = parsed + .document + .nodes + .iter() + .map(|n: &Node| n.source().and_then(|s| s.page)) + .collect(); + assert_eq!(pages, vec![Some(1), Some(2)]); + + let parsed = Markkit::builder() + .build() + .parse_document_sync(&pdf, DocumentHint::default()) + .unwrap(); + assert!(parsed.document.nodes.iter().all(|n| n.source().is_none())); +} diff --git a/crates/markkit/tests/pdf_security.rs b/crates/markkit/tests/pdf_security.rs new file mode 100644 index 0000000..ce2fdcd --- /dev/null +++ b/crates/markkit/tests/pdf_security.rs @@ -0,0 +1,351 @@ +//! PDF security tests (threat model T8): decompression bombs, interpretation +//! amplification, unverifiable/encrypted input, malformed files never panic. + +#![cfg(feature = "pdf")] + +mod common; + +use common::*; +use markkit::{DocumentHint, Error, Markkit}; +use proptest::prelude::*; + +fn run(m: &Markkit, bytes: &[u8]) -> markkit::Result<markkit::Extraction> { + m.extract_document_sync(bytes, DocumentHint::content_type("application/pdf")) +} + +fn catalog_page(contents: &str) -> Vec<Vec<u8>> { + vec![ + b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(), + b"<< /Type /Pages /Kids [4 0 R] /Count 1 >>".to_vec(), + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>".to_vec(), + format!( + "<< /Type /Page /Parent 2 0 R /Resources << /Font << /F1 3 0 R >> /XObject << /X1 6 0 R >> >> /Contents {contents} >>" + ) + .into_bytes(), + ] +} + +// ─── Decompression bombs (T8a) ────────────────────────────────────── + +#[test] +fn flate_bomb_content_stream_rejected() { + // 80 MiB of spaces deflates to ~80 KiB; the per-stream cap is 64 MiB. + let bomb = zlib(&vec![b' '; 80 * 1024 * 1024]); + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("/Filter /FlateDecode", &bomb)); + let err = run(&Markkit::builder().build(), &pdf_bytes(&objs)).unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); +} + +#[test] +fn bomb_in_unreferenced_object_stream_rejected_before_parsing() { + // hayro would decode an object stream while resolving objects; the + // pre-scan must catch it even though nothing references it as content. + let bomb = zlib(&vec![b'0'; 2 * 1024 * 1024]); + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("", b"BT /F1 12 Tf 72 700 Td (ok) Tj ET")); + objs.push(pdf_stream( + "/Type /ObjStm /N 1 /First 4 /Filter /FlateDecode", + &bomb, + )); + let m = Markkit::builder().max_pdf_stream_size(1024 * 1024).build(); + let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); +} + +#[test] +fn chained_filters_are_measured_after_every_stage() { + // ASCIIHex wrapping a Flate bomb: the raw hex never inflates on its own. + let z = zlib(&vec![b'a'; 4 * 1024 * 1024]); + let hex: String = z.iter().map(|b| format!("{b:02x}")).collect(); + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream( + "/Filter [/AHx /Fl]", + format!("{hex}>").as_bytes(), + )); + let m = Markkit::builder().max_pdf_stream_size(1024 * 1024).build(); + let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); +} + +#[test] +fn escaped_filter_name_cannot_evade_scan() { + let bomb = zlib(&vec![b' '; 2 * 1024 * 1024]); + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("/Filter /Fl#61teDecode", &bomb)); + let m = Markkit::builder().max_pdf_stream_size(1024 * 1024).build(); + let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); +} + +#[test] +fn total_decoded_size_capped_across_streams() { + let chunk = zlib(&vec![b' '; 600 * 1024]); + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("/Filter /FlateDecode", &chunk)); + objs.push(pdf_stream("/Filter /FlateDecode", &chunk)); + let m = Markkit::builder() + .max_pdf_stream_size(1024 * 1024) + .max_pdf_decoded_size(1024 * 1024) + .build(); + let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + // Each stream alone is fine. + assert!(run(&Markkit::builder().build(), &pdf_bytes(&objs)).is_ok()); +} + +#[test] +fn run_length_bomb_rejected() { + let rl: Vec<u8> = std::iter::repeat_n([129u8, b' '], 20_000) + .flatten() + .collect(); // 2.5 MB + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("/Filter /RunLengthDecode", &rl)); + let m = Markkit::builder().max_pdf_stream_size(1024 * 1024).build(); + assert!(matches!( + run(&m, &pdf_bytes(&objs)), + Err(Error::ResourceLimit(_)) + )); +} + +// ─── Interpretation amplification (T8b) ───────────────────────────── + +#[test] +fn form_xobject_reuse_hits_interpretation_budget() { + let form_body = "BT /F1 12 Tf 72 700 Td (form text) Tj ET\n".repeat(200); + let page = "/X1 Do\n".repeat(5_000); + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("", page.as_bytes())); + objs.push(pdf_stream( + "/Type /XObject /Subtype /Form /BBox [0 0 600 800] /Resources << /Font << /F1 3 0 R >> >>", + form_body.as_bytes(), + )); + let m = Markkit::builder() + .max_pdf_decoded_size(4 * 1024 * 1024) + .build(); + let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); +} + +#[test] +fn self_referencing_form_does_not_recurse_forever() { + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("", b"/X1 Do BT /F1 12 Tf 72 700 Td (top) Tj ET")); + objs.push(pdf_stream( + "/Type /XObject /Subtype /Form /Resources << /XObject << /X1 6 0 R >> >>", + b"/X1 Do", + )); + let e = run(&Markkit::builder().build(), &pdf_bytes(&objs)).unwrap(); + assert_eq!(e.markdown.trim(), "top"); +} + +#[test] +fn repeated_contents_reference_is_charged_each_time() { + let content = "BT /F1 12 Tf 72 700 Td (x) Tj ET\n".repeat(1000); + let refs = "5 0 R ".repeat(2000); + let mut objs = catalog_page(&format!("[{refs}]")); + objs.push(pdf_stream("", content.as_bytes())); + let m = Markkit::builder() + .max_pdf_decoded_size(8 * 1024 * 1024) + .build(); + let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); +} + +#[test] +fn deep_save_state_nesting_and_degenerate_matrices() { + let content = format!( + "{}{} BT /F1 1e38 Tf 1e38 0 0 1e38 1e38 1e38 Tm (big) Tj ET 0 0 0 0 0 0 cm BT /F1 12 Tf (zero) Tj ET", + "q ".repeat(100_000), + "Q ".repeat(100_000) + ); + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("", content.as_bytes())); + let _ = run(&Markkit::builder().build(), &pdf_bytes(&objs)).unwrap(); +} + +#[test] +fn huge_cmap_range_and_width_arrays_are_cheap() { + let cmap = b"begincodespacerange <0000> <FFFF> endcodespacerange beginbfrange <0000> <FFFF> <0041> endbfrange"; + let font = "<< /Type /Font /Subtype /Type0 /BaseFont /F /Encoding /Identity-H /DescendantFonts [8 0 R] /ToUnicode 7 0 R >>"; + let desc = "<< /Type /Font /Subtype /CIDFontType2 /W [0 4294967295 500] >>"; + let mut objs = catalog_page("5 0 R"); + objs[3] = + b"<< /Type /Page /Parent 2 0 R /Resources << /Font << /F2 6 0 R >> >> /Contents 5 0 R >>" + .to_vec(); + objs.push(pdf_stream( + "", + b"BT /F2 12 Tf 72 700 Td <00000001FFFF> Tj ET", + )); + objs.push(font.as_bytes().to_vec()); + objs.push(pdf_stream("", cmap)); + objs.push(desc.as_bytes().to_vec()); + let e = run(&Markkit::builder().build(), &pdf_bytes(&objs)).unwrap(); + assert!(e.markdown.trim().starts_with("AB"), "{}", e.markdown); +} + +// ─── Refused input ────────────────────────────────────────────────── + +#[test] +fn encrypted_pdf_is_explicitly_unsupported() { + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("", b"BT /F1 12 Tf 72 700 Td (secret) Tj ET")); + objs.push(b"<< /Filter /Standard /V 1 /R 2 /O <00> /U <00> /P -4 >>".to_vec()); + let pdf = pdf_bytes_with_trailer(&objs, "/Encrypt 6 0 R /ID [<00> <00>]"); + let err = run(&Markkit::builder().build(), &pdf).unwrap_err(); + assert!(matches!(err, Error::UnsupportedFormat(_)), "{err:?}"); +} + +#[test] +fn indirect_filter_is_refused_not_trusted() { + let mut objs = catalog_page("5 0 R"); + objs.push(pdf_stream("/Filter 6 0 R", &zlib(b"BT ET"))); + objs.push(b"/FlateDecode".to_vec()); + let err = run(&Markkit::builder().build(), &pdf_bytes(&objs)).unwrap_err(); + assert!(matches!(err, Error::PdfExtract(_)), "{err:?}"); +} + +#[test] +fn not_a_pdf_is_an_error_not_a_panic() { + let m = Markkit::builder().build(); + for bytes in [ + &b""[..], + b"%PDF-", + b"%PDF-1.7\n%%EOF", + b"%PDF-1.4\n1 0 obj << >> endobj", + ] { + assert!(run(&m, bytes).is_err(), "{bytes:?}"); + } +} + +#[test] +fn max_input_size_honoured_for_pdf() { + let pdf = pdf_pages(&["BT /F1 12 Tf 72 700 Td (x) Tj ET"]); + let m = Markkit::builder().max_input_size(pdf.len() - 1).build(); + assert!(matches!(run(&m, &pdf), Err(Error::ResourceLimit(_)))); +} + +// ─── Malformed input never panics (S9) ────────────────────────────── + +fn fixtures() -> Vec<Vec<u8>> { + let cmap = b"begincodespacerange <0000> <FFFF> endcodespacerange 1 beginbfchar <0001> <0041> endbfchar"; + let font = "<< /Type /Font /Subtype /Type0 /BaseFont /F /Encoding /Identity-H /DescendantFonts [<< /Subtype /CIDFontType2 /W [1 [500]] >>] /ToUnicode 7 0 R >>"; + let mut type0 = catalog_page("5 0 R"); + type0[3] = + b"<< /Type /Page /Parent 2 0 R /Resources << /Font << /F2 6 0 R >> >> /Contents 5 0 R >>" + .to_vec(); + type0.push(pdf_stream( + "/Filter /FlateDecode", + &zlib(b"BT /F2 12 Tf 72 700 Td <0001> Tj ET"), + )); + type0.push(font.as_bytes().to_vec()); + type0.push(pdf_stream("", cmap)); + + let mut form = catalog_page("5 0 R"); + form.push(pdf_stream("", b"q 1 0 0 1 10 10 cm /X1 Do Q")); + form.push(pdf_stream( + "/Type /XObject /Subtype /Form /Resources << /Font << /F1 3 0 R >> >>", + b"BT /F1 12 Tf [(a) -500 (b)] TJ ET", + )); + vec![ + pdf_pages(&[ + "BT /F1 18 Tf 72 720 Td (Title) Tj ET BT /F1 11 Tf 14 TL 72 680 Td (one) Tj T* (two) Tj ET", + "BT /F1 11 Tf 72 700 Td (\\225 item) Tj ET", + ]), + pdf_bytes(&type0), + pdf_bytes(&form), + ] +} + +#[test] +fn fixtures_parse_cleanly() { + let m = Markkit::builder().build(); + for f in fixtures() { + let e = run(&m, &f).unwrap(); + assert!(!e.markdown.trim().is_empty()); + } +} + +#[test] +fn truncated_pdfs_never_panic() { + let m = Markkit::builder().build(); + for f in fixtures() { + for cut in 0..f.len() { + let _ = run(&m, &f[..cut]); + } + } +} + +proptest! { + #![proptest_config(ProptestConfig::with_cases(256))] + + #[test] + fn random_bytes_after_header_never_panic(data in proptest::collection::vec(any::<u8>(), 0..2048)) { + let mut bytes = b"%PDF-1.7\n".to_vec(); + bytes.extend(data); + let _ = run(&Markkit::builder().build(), &bytes); + } + + #[test] + fn mutated_pdfs_never_panic( + idx in 0usize..3, + flips in proptest::collection::vec((any::<prop::sample::Index>(), any::<u8>()), 1..16), + ) { + let mut bytes = fixtures()[idx].clone(); + for (pos, val) in flips { + let i = pos.index(bytes.len()); + bytes[i] = val; + } + let _ = run(&Markkit::builder().build(), &bytes); + } + + #[test] + fn arbitrary_content_stream_never_panics(content in "[ -~\\n]{0,400}") { + let _ = run(&Markkit::builder().build(), &pdf_pages(&[&content])); + } +} + +// ─── Shared limits ────────────────────────────────────────────────── + +#[test] +fn zero_timeout_rejects() { + let bytes = pdf_pages(&["BT /F1 12 Tf 72 720 Td (x) Tj ET"]); + let m = Markkit::builder() + .timeout(std::time::Duration::ZERO) + .build(); + match run(&m, &bytes) { + Err(Error::ResourceLimit(msg)) => assert!(msg.contains("timeout"), "{msg}"), + other => panic!("expected timeout, got {other:?}"), + } +} + +#[test] +fn pdf_limits_settable_via_limits_struct() { + let bytes = pdf_pages(&[ + "BT /F1 12 Tf 72 720 Td (one) Tj ET", + "BT /F1 12 Tf 72 720 Td (two) Tj ET", + ]); + let m = Markkit::builder() + .limits(markkit::Limits::default().max_pdf_pages(1)) + .build(); + let out = run(&m, &bytes).unwrap(); + assert!(out.markdown.contains("one") && !out.markdown.contains("two")); +} + +#[test] +fn output_cap_rejects() { + let content: String = (0..200) + .map(|i| { + format!( + "BT /F1 12 Tf 72 {} Td (paragraph {i}) Tj ET\n", + 7000 - i * 30 + ) + }) + .collect(); + let bytes = pdf_pages(&[content.as_str()]); + let m = Markkit::builder().max_output_size(500).build(); + match run(&m, &bytes) { + Err(Error::ResourceLimit(msg)) => assert!(msg.contains("max_output_size"), "{msg}"), + other => panic!("expected output cap, got {other:?}"), + } +} diff --git a/docs/getting-started.md b/docs/getting-started.md index cae9b8f..3dfc280 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -9,11 +9,11 @@ Add markkit to your `Cargo.toml`: markkit = "0.1" ``` -For DOCX and spreadsheets: +For PDF, DOCX and spreadsheets: ```toml [dependencies] -markkit = { version = "0.1", features = ["docx", "spreadsheet"] } +markkit = { version = "0.1", features = ["pdf", "docx", "spreadsheet"] } ``` ## Basic Usage @@ -86,6 +86,6 @@ Node.js and WASM. ## Next Steps - [HTML extraction](../knowledge/formats/html-extraction.md): element mapping and stripping rules -- [Format support](../knowledge/formats/format-support.md): formats, detection, DOCX and spreadsheet mapping +- [Format support](../knowledge/formats/format-support.md): formats, detection, DOCX, spreadsheet and PDF mapping - [Threat model](../knowledge/security/threat-model.md): security boundaries and limits - [Examples](../examples/): runnable with `cargo run -p markkit --example <name>` diff --git a/knowledge/formats/format-support.md b/knowledge/formats/format-support.md index 436d7ba..1d090a3 100644 --- a/knowledge/formats/format-support.md +++ b/knowledge/formats/format-support.md @@ -1,12 +1,13 @@ --- type: Subsystem Design title: Format Support -description: Supported and planned formats, bytes-in API, format detection, DOCX, spreadsheet, plain text and Markdown mapping. +description: Supported and planned formats, bytes-in API, format detection, DOCX, spreadsheet, PDF, plain text and Markdown mapping. tags: - markkit - formats - docx - spreadsheet + - pdf - detection - markdown --- @@ -23,7 +24,7 @@ tags: - **DOCX** (`docx`): Word documents via a direct `zip` + `quick-xml` reader (no docx-rs) - **XLSX/XLSM, XLS, ODS** (`spreadsheet`): Workbooks to Markdown tables via calamine - **Markdown** (`markdown`): Re-parsed with pulldown-cmark into the IR and rendered back (`text/markdown`, `text/x-markdown`, `.md`, `.markdown`, `.mdown`, `.mkd`, `.mkdn`) -- **PDF** (`pdf`): Planned. Returns `Error::UnsupportedFormat` (never an empty success) +- **PDF** (`pdf`): Text layer via hayro-syntax (pure Rust); no OCR, encrypted PDFs rejected ### Detected, not supported - **DOC** (Word 97-2003 binary), **PPTX**, **XLSB**: explicit `UnsupportedFormat` errors @@ -65,6 +66,7 @@ concatenate cleanly); an empty document renders `""`. This is intended and stabl | XLSX | `docProps/core.xml` `dc:title` | | ODS | `meta.xml` `dc:title` | | XLS | `\u{5}SummaryInformation` property set, `PIDSI_TITLE` (code page aware: 1200 UTF-16, UTF-8, else Latin-1) | +| PDF | `/Info /Title` (UTF-16BE with BOM, else PDFDocEncoding) | Whitespace is collapsed, titles are capped at 1024 characters, and a missing or malformed property part yields `None` without failing the extraction. @@ -73,7 +75,8 @@ malformed property part yields `None` without failing the extraction. only the Markdown. Parsing is CPU-bound; async callers should use the sync variant in `spawn_blocking`. -Warning codes: `truncated_sheet`, `hidden_sheet_skipped`, `sheet_error`, `invalid_utf8`. +Warning codes: `truncated_sheet`, `hidden_sheet_skipped`, `sheet_error`, `truncated_pdf`, +`pdf_unmapped_font`, `pdf_no_text`, `invalid_utf8`. ## Format Detection @@ -133,6 +136,36 @@ numbering through the main part's relationships. - Only cached cell values are read: formulas are not evaluated, VBA/XLM macros are never executed or rendered. +## PDF Mapping + +Text layer only (`crates/markkit/src/pdf/`). Layout is reconstructed from glyph positions: + +| PDF | Markdown | +|---|---| +| `Tj` / `TJ` / `'` / `"` runs on one baseline, moving right | One line; gaps > 0.15 em and `TJ` adjustments < −200 become spaces | +| Consecutive lines | One paragraph, unless the vertical step exceeds 1.3× the page's most common line pitch, the text moves up (new column), or the size changes by > 15% | +| `letter-` at line end + lowercase next line; soft hyphen | Joined without the hyphen | +| Paragraph ≤ 3 lines, ≤ 200 chars, size ≥ 1.2× body size (size with most characters) | Heading; distinct heading sizes, largest first, map to `#`..`######` | +| Line starting `•`, `●`, `▪`, `◦`, `‣`, `∙`, `■` or `–` + space | Unordered list item (consecutive items merge) | +| Form XObjects (`Do`) | Followed with their matrix and resources; depth 8, cycles skipped | +| Pages | Paragraphs never cross pages; no page separators | +| `/Info /Title` | `Extraction::title` (UTF-16BE / PDFDocEncoding decoded) | +| `track_sources(true)` | `Source::page` (1-based) on each top-level block | +| Images, annotations, outlines, form fields | Skipped | + +Unicode per font, in order: `ToUnicode` CMap (`bfchar`, `bfrange` incl. arrays, code +lengths from `codespacerange`); simple fonts: `/Encoding` (WinAnsi, MacRoman, Standard) ++ `/Differences` glyph names (AGL subset, `uniXXXX`, `uXXXX`, `a_b` ligatures, `.suffix` +stripped); otherwise WinAnsi. Type0 fonts without `ToUnicode` are skipped with +`pdf_unmapped_font` (no CID guessing). Ligature presentation forms are expanded +(`fi` → `fi`); control characters are dropped. Text render mode is ignored, so invisible +OCR layers (mode 3) are extracted. Encrypted documents (an `/Encrypt` key anywhere) +return `UnsupportedFormat`; a page set with no text emits `pdf_no_text`. + +Known gaps (follow-ups): multi-column reading order relies on content-stream order, +no table detection, no repeated header/footer removal, bold/italic not mapped, numbered +lists stay paragraphs, rotated text is not re-oriented. + ## Plain Text and Markdown Shared decoding: UTF-8 (BOM stripped); UTF-16 LE/BE only with a BOM; otherwise lossy UTF-8 @@ -172,8 +205,12 @@ Office formats additionally enforce (see [Threat Model](../security/threat-model - `max_spreadsheet_cells` (4 000 000) for the dense grid calamine builds for XLS/ODS - XML nesting depth 256 for DOCX parts +PDF additionally enforces (T8): `max_pdf_pages` (10 000; truncate + `truncated_pdf`), +`max_pdf_stream_size` (64 MiB per decoded stream), `max_pdf_decoded_size` (256 MiB total +decoded in the pre-scan, and separately the total content interpreted, form reuse included). + ## See also - [HTML Extraction](html-extraction.md), the Tier 1 path - [IR Specification](../foundations/ir.md), the shared output of every parser -- [Threat Model](../security/threat-model.md), T6 malicious office documents +- [Threat Model](../security/threat-model.md), T6 malicious office documents, T7 text and Markdown, T8 malicious PDFs diff --git a/knowledge/log.md b/knowledge/log.md index 64d0daf..b6fd9a9 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -21,6 +21,8 @@ * **Page metadata**: HTML `extract_document` now fills `title` and a new `Extraction::metadata` (description, author, published, language, canonical URL, site name, image) from `<meta>`, Open Graph, `<link rel=canonical>` and bounded JSON-LD. Precedence table in [HTML Extraction](formats/html-extraction.md); scheme filter and JSON-LD bounds in [Threat Model](security/threat-model.md) T1 / T2f. * **IR v2**: footnotes, definition lists, task lists, figures, table alignment and spans; `Document.ir_version` (`IR_VERSION = 2`), `#[non_exhaustive]` IR types, `Node::is_block`/`is_inline`. HTML maps GitHub/Pandoc footnotes, `<dl>`, checkbox items, captioned figures, `colspan`/`rowspan`/alignment and row-header `<th>`. New threat T2e and invariant S13 (span grid budget, footnote label sanitizing). See [IR Specification](foundations/ir.md), [HTML Extraction](formats/html-extraction.md), [Threat Model](security/threat-model.md). * **Agent tool**: new [Tool Contract](integrations/tool-contract.md) concept in a new `integrations/` domain. `MarkkitTool` (opt-in `tool` feature) follows the Everruns toolkit-library contract; tool arguments are a new untrusted surface in the [Threat Model](security/threat-model.md). + +* **PDF support**: the `pdf` feature extracts the text layer with hayro-syntax (pure Rust). New threat T8 (bombs via hayro's unbounded object-stream decoding, interpretation amplification, hidden text) with a raw-byte stream pre-scan and bounded decoders; invariant S19; `Limits` fields `max_pdf_pages`, `max_pdf_stream_size`, `max_pdf_decoded_size`. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md). * **URL fetch hardening**: T3 SSRF is mitigated. `extract_url` blocks non-public addresses by default (URL, resolver and redirect layers), enforces scheme, allowlist, redirect, body and time limits; new invariant S12 and `url_security` tests. Proxy-resolved names are the documented residual. See [Threat Model](security/threat-model.md) and [Security Model](security/security-model.md). * **HTTP client opt-in**: `http_client` dropped from the `markkit` default features; the default build has no reqwest/hyper. CLI and bindings enable it explicitly; CI asserts the default tree stays network-free. See [Threat Model](security/threat-model.md) T3 and [Maintenance](operations/maintenance.md). * **Dependency trim**: unused dependencies removed (lockfile shrinks by ~460 lines; the `instant`/`paste` unmaintained advisories and their `deny.toml` ignores are gone with the stub bench deps). quick-xml moves to 0.42, one version shared with calamine; markkit's XML walkers now read `&str` names and values. See [Threat Model](security/threat-model.md) T6d and [Maintenance](operations/maintenance.md). diff --git a/knowledge/operations/testing.md b/knowledge/operations/testing.md index 530e9d7..614e090 100644 --- a/knowledge/operations/testing.md +++ b/knowledge/operations/testing.md @@ -24,7 +24,9 @@ tags: | `crates/markkit/tests/document_detection.rs` | Format detection order and explicit unsupported errors | | `crates/markkit/tests/document_security.rs` | Zip bombs, grid amplification, XML entities, malformed containers | | `crates/markkit/tests/limits.rs` | Every `Limits` field enforced for HTML, DOCX, spreadsheets; small-stack depth ceiling | -| `crates/markkit/tests/common/` | In-memory DOCX/XLSX/XLS/ODS fixture builders | +| `crates/markkit/tests/pdf_integration.rs` | PDF text layout, fonts/encodings, forms, page cap | +| `crates/markkit/tests/pdf_security.rs` | PDF bombs, interpretation budget, encrypted/unverifiable input, malformed PDFs | +| `crates/markkit/tests/common/` | In-memory DOCX/XLSX/XLS/ODS/PDF fixture builders | | `crates/markkit-cli/tests/cli_integration.rs` | CLI flags and output | | `crates/markkit-cli/tests/mcp_integration.rs` | `markkit mcp` stdio session end to end | | `crates/markkit/benches/extraction.rs` | Criterion micro-benchmarks | diff --git a/knowledge/security/security-model.md b/knowledge/security/security-model.md index f2998c8..2942497 100644 --- a/knowledge/security/security-model.md +++ b/knowledge/security/security-model.md @@ -43,6 +43,11 @@ Markkit processes untrusted input from the internet. Key threats: - **Mitigation**: Markdown re-parsed into the IR; raw HTML never copied (HTML blocks go through the HTML parser) - See [Threat Model](threat-model.md) T7 +### TM-PDF: Malicious PDFs +- **Mitigation**: Every stream decode-checked against size limits before parsing (object streams included); unverifiable filters and encrypted files refused +- **Mitigation**: Interpretation budget (form XObject reuse), form depth limit, page cap +- **Residual**: invisible text (OCR layers) is extracted; see [Threat Model](threat-model.md) T8 + ### TM-MEM: Memory exhaustion - **Mitigation**: Streaming processing where possible - **Mitigation**: Memory budgets for large documents diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 3598fde..d0f18e7 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -1,7 +1,7 @@ --- type: Threat Model title: Threat Model -description: Attack surface, threats T1-T6 with mitigations and test coverage, resource limits, and security invariants. +description: Attack surface, threats T1-T7 with mitigations and test coverage, resource limits, and security invariants. tags: - markkit - security @@ -325,6 +325,65 @@ in a `.txt`) is emitted as `Node::Text` and escaped by the renderer (S17), like - `security_text_and_markdown_share_limits` (timeout, `max_nodes`) - `security_plain_text_markup_is_escaped` +### T8: Malicious PDFs + +**Description**: Crafted PDFs aimed at exhausting memory/CPU through stream +decompression or content amplification, crashing the parser, or hiding text. + +**Impact**: High -- OOM abort, CPU exhaustion. + +**Likelihood**: Medium -- PDFs are the most common document type fetched by agents. + +**Sub-threats and mitigations** (`crates/markkit/src/pdf/`): + +#### T8a: Decompression bombs +- hayro-syntax decodes object and xref streams itself, without limits, while resolving + objects. So before hayro sees the file, `guard::prescan` finds every `stream` keyword + in the raw bytes (streams cannot be nested in object streams), reads `/Filter` from the + preceding dictionary text (`#xx` name escapes resolved, abbreviations accepted) and + decodes the chain with markkit's own bounded Flate (miniz_oxide), LZW, RunLength, + ASCIIHex and ASCII85 decoders, measuring every stage. Over `max_pdf_stream_size` + (64 MiB) for one stream or `max_pdf_decoded_size` (256 MiB) in total → + `Error::ResourceLimit`. +- Streams markkit interprets (page content, form XObjects, ToUnicode CMaps) are decoded + with the same bounded decoders, never hayro's `decoded()`. +- Refused instead of trusted: an indirect or repeated `/Filter` (`Error::PdfExtract`), + and encrypted files (`/Encrypt` key; the scan would only see ciphertext) → + `UnsupportedFormat`. Image filters end a chain; hayro is built without image decoders. + +#### T8b: Interpretation amplification +- Every decoded content stream is charged to an interpretation budget equal to + `max_pdf_decoded_size`, per use: a form XObject drawn 10 000 times, or one stream listed + 10 000 times in `/Contents`, exhausts it → `ResourceLimit`. Operators are charged too. +- Form XObjects: depth 8, recursion into a form already on the stack is skipped. +- `q` nesting capped at 256 saved states; non-finite matrices are ignored. +- CMap `bfrange`s are never expanded (binary-searched ranges); `/W` arrays capped at 65 536 entries. +- `max_pdf_pages` (10 000) truncates with a `truncated_pdf` warning. + +#### T8c: Parser crashes on malformed input +- hayro-syntax repairs broken xref tables and returns `None` on bad objects; markkit + maps load failures to `Error::PdfExtract`. Parsing runs under `catch_unwind` like the + office parsers (defence in depth). + +#### T8d: Hidden text +- Residual: text render mode 3 (invisible) is extracted, because that is how OCR layers + of scanned PDFs are stored; white-on-white or off-page text is also extracted. Treat + PDF text as untrusted content (prompt-injection vector), as with any source. + +**Test Coverage** (`tests/pdf_security.rs`): +- `flate_bomb_content_stream_rejected`, `bomb_in_unreferenced_object_stream_rejected_before_parsing`, + `chained_filters_are_measured_after_every_stage`, `escaped_filter_name_cannot_evade_scan`, + `total_decoded_size_capped_across_streams`, `run_length_bomb_rejected` +- `form_xobject_reuse_hits_interpretation_budget`, `repeated_contents_reference_is_charged_each_time`, + `self_referencing_form_does_not_recurse_forever`, `deep_save_state_nesting_and_degenerate_matrices`, + `huge_cmap_range_and_width_arrays_are_cheap` +- `encrypted_pdf_is_explicitly_unsupported`, `indirect_filter_is_refused_not_trusted`, + `not_a_pdf_is_an_error_not_a_panic`, `max_input_size_honoured_for_pdf`, + `zero_timeout_rejects`, `output_cap_rejects`, `pdf_limits_settable_via_limits_struct` +- `truncated_pdfs_never_panic`, `random_bytes_after_header_never_panic`, + `mutated_pdfs_never_panic`, `arbitrary_content_stream_never_panics` +- Unit: `pdf::guard::tests::*` (decoders, caps, filter parsing), `pdf::font::tests::*` + ### T5: Information Disclosure **Description**: Error messages or output leak sensitive information. @@ -354,6 +413,10 @@ in a `.txt`) is emitted as `Node::Text` and escaped by the renderer (S17), like | ZIP entry size (decompressed) | 64 MiB | `max_archive_entry_size(u64)` | While decompressing | | ZIP total size (decompressed) | 256 MiB | `max_archive_total_size(u64)` | While decompressing | | Spreadsheet grid (XLS/ODS) | 4 000 000 cells | `max_spreadsheet_cells(u64)` | Pre-scan before calamine | +| PDF stream (decoded) | 64 MiB | `max_pdf_stream_size(usize)` | Pre-scan + every interpreted stream | +| PDF decoded total / interpreted content | 256 MiB | `max_pdf_decoded_size(usize)` | Pre-scan; interpretation budget | +| PDF pages | 10 000 | `max_pdf_pages(usize)` | Page loop (truncate + warn) | +| PDF form XObject depth | 8 | Hardcoded `MAX_FORM_DEPTH` | `Do` operator | | Rows / columns per sheet | 10 000 / 256 | `max_sheet_rows`, `max_sheet_cols` | While collecting cells (truncate + warn) | ## Malicious Input Categories @@ -392,9 +455,9 @@ in a `.txt`) is emitted as `Node::Text` and escaped by the renderer (S17), like | S6 | Deep nesting does not cause stack overflow | `html_security::deeply_nested_elements` | | S7 | Non-content elements produce no output | `html_integration::script_stripped`, `style_stripped`, `form_elements_stripped` | | S8 | Archive decompression is bounded by measured output, not headers | `document_security::*zip_bomb*`, `archive_entry_count_capped` | -| S9 | Malformed office documents produce errors, never panics | `document_security::truncated_fixtures_never_panic`, `*_never_panic` proptests | +| S9 | Malformed office documents and PDFs produce errors, never panics | `document_security::truncated_fixtures_never_panic`, `pdf_security::*_never_panic`, `*_never_panic` proptests | | S10 | Spreadsheet parsing never allocates a grid beyond `max_spreadsheet_cells` | `document_security::xls_*_rejected`, `ods_repeated_rows_and_columns_rejected` | -| S11 | Unsupported formats fail explicitly (no empty success) | `document_detection::pdf_is_explicitly_unsupported_not_empty` | +| S11 | Unsupported formats fail explicitly (no empty success) | `document_detection::*unsupported*`, `pdf_security::encrypted_pdf_is_explicitly_unsupported` | | S12 | Default `extract_url` never connects to a non-public address | `url_security::*_blocked*`, `fetch::tests::non_public_addresses_forbidden` | | S13 | Table spans never grow the render grid past `MAX_TABLE_GRID_CELLS`; footnote labels cannot leave `[^...]` | `html_security::hostile_colspan_rowspan_bounded`, `html_integration::footnote_label_cannot_break_out` | | S14 | Returned Markdown never exceeds `max_output_size` | `limits::*output_cap*` | @@ -402,9 +465,10 @@ in a `.txt`) is emitted as `Node::Text` and escaped by the renderer (S17), like | S16 | Extraction stops with `ResourceLimit` once `timeout` passes (checked at least after each non-interruptible third-party step) | `limits::*zero_timeout_rejects`, `limits::tests::*` | | S17 | Rendered markdown has no raw HTML, no non-allowlisted link/image scheme, and document text cannot create Markdown structure | `output_safety::*` (pulldown-cmark oracle + proptests) | | S18 | Text and Markdown output never contains C0 controls (except tab/newline); Markdown raw HTML (blocks, inline tags) is never copied | `text_integration::security_*` | +| S19 | No PDF stream is decoded past `max_pdf_stream_size`, by markkit or hayro | `pdf_security::*bomb*`, `pdf_security::chained_filters_are_measured_after_every_stage` | ## See also - [Security Model](security-model.md), the one-page summary of these threats -- [Format Support](../formats/format-support.md), office-format limits referenced by T6 +- [Format Support](../formats/format-support.md), office-format and PDF limits referenced by T6 and T8 - [Testing Strategy](../operations/testing.md), where the security suites live