[chaliy]

Requested by Михайло · project thread

What changed

PDFs now convert to Markdown instead of returning UnsupportedFormat (pdf feature; already on in the CLI all_formats build and the Python/Node bindings). WASM still excludes pdf.

• Text layer only, pure Rust: hayro-syntax 0.8 with default-features = false (adds only rustc-hash + smallvec) and miniz_oxide for bounded inflate.
• Glyph runs are placed using the text/graphics state (Tm, CTM, Tc/Tw/Tz/TL/Ts, TJ kerning), then rebuilt into lines, paragraphs (gap vs. the page's most common line pitch), font-size headings, and bullet lists. Hyphenated line breaks are joined, ligatures are expanded, and form XObjects are followed.
• Fonts: ToUnicode CMaps (bfchar/bfrange, codespace code lengths), WinAnsi/MacRoman/Standard plus /Differences glyph names. Type0 fonts with no ToUnicode are skipped with a warning rather than emitting CID garbage.
• Extraction: title comes from /Info /Title, stats.pages is filled, and track_sources records the page per block. New warnings: truncated_pdf, pdf_unmapped_font, pdf_no_text.
• New Limits fields (also builder setters, CLI flags --max-pdf-*, Python Limits(...) args, Node maxPdf* options): max_pdf_pages (10 000), max_pdf_stream_size (64 MiB), max_pdf_decoded_size (256 MiB). PDF extraction also honours timeout and max_output_size from feat(limits): unified Limits with timeout, output, node and depth caps #27.
• Encrypted PDFs return UnsupportedFormat.
• Rendering goes through the shared escaping renderer from fix(render): escape markdown output and allowlist URL schemes #23, so PDF text cannot produce Markdown structure or raw HTML.

Why

This is a roadmap initiative: PDF is the most common document format agents get, and it was the biggest format gap.

Security (new threat T8, invariant S19)

hayro-syntax has no decode limits, and it decodes object streams by itself while resolving objects. A bomb disguised as an object stream would exhaust memory before markkit sees a page. Mitigations:

• Pre-scan before hayro runs: every stream keyword in the raw bytes is found, its /Filter chain is read from the dictionary text (#xx escapes and abbreviations handled), and the chain is decoded with markkit's own bounded Flate, LZW, RunLength, AHx and A85 decoders. Each stage is measured, and the document is rejected past the per-stream or total cap.
• Bounded decoding for what we interpret: page content, forms, and CMaps go through the same bounded decoders and never through hayro's decoded().
• Interpretation budget: each use is charged, so a form XObject drawn 10k times, or one stream listed 10k times in /Contents, cannot multiply CPU. Forms have a depth limit of 8 and are cycle-safe. q nesting is capped and CMap ranges are never expanded. The call deadline is checked per page and in the operator loop.
• Refused rather than trusted: an indirect or duplicated /Filter, and encrypted files, since the scan would see only ciphertext.
• Documented residual risk: invisible text (render mode 3, used by OCR layers) is extracted.

Also fixed while merging: with only pdf (or only markdown) enabled, extraction skipped the bounded renderer, so max_output_size and the deadline were not applied to rendering. Covered by pdf_security::output_cap_rejects.

Before / After

Before:

$ markkit extract --file report.pdf
Error: Unsupported format: PDF support requires the 'pdf' feature

After, on a LibreOffice-exported PDF (Type0 fonts with ToUnicode):

# Office Heading

LibreOffice paragraph with bold and Unicode: Київ, 日本.

- item one
- item two

On a 4 000-paragraph reportlab PDF (290 KB), output is 1.43 MB: 0.17 s, against 0.30 s for pdftotext on the same file.

Tests: pdf_integration (19), pdf_security (23, including timeout, output cap, truncation, mutation and random-byte proptests), plus unit tests for the decoders, CMap and glyph names; CLI flag-mapping test and a Node invalid-option test for the new limits. cargo test --workspace --all-features, clippy -D warnings (all features, default, pdf-only, no features) and check_okf pass locally.

Risk

• Medium: this is a new parser on untrusted input, but it is feature-gated and off in the default build.
• Layout is heuristic. Known gaps for follow-ups: multi-column reading order (content-stream order is used), tables, repeated header/footer removal, bold/italic, numbered lists.
• A Flate-compressed image over 64 MiB, or more than 256 MiB of decoded Flate streams in total, rejects the whole document. Both limits are configurable.
• A fuzz target will come once the fuzz crate lands.

Checklist

• Tests added or updated
• Backward compatibility considered (new Limits fields; the pdf feature's behavior changes from an error to extraction)

🤖 Generated with Claude Code