[chaliy]

Requested by Михайло · project thread

What changed

extract_document/parse_document now accept plain text and Markdown. That covers the CLI, extract_bytes, extract_url, Python, Node and WASM. This is the first of the roadmap's "cheap formats" PRs.

• Plain text (text/plain, .txt, .text):
  • Always on, with no new dependencies.
  • Blank lines separate paragraphs. Single newlines become hard breaks.
  • The text goes through the IR as Node::Text, so the renderer's escaping applies to it like any other format. A <script> line in a .txt comes out escaped.
• Markdown (text/markdown, text/x-markdown, .md/.markdown/.mdown/.mkd/.mkdn):
  • Behind a new markdown feature. It is part of all_formats, so the CLI, Python and Node get it, and it is enabled in the WASM package too.
  • Re-parsed, never passed through: pulldown-cmark parses it into IR v2 and the renderer writes it back out.
  • Supported syntax: GFM tables (with alignment), strikethrough, task lists, footnotes and definition lists. Soft breaks reflow into spaces.
  • YAML front matter is not rendered. Its title: becomes title; without one, the first # heading is used.
  • HTML blocks go through the HTML parser, or are dropped without the html feature. Inline HTML tags are dropped.
• Limits: both formats use the call's Limits budget. Each line or event checks timeout, start tags count against max_nodes, and nesting is capped at max_nesting_depth. HTML blocks share the same budget.
• Shared decoding:
  • UTF-8 with the BOM stripped, and UTF-16 LE/BE when there is a BOM. Anything else is decoded lossily with an invalid_utf8 warning.
  • CRLF/CR become LF. C0 controls other than tab and newline, and DEL, are dropped.
• Detection: text is never sniffed from bytes, so unlabelled input stays UnsupportedFormat.
• Format::PlainText now serializes as "text", the same as as_str().
• Transforms run on text and Markdown documents as they do for other formats.

Why

Roadmap initiative "cheap formats". Plain text was listed as planned but returned UnsupportedFormat. Normalizing Markdown through the IR gives it the same security guarantees as every other format.

Before / After

Before:

$ markkit extract --file notes.txt
Error: Unsupported format: text

After:

$ printf 'line one\nline two\n' > notes.txt && markkit extract --file notes.txt
line one  
line two
$ printf 'Title\n=====\n\n* a\n\n<script>x()</script>\n' > doc.md && markkit extract --file doc.md
# Title

- a

Tests:

• tests/text_integration.rs has 18 tests. 12 of them are security tests for the new threat T7 and invariant S18: control characters, invalid UTF-8/UTF-16, raw HTML, escaped plain-text markup, 100k-deep nesting, timeout/max_nodes, size limits and random bytes.
• There are also unit tests in text/markdown (including the depth limit and max_nodes), a CLI test and a WASM node test.
• All of these pass: the workspace with default features and with all features, the docx,spreadsheet,markdown no-HTTP build, clippy and check-okf.

Risk

• Low. This PR only adds behavior.
• Markdown output is a normalization of the input, not a byte-identical copy.
• New optional dependency: pulldown-cmark 0.13 (MIT, default features off). It is the same crate that fix(render): escape markdown output and allowlist URL schemes #23 uses as its test oracle.

Checklist

• Tests added or updated
• Backward compatibility considered (new feature flag, CHANGELOG Added entry, knowledge T7/S18)

---

🤖 Generated with Claude Code