Repository navigation
feat: IR transform passes between parse and render - #36
Merged
Merged
Conversation
- transform::Transform trait, TransformChain on MarkkitBuilder (.transform(..)), applied by parse_html/parse_document/extract_* - built-ins NormalizeHeadings, DropImages; transform::from_fn - Document::visit/visit_mut, Node::for_each_child_list[_ref] - Error is #[non_exhaustive], new Error::Transform - examples/transforms.rs
2 tasks done
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMDLeuybnBNqgVL9B9kMmARU)_
## What changed
Markdown output is now safe to render, for every input format. The PR is
rebased onto current `main`, so it covers IR v2 (footnotes, definition
lists, figures, task items, table spans), math (#34), transforms (#36),
relative URL resolution (#31) and the limits from #27. #27's output and
deadline checks are kept in the reworked renderer.
- **Text renders verbatim.** These characters are escaped:
- backtick, `*`, `_` (at word boundaries), `[`, `]`, the less-than sign,
`~`, `|` and entity-like `&`;
- line-start block markers: `#`, `>`, `-`, `+`, `=`, `1.`.
Document text can no longer produce raw HTML, links, emphasis, headings,
lists or tables.
- **URL scheme allowlist.** Links keep `http`, `https`, `mailto`, `tel`,
`ftp` and relative URLs. Images keep `http`, `https`, relative URLs and
raster `data:image/*`. Any other scheme is dropped and its link text or
alt text is kept: `javascript:`, `vbscript:`, `data:text/html`, `file:`.
Scheme detection ignores whitespace and control characters and reads the
whole scheme.
- **Destinations and titles.** Parentheses, angle brackets, backslash
and spaces are percent-encoded. Entity-like `&` is escaped, so
`javascript:` cannot decode into a scheme. Empty destinations are
written `<>`. Titles escape quotes, backslash and less-than.
- **Code.** Fences and spans are longer than any backtick run inside
them, and adjacent spans cannot merge their fences. The code-block
language keeps only its first word.
- **Math renders as code.** TeX comes from page markup, and plain GFM
has no math syntax, so `$tex$` let an annotation containing an img tag
or a link through. Math now uses GitHub's forms: `` $`tex`$ `` inline
and a ```` ```math ```` fence for display math. GitHub renders both as
math. Every other renderer shows them as inert code.
- **Structure stays in its container.**
- Block nodes inside inline content are flattened to one line. Examples:
a code block or display math in a table cell, a paragraph in a heading.
- List item bodies are indented by the marker width, so content nested
under `1.` items now nests correctly.
- Every block starts on its own line, and hard breaks never produce a
blank line.
- Nested links render as text.
- **Raw text in the IR.** The DOCX and spreadsheet parsers stop
pre-escaping table pipes, so the JSON IR holds raw text. The renderer
escapes pipes in table context.
All escaping lives in the new `render/escape.rs`, with the design
decisions in its header. The new invariant is S17, since #27 took S14 to
S16.
## Why
Threat model T1 promised "no HTML pass-through", but the renderer
escaped nothing. On `main`, a probe page showed four problems:
- it produced a raw img tag with an onerror handler;
- it kept `javascript:` links;
- a triple-backtick line inside a `pre` block closed the fence;
- links broke on `)`.
A fuzzing pass against the new oracle then found eight more ways content
could escape its container. The cargo-fuzz harness comes in a follow-up
PR. Each finding is a regression test here.
## Before / After
Input (HTML) is a paragraph containing `# not heading *x*`, followed by:
- an escaped img tag with an onerror handler,
- a link to `javascript:alert(1)`,
- a link to `http://x/a)b`,
- a `pre/code` block holding a triple-backtick line, then `evil`, then
another triple-backtick line.
Before:
- the paragraph rendered as a heading with emphasis and a live img tag;
- the script link was kept;
- the second URL was split at `)`;
- the inner backticks closed the fence, so `evil` ended up outside the
code block.
After:
~~~text
\# not heading \*x\* \[img tag with backslash-escaped less-than, shown
as text]
click[p](http://x/a%29b)
````
```
evil
```
````
~~~
The new `tests/output_safety.rs` has 39 tests, 4 of them for math. 22 of
them failed on the old `main`. A pulldown-cmark (GFM) oracle checks four
things:
- no raw HTML,
- no unsafe destination,
- the expected block structure,
- verbatim text.
Proptests cover arbitrary text, URLs and code. #27's `limits` tests pass
in debug and release, including the deep-nesting tests under a small
stack.
`just pre-pr` passes locally except `cargo deny`, which is not installed
here; the CI Audit job runs it. New dev-dependency: `pulldown-cmark`
(MIT), used in tests only.
## Risk
- Medium.
- Output changes:
- Text with Markdown-significant characters is now backslash-escaped.
- Links and images with schemes outside the allowlist are dropped.
- Empty link targets are now written `<>`.
- Content nested under ordered list items is now indented 3 or more
spaces.
- Math now renders as `` $`tex`$ `` and ```` ```math ````.
- Tests updated to match:
- one golden fixture;
- the four math assertions from #34;
- #31's `javascript:` pass-through test, now
`absolute_dangerous_schemes_are_not_resolved`. Base resolution still
never rewrites the URL, and the allowlist then drops it.
## Checklist
- [x] Tests added or updated
- [x] Backward compatibility considered (CHANGELOG Unreleased entry)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
First extensibility step: callers can rewrite the IR after parsing and before rendering, for every format.
transform::Transformtrait (name,apply(&mut Document) -> Result<()>), registered withMarkkitBuilder::transform(..). Passes run in order forparse_html,parse_documentand everyextract_*path (HTML, DOCX, spreadsheets).NormalizeHeadings(shallowest heading becomes#),DropImages.transform::from_fn(name, closure)for one-offs.Document::visit/visit_mut,Node::for_each_child_list[_ref].Error::Transform;Erroris now#[non_exhaustive].examples/transforms.rs(wired intojust examplesand CI).Why
Roadmap "Extensibility": post-processing (heading normalization, link stripping, custom cleanup) previously meant re-parsing the Markdown. Working on the IR is cheaper and format-independent.
Before / After
Before: no hook; output fixed by the parser.
After (
cargo run -p markkit --example transforms, input starts at<h3>and has a link):No transforms configured: output unchanged.
Risk
Erroris#[non_exhaustive], so exhaustive matches need_.Checklist
parse_htmlandextract_document, DOCX, built-ins, doctest gated onhtml)Errorbreak)foundations/architecture.md,operations/examples.md, log), CHANGELOGGenerated by Claude Code