Repository navigation
feat(html): resolve relative URLs against a base URL - #31
Merged
Merged
Conversation
extract_document on HTML fills Extraction::title and new Extraction::metadata (description, author, published, language, canonical URL, site name, image). Values are cleaned and capped, URL fields drop non-http(s) schemes, JSON-LD reading is bounded. CLI --json includes format, title and metadata for files and stdin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CEs5BsitrunoTsYY5SYtFy
Links, images and metadata URLs resolve against MarkkitBuilder::base_url (CLI --base-url) with the page <base href> on top; extract_url uses the final response URL. Only scheme-less references change; page bases are limited to http(s). Adds Markkit::extract_url_document.
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
This was referenced Oct 27, 2026
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMDLeuybnBNqgVL9B9kMmARU)_
## What changed
Markdown output is now safe to render, for every input format. The PR is
rebased onto current `main`, so it covers IR v2 (footnotes, definition
lists, figures, task items, table spans), math (#34), transforms (#36),
relative URL resolution (#31) and the limits from #27. #27's output and
deadline checks are kept in the reworked renderer.
- **Text renders verbatim.** These characters are escaped:
- backtick, `*`, `_` (at word boundaries), `[`, `]`, the less-than sign,
`~`, `|` and entity-like `&`;
- line-start block markers: `#`, `>`, `-`, `+`, `=`, `1.`.
Document text can no longer produce raw HTML, links, emphasis, headings,
lists or tables.
- **URL scheme allowlist.** Links keep `http`, `https`, `mailto`, `tel`,
`ftp` and relative URLs. Images keep `http`, `https`, relative URLs and
raster `data:image/*`. Any other scheme is dropped and its link text or
alt text is kept: `javascript:`, `vbscript:`, `data:text/html`, `file:`.
Scheme detection ignores whitespace and control characters and reads the
whole scheme.
- **Destinations and titles.** Parentheses, angle brackets, backslash
and spaces are percent-encoded. Entity-like `&` is escaped, so
`javascript:` cannot decode into a scheme. Empty destinations are
written `<>`. Titles escape quotes, backslash and less-than.
- **Code.** Fences and spans are longer than any backtick run inside
them, and adjacent spans cannot merge their fences. The code-block
language keeps only its first word.
- **Math renders as code.** TeX comes from page markup, and plain GFM
has no math syntax, so `$tex$` let an annotation containing an img tag
or a link through. Math now uses GitHub's forms: `` $`tex`$ `` inline
and a ```` ```math ```` fence for display math. GitHub renders both as
math. Every other renderer shows them as inert code.
- **Structure stays in its container.**
- Block nodes inside inline content are flattened to one line. Examples:
a code block or display math in a table cell, a paragraph in a heading.
- List item bodies are indented by the marker width, so content nested
under `1.` items now nests correctly.
- Every block starts on its own line, and hard breaks never produce a
blank line.
- Nested links render as text.
- **Raw text in the IR.** The DOCX and spreadsheet parsers stop
pre-escaping table pipes, so the JSON IR holds raw text. The renderer
escapes pipes in table context.
All escaping lives in the new `render/escape.rs`, with the design
decisions in its header. The new invariant is S17, since #27 took S14 to
S16.
## Why
Threat model T1 promised "no HTML pass-through", but the renderer
escaped nothing. On `main`, a probe page showed four problems:
- it produced a raw img tag with an onerror handler;
- it kept `javascript:` links;
- a triple-backtick line inside a `pre` block closed the fence;
- links broke on `)`.
A fuzzing pass against the new oracle then found eight more ways content
could escape its container. The cargo-fuzz harness comes in a follow-up
PR. Each finding is a regression test here.
## Before / After
Input (HTML) is a paragraph containing `# not heading *x*`, followed by:
- an escaped img tag with an onerror handler,
- a link to `javascript:alert(1)`,
- a link to `http://x/a)b`,
- a `pre/code` block holding a triple-backtick line, then `evil`, then
another triple-backtick line.
Before:
- the paragraph rendered as a heading with emphasis and a live img tag;
- the script link was kept;
- the second URL was split at `)`;
- the inner backticks closed the fence, so `evil` ended up outside the
code block.
After:
~~~text
\# not heading \*x\* \[img tag with backslash-escaped less-than, shown
as text]
click[p](http://x/a%29b)
````
```
evil
```
````
~~~
The new `tests/output_safety.rs` has 39 tests, 4 of them for math. 22 of
them failed on the old `main`. A pulldown-cmark (GFM) oracle checks four
things:
- no raw HTML,
- no unsafe destination,
- the expected block structure,
- verbatim text.
Proptests cover arbitrary text, URLs and code. #27's `limits` tests pass
in debug and release, including the deep-nesting tests under a small
stack.
`just pre-pr` passes locally except `cargo deny`, which is not installed
here; the CI Audit job runs it. New dev-dependency: `pulldown-cmark`
(MIT), used in tests only.
## Risk
- Medium.
- Output changes:
- Text with Markdown-significant characters is now backslash-escaped.
- Links and images with schemes outside the allowlist are dropped.
- Empty link targets are now written `<>`.
- Content nested under ordered list items is now indented 3 or more
spaces.
- Math now renders as `` $`tex`$ `` and ```` ```math ````.
- Tests updated to match:
- one golden fixture;
- the four math assertions from #34;
- #31's `javascript:` pass-through test, now
`absolute_dangerous_schemes_are_not_resolved`. Base resolution still
never rewrites the URL, and the allowlist then drops it.
## Checklist
- [x] Tests added or updated
- [x] Backward compatibility considered (CHANGELOG Unreleased entry)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
…WASM (#40) <!-- claude-projects-attribution: {"github_login":"chaliy"} --> _Requested by **Михайло** · [project thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMGqDAMfmyQqFCQtq87yxmfS)_ ## What changed This is the follow-up to #27. Every `markkit::Limits` field can now be set outside Rust: - **CLI**: `markkit extract` gets a "Limits" help section with one flag per limit: `--max-input-size` (`--max-size` still works as an alias), `--max-output-size`, `--timeout` (seconds, fractions allowed), `--max-nesting-depth`, `--max-nodes`, the archive and sheet flags, and `--max-redirects`. - **Python**: new `markkit.Limits(...)` class, a frozen keyword-only constructor with readable attributes. Bad values raise `ValueError`/`OverflowError`. Every function accepts `limits=`. - **Node.js**: `MarkkitOptions` gets flat keys `timeout` (seconds, float, and it wins over `maxProcessingSecs`), `maxOutputSize`, `maxNodes` and `maxNestingDepth`. A negative or NaN `timeout` is rejected. - **WASM**: `Options` gets `maxOutputSize`, `maxNodes` and `maxNestingDepth`. There is no `timeout`, because wasm32 has no clock. Unknown keys throw, so passing `timeout` fails loudly instead of being ignored. - **Parity**: Node and WASM options also get `baseUrl` and `mainContent` (#31, #38), and Python `extract_html_sync_with_options` gets `base_url=` and `main_content=`, so the bindings match the core builder. The binding READMEs, the knowledge runtime docs and parity table, CHANGELOG and log are updated. ## Why The user asked for limits on every aspect. #27 added them to the core, and this PR makes them reachable from every runtime. ## Before / After Before: the CLI had only `--max-size`. Python had no limit arguments, and Node and WASM had no output, node or depth caps. After (CLI, table-amplification input that renders 12 MB of Markdown): ``` $ markkit extract --file amp.html --max-output-size 1000000 Error: Resource limit exceeded: output exceeds max_output_size 1000000 $ markkit extract --file amp.html --timeout 0 Error: Resource limit exceeded: processing exceeded timeout of 0ns ``` Python smoke run: ``` >>> markkit.extract_html_sync('<p>x</p>'*100, limits=markkit.Limits(max_output_size=10)) Resource limit exceeded: output exceeds max_output_size 10 # raised >>> markkit.Limits(timeout=2.5) Limits(max_input_size=52428800, max_output_size=134217728, timeout=2.5, max_nesting_depth=256, ...) ``` Test results: Node `options.test.cjs` passes 9 of 9, WASM tests pass 19 of 19, and CLI unit and integration tests pass. ## Risk - Low. All additions are optional, and defaults are unchanged. `--max-size` is kept. ## Checklist - [x] Tests added or updated - [x] Backward compatibility considered 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
canonical_url,image) now resolve against a base URL.MarkkitBuilder::base_url(..)/ CLI--base-url;extract_urluses the final response URL after redirects. A page<base href>applies on top, as in browsers.Markkit::extract_url_documentreturns the fullExtraction(title, metadata) for a URL; CLI--jsonon URLs now includesformat,title,metadata.#fragmentreferences are left byte-for-byte.Second of three PRs for the roadmap's "Readability and metadata" item (after #20); main-content detection is next.
Why
Extracted Markdown from web pages had dead
/pathandimg.pngreferences once it left the page's origin.Before / After
Before:
[b](../b) [i](i.png)(and--base-urldidn't exist)After:
[b](https://ex.com/a/b) [i](https://ex.com/a/c/i.png)Risk
extract_urloutput changes (relative URLs become absolute), which is the intent.javascript:/data:are never normalized or produced by resolution; a page<base href>is accepted only when it resolves tohttp(s)(nofile:rebasing); caller bases that can't be a base (data:,javascript:) are ignored. New deps:url(already in the tree via reqwest; now also under thehtmlfeature).cargo denyclean.Checklist
tests/html_urls.rsincl. security cases,url_security::relative_urls_resolve_against_final_url_after_redirect, unit tests)Generated by Claude Code