Repository navigation
feat(limits): unified Limits with timeout, output, node and depth caps - #27
Merged
Merged
Conversation
Move every resource limit into one markkit::Limits struct (bashkit ExecutionLimits style) on MarkkitBuilder. New limits, generous defaults: - timeout (60s) now bounds parsing and rendering, not only URL fetch - max_output_size (128 MiB), checked per block/row; stops table amplification - max_nodes (5M) for HTML DOM and DOCX XML nodes - max_nesting_depth (256) configurable, hard ceiling 1024 for stack safety Breaking: limit fields moved to builder.limits; max_processing_secs field is now limits.timeout (setter kept).
2 tasks done
…-mx19ge # Conflicts: # CHANGELOG.md # crates/markkit/src/extract.rs
…-mx19ge # Conflicts: # crates/markkit/src/config.rs # crates/markkit/src/extract.rs # crates/markkit/src/html/parser.rs
…-mx19ge # Conflicts: # CHANGELOG.md # crates/markkit/src/extract.rs # crates/markkit/src/spreadsheet.rs
…-mx19ge # Conflicts: # knowledge/security/threat-model.md
2 tasks done
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
2 tasks done
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMDLeuybnBNqgVL9B9kMmARU)_
## What changed
Markdown output is now safe to render, for every input format. The PR is
rebased onto current `main`, so it covers IR v2 (footnotes, definition
lists, figures, task items, table spans), math (#34), transforms (#36),
relative URL resolution (#31) and the limits from #27. #27's output and
deadline checks are kept in the reworked renderer.
- **Text renders verbatim.** These characters are escaped:
- backtick, `*`, `_` (at word boundaries), `[`, `]`, the less-than sign,
`~`, `|` and entity-like `&`;
- line-start block markers: `#`, `>`, `-`, `+`, `=`, `1.`.
Document text can no longer produce raw HTML, links, emphasis, headings,
lists or tables.
- **URL scheme allowlist.** Links keep `http`, `https`, `mailto`, `tel`,
`ftp` and relative URLs. Images keep `http`, `https`, relative URLs and
raster `data:image/*`. Any other scheme is dropped and its link text or
alt text is kept: `javascript:`, `vbscript:`, `data:text/html`, `file:`.
Scheme detection ignores whitespace and control characters and reads the
whole scheme.
- **Destinations and titles.** Parentheses, angle brackets, backslash
and spaces are percent-encoded. Entity-like `&` is escaped, so
`javascript:` cannot decode into a scheme. Empty destinations are
written `<>`. Titles escape quotes, backslash and less-than.
- **Code.** Fences and spans are longer than any backtick run inside
them, and adjacent spans cannot merge their fences. The code-block
language keeps only its first word.
- **Math renders as code.** TeX comes from page markup, and plain GFM
has no math syntax, so `$tex$` let an annotation containing an img tag
or a link through. Math now uses GitHub's forms: `` $`tex`$ `` inline
and a ```` ```math ```` fence for display math. GitHub renders both as
math. Every other renderer shows them as inert code.
- **Structure stays in its container.**
- Block nodes inside inline content are flattened to one line. Examples:
a code block or display math in a table cell, a paragraph in a heading.
- List item bodies are indented by the marker width, so content nested
under `1.` items now nests correctly.
- Every block starts on its own line, and hard breaks never produce a
blank line.
- Nested links render as text.
- **Raw text in the IR.** The DOCX and spreadsheet parsers stop
pre-escaping table pipes, so the JSON IR holds raw text. The renderer
escapes pipes in table context.
All escaping lives in the new `render/escape.rs`, with the design
decisions in its header. The new invariant is S17, since #27 took S14 to
S16.
## Why
Threat model T1 promised "no HTML pass-through", but the renderer
escaped nothing. On `main`, a probe page showed four problems:
- it produced a raw img tag with an onerror handler;
- it kept `javascript:` links;
- a triple-backtick line inside a `pre` block closed the fence;
- links broke on `)`.
A fuzzing pass against the new oracle then found eight more ways content
could escape its container. The cargo-fuzz harness comes in a follow-up
PR. Each finding is a regression test here.
## Before / After
Input (HTML) is a paragraph containing `# not heading *x*`, followed by:
- an escaped img tag with an onerror handler,
- a link to `javascript:alert(1)`,
- a link to `http://x/a)b`,
- a `pre/code` block holding a triple-backtick line, then `evil`, then
another triple-backtick line.
Before:
- the paragraph rendered as a heading with emphasis and a live img tag;
- the script link was kept;
- the second URL was split at `)`;
- the inner backticks closed the fence, so `evil` ended up outside the
code block.
After:
~~~text
\# not heading \*x\* \[img tag with backslash-escaped less-than, shown
as text]
click[p](http://x/a%29b)
````
```
evil
```
````
~~~
The new `tests/output_safety.rs` has 39 tests, 4 of them for math. 22 of
them failed on the old `main`. A pulldown-cmark (GFM) oracle checks four
things:
- no raw HTML,
- no unsafe destination,
- the expected block structure,
- verbatim text.
Proptests cover arbitrary text, URLs and code. #27's `limits` tests pass
in debug and release, including the deep-nesting tests under a small
stack.
`just pre-pr` passes locally except `cargo deny`, which is not installed
here; the CI Audit job runs it. New dev-dependency: `pulldown-cmark`
(MIT), used in tests only.
## Risk
- Medium.
- Output changes:
- Text with Markdown-significant characters is now backslash-escaped.
- Links and images with schemes outside the allowlist are dropped.
- Empty link targets are now written `<>`.
- Content nested under ordered list items is now indented 3 or more
spaces.
- Math now renders as `` $`tex`$ `` and ```` ```math ````.
- Tests updated to match:
- one golden fixture;
- the four math assertions from #34;
- #31's `javascript:` pass-through test, now
`absolute_dangerous_schemes_are_not_resolved`. Base resolution still
never rewrites the URL, and the allowlist then drops it.
## Checklist
- [x] Tests added or updated
- [x] Backward compatibility considered (CHANGELOG Unreleased entry)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
…WASM (#40) <!-- claude-projects-attribution: {"github_login":"chaliy"} --> _Requested by **Михайло** · [project thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMGqDAMfmyQqFCQtq87yxmfS)_ ## What changed This is the follow-up to #27. Every `markkit::Limits` field can now be set outside Rust: - **CLI**: `markkit extract` gets a "Limits" help section with one flag per limit: `--max-input-size` (`--max-size` still works as an alias), `--max-output-size`, `--timeout` (seconds, fractions allowed), `--max-nesting-depth`, `--max-nodes`, the archive and sheet flags, and `--max-redirects`. - **Python**: new `markkit.Limits(...)` class, a frozen keyword-only constructor with readable attributes. Bad values raise `ValueError`/`OverflowError`. Every function accepts `limits=`. - **Node.js**: `MarkkitOptions` gets flat keys `timeout` (seconds, float, and it wins over `maxProcessingSecs`), `maxOutputSize`, `maxNodes` and `maxNestingDepth`. A negative or NaN `timeout` is rejected. - **WASM**: `Options` gets `maxOutputSize`, `maxNodes` and `maxNestingDepth`. There is no `timeout`, because wasm32 has no clock. Unknown keys throw, so passing `timeout` fails loudly instead of being ignored. - **Parity**: Node and WASM options also get `baseUrl` and `mainContent` (#31, #38), and Python `extract_html_sync_with_options` gets `base_url=` and `main_content=`, so the bindings match the core builder. The binding READMEs, the knowledge runtime docs and parity table, CHANGELOG and log are updated. ## Why The user asked for limits on every aspect. #27 added them to the core, and this PR makes them reachable from every runtime. ## Before / After Before: the CLI had only `--max-size`. Python had no limit arguments, and Node and WASM had no output, node or depth caps. After (CLI, table-amplification input that renders 12 MB of Markdown): ``` $ markkit extract --file amp.html --max-output-size 1000000 Error: Resource limit exceeded: output exceeds max_output_size 1000000 $ markkit extract --file amp.html --timeout 0 Error: Resource limit exceeded: processing exceeded timeout of 0ns ``` Python smoke run: ``` >>> markkit.extract_html_sync('<p>x</p>'*100, limits=markkit.Limits(max_output_size=10)) Resource limit exceeded: output exceeds max_output_size 10 # raised >>> markkit.Limits(timeout=2.5) Limits(max_input_size=52428800, max_output_size=134217728, timeout=2.5, max_nesting_depth=256, ...) ``` Test results: Node `options.test.cjs` passes 9 of 9, WASM tests pass 19 of 19, and CLI unit and integration tests pass. ## Risk - Low. All additions are optional, and defaults are unchanged. `--max-size` is kept. ## Checklist - [x] Tests added or updated - [x] Backward compatibility considered 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
Every resource an input can drive is now bounded by one
markkit::Limitsstruct (bashkitExecutionLimitsstyle). It is set as a whole withMarkkitBuilder::limits(..), or per field with the existing and new builder setters. Any breach returnsError::ResourceLimit, which names the limit. Defaults are generous.New limits:
timeout(60 s) is one wall-clock budget per call. It covers URL fetch, parsing and rendering, and it applies toextract_*andparse_document*alike. Before this,max_processing_secsbounded only the fetch. The deadline is checked cooperatively in the HTML walk, XML tree build, DOCX conversion, spreadsheet cell loops and the renderer. html5ever and calamine steps can't be interrupted, so the deadline is checked right after each one finishes.max_output_size(128 MiB) is checked per block, list item, definition item and table row, including inside nested renderers. This stops table amplification, where one wide row pads every other row and output grows as rows x cols.max_nodes(5 000 000) counts HTML DOM nodes, and DOCX XML nodes summed over all parts.max_nesting_depth(256) replaces the hardcoded HTMLMAX_DEPTH=200and XMLMAX_XML_DEPTH=256. It is clamped toHARD_MAX_NESTING_DEPTH= 512, so no setting can overflow the stack. 512 was measured against a 2 MiB release stack and leaves about 2x headroom.The existing archive, spreadsheet, input and redirect limits moved into
Limitsunchanged.Knowledge updates: threat model T2 (new T2g output amplification), the resource-limit table and invariants S14 to S16; the architecture, IR, HTML extraction and testing docs; a README limits table; and CHANGELOG. Main is merged in (IR v2, provenance/
parse_document, page metadata, MarkkitTool), and its new paths use the same budget.Exposing these limits in the CLI and the Python/Node bindings follows in a separate PR.
Why
The user asked for limits on every aspect, with big defaults. Output size had no cap, and the time limit was not enforced during parsing.
Before / After
Before: a
<table>with one 2000-cell row plus 2000 one-cell rows rendered about 16 MB of Markdown with no bound, andmax_processing_secshad no effect on parsing.After:
tests/limits.rshas 16 tests. They cover each limit for HTML, DOCX and spreadsheets, plus blockquote, list, div and inline nesting 4096 levels deep under a small thread stack (2 MiB in release, 8 MiB in debug). They pass in both debug and release.Risk
MarkkitBuildertoMarkkitBuilder::limits(config().max_input_sizebecomesconfig().limits.max_input_size, and themax_processing_secsfield becomeslimits.timeout: Duration). The builder setters are unchanged.Checklist
🤖 Generated with Claude Code