Repository navigation
feat(html): Readability-style main-content detection - #38
Merged
Merged
Conversation
extract_document on HTML fills Extraction::title and new Extraction::metadata (description, author, published, language, canonical URL, site name, image). Values are cleaned and capped, URL fields drop non-http(s) schemes, JSON-LD reading is bounded. CLI --json includes format, title and metadata for files and stdin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CEs5BsitrunoTsYY5SYtFy
Links, images and metadata URLs resolve against MarkkitBuilder::base_url (CLI --base-url) with the page <base href> on top; extract_url uses the final response URL. Only scheme-less references change; page bases are limited to http(s). Adds Markkit::extract_url_document.
2 tasks done
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
…WASM (#40) <!-- claude-projects-attribution: {"github_login":"chaliy"} --> _Requested by **Михайло** · [project thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMGqDAMfmyQqFCQtq87yxmfS)_ ## What changed This is the follow-up to #27. Every `markkit::Limits` field can now be set outside Rust: - **CLI**: `markkit extract` gets a "Limits" help section with one flag per limit: `--max-input-size` (`--max-size` still works as an alias), `--max-output-size`, `--timeout` (seconds, fractions allowed), `--max-nesting-depth`, `--max-nodes`, the archive and sheet flags, and `--max-redirects`. - **Python**: new `markkit.Limits(...)` class, a frozen keyword-only constructor with readable attributes. Bad values raise `ValueError`/`OverflowError`. Every function accepts `limits=`. - **Node.js**: `MarkkitOptions` gets flat keys `timeout` (seconds, float, and it wins over `maxProcessingSecs`), `maxOutputSize`, `maxNodes` and `maxNestingDepth`. A negative or NaN `timeout` is rejected. - **WASM**: `Options` gets `maxOutputSize`, `maxNodes` and `maxNestingDepth`. There is no `timeout`, because wasm32 has no clock. Unknown keys throw, so passing `timeout` fails loudly instead of being ignored. - **Parity**: Node and WASM options also get `baseUrl` and `mainContent` (#31, #38), and Python `extract_html_sync_with_options` gets `base_url=` and `main_content=`, so the bindings match the core builder. The binding READMEs, the knowledge runtime docs and parity table, CHANGELOG and log are updated. ## Why The user asked for limits on every aspect. #27 added them to the core, and this PR makes them reachable from every runtime. ## Before / After Before: the CLI had only `--max-size`. Python had no limit arguments, and Node and WASM had no output, node or depth caps. After (CLI, table-amplification input that renders 12 MB of Markdown): ``` $ markkit extract --file amp.html --max-output-size 1000000 Error: Resource limit exceeded: output exceeds max_output_size 1000000 $ markkit extract --file amp.html --timeout 0 Error: Resource limit exceeded: processing exceeded timeout of 0ns ``` Python smoke run: ``` >>> markkit.extract_html_sync('<p>x</p>'*100, limits=markkit.Limits(max_output_size=10)) Resource limit exceeded: output exceeds max_output_size 10 # raised >>> markkit.Limits(timeout=2.5) Limits(max_input_size=52428800, max_output_size=134217728, timeout=2.5, max_nesting_depth=256, ...) ``` Test results: Node `options.test.cjs` passes 9 of 9, WASM tests pass 19 of 19, and CLI unit and integration tests pass. ## Risk - Low. All additions are optional, and defaults are unchanged. `--max-size` is kept. ## Checklist - [x] Tests added or updated - [x] Backward compatibility considered 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
MarkkitBuilder::main_content(true)/ CLI--main-content: keeps only the article body of a web page and drops top bars, banners, sidebars, comments, share bars and related-link blocks.html/readability.rs): paragraph text length and commas credited up three ancestor levels, tag and class/id weights, link-density scaling, related siblings kept. Algorithm documented inknowledge/formats/html-extraction.md.track_sources) and with the base URL / metadata from feat(html): extract page metadata from meta, Open Graph and JSON-LD #20 and feat(html): resolve relative URLs against a base URL #31.Last of three PRs for the roadmap's "Readability and metadata" item.
Why
Web pages leaked nav, ads and comments into the Markdown unless the caller wrote CSS selectors per site.
Before / After
tests/fixtures/html/news_page.html(a news page with a top bar, banner, sidebar, share bar, comments and related links):Before (
markkit extract --file news_page.html):After (
--main-content): only the article, exactlytests/fixtures/html/news_page.main.md:The full release notes, including a migration guide, are available on the project blog.