From d2131b38e117714483afe30b1a4e94274f9fd6ca Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Mon, 26 Oct 2026 23:42:45 +0000 Subject: [PATCH 1/5] feat(limits): unified Limits with timeout, output, node and depth caps Move every resource limit into one markkit::Limits struct (bashkit ExecutionLimits style) on MarkkitBuilder. New limits, generous defaults: - timeout (60s) now bounds parsing and rendering, not only URL fetch - max_output_size (128 MiB), checked per block/row; stops table amplification - max_nodes (5M) for HTML DOM and DOCX XML nodes - max_nesting_depth (256) configurable, hard ceiling 1024 for stack safety Breaking: limit fields moved to builder.limits; max_processing_secs field is now limits.timeout (setter kept). --- CHANGELOG.md | 25 ++ README.md | 36 ++- crates/markkit/src/archive.rs | 8 +- crates/markkit/src/config.rs | 82 +++--- crates/markkit/src/docx.rs | 52 +++- crates/markkit/src/extract.rs | 109 ++++---- crates/markkit/src/fetch.rs | 16 +- crates/markkit/src/html/parser.rs | 54 +++- crates/markkit/src/lib.rs | 2 + crates/markkit/src/limits.rs | 317 +++++++++++++++++++++++ crates/markkit/src/render/markdown.rs | 94 ++++++- crates/markkit/src/spreadsheet.rs | 40 ++- crates/markkit/src/xml.rs | 64 ++++- crates/markkit/tests/html_integration.rs | 5 +- crates/markkit/tests/limits.rs | 269 +++++++++++++++++++ examples/configuration.rs | 13 + knowledge/formats/html-extraction.md | 2 +- knowledge/foundations/architecture.md | 6 +- knowledge/foundations/ir.md | 2 +- knowledge/log.md | 1 + knowledge/operations/testing.md | 1 + knowledge/security/security-model.md | 2 +- knowledge/security/threat-model.md | 48 +++- 23 files changed, 1088 insertions(+), 160 deletions(-) create mode 100644 crates/markkit/src/limits.rs create mode 100644 crates/markkit/tests/limits.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b78190..a1297a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Security +- Resource limits on every aspect, in one `markkit::Limits` struct (`MarkkitBuilder::limits(..)` or + per-field setters). New: `timeout` (default 60 s) now bounds parsing and rendering, not just URL + fetch; `max_output_size` (128 MiB) stops output amplification (e.g. one wide table row padding + thousands of rows); `max_nodes` (5 000 000) caps parsed HTML DOM / DOCX XML nodes; + `max_nesting_depth` (256) is configurable, with a hard ceiling of 1024 that keeps deep input off + the host stack. Violations return `Error::ResourceLimit` naming the limit. + - `extract_url` (CLI, Python and Node `extract_url` too) now refuses non-public destinations by default: loopback, private, link-local / cloud metadata and other reserved addresses, checked on the URL, after DNS resolution and on every redirect. Only `http`/`https` are fetched. Redirects are capped @@ -18,6 +25,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` + (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is + now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, + `max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth + 256 (was 200). + - Design docs moved from `specs/` to an OKF v0.2 knowledge bundle in `knowledge/`, validated in CI. - CI workflows declare read-only default permissions and do not persist checkout credentials. - CI runs every example; `just bench` now runs the criterion benchmarks. @@ -48,6 +61,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` + (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is + now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, + `max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth + 256 (was 200). + - Release automation: merging `chore(release): prepare vX.Y.Z` to main tags, creates the GitHub Release and publishes to crates.io (idempotent); manual dispatch from main also works @@ -94,6 +113,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` + (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is + now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, + `max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth + 256 (was 200). + - `docx` feature now uses `zip` + `quick-xml` directly; `docx-rs` dependency removed - `calamine` upgraded to 0.36 (pulls quick-xml 0.41, clearing RUSTSEC-2026-0194/0195); `zip` upgraded to 8 (deflate only), shared with calamine - PDF extraction returns `Error::UnsupportedFormat` instead of an empty string diff --git a/README.md b/README.md index 0d9e55a..9faf2e0 100644 --- a/README.md +++ b/README.md @@ -166,10 +166,36 @@ markkit = { version = "0.1", default-features = false, features = ["docx", "spre # add "html" if you also convert HTML ``` -Office-document limits are configurable on the builder: `max_archive_entries`, -`max_archive_entry_size`, `max_archive_total_size`, `max_spreadsheet_cells`, -`max_sheet_rows`, `max_sheet_cols`, `include_hidden_sheets`. URL fetching: -`allow_private_networks`, `allow_host`, `max_redirects`. +### Resource limits + +Every resource an input can drive is bounded by `markkit::Limits`. Defaults are +generous; violations return `Error::ResourceLimit` naming the limit. + +| Limit | Default | Bounds | +|-------|---------|--------| +| `max_input_size` | 50 MiB | Input bytes, fetched URL body | +| `max_output_size` | 128 MiB | Markdown output | +| `timeout` | 60 s | Fetch + parse + render, per call | +| `max_nesting_depth` | 256 (hard max 1024) | HTML flattens deeper content; deeper DOCX XML is rejected | +| `max_nodes` | 5 000 000 | HTML DOM nodes, DOCX XML nodes | +| `max_archive_entries` | 10 000 | ZIP entries (DOCX/XLSX/ODS) | +| `max_archive_entry_size` | 64 MiB | Decompressed bytes per ZIP entry | +| `max_archive_total_size` | 256 MiB | Decompressed bytes per ZIP | +| `max_spreadsheet_cells` | 4 000 000 | XLS/ODS grid before parsing | +| `max_sheet_rows` / `max_sheet_cols` | 10 000 / 256 | Rendered per sheet (truncated with a warning) | +| `max_redirects` | 5 | `extract_url` redirects | + +```rust +use markkit::{Limits, Markkit}; +use std::time::Duration; + +let markkit = Markkit::builder() + .limits(Limits::default().timeout(Duration::from_secs(10)).max_output_size(8 << 20)) + .build(); +``` + +Each limit also has a builder setter (`.max_nodes(..)`, `.timeout(..)`, ...). +Other options: `include_hidden_sheets`, `allow_private_networks`, `allow_host`. ## Security @@ -180,7 +206,7 @@ XML entities are never expanded; macros and formulas are never executed; hidden sheets are skipped by default. `extract_url` accepts only http/https, refuses loopback, private, link-local and cloud-metadata addresses (also after DNS resolution and on every redirect), caps redirects, and bounds the body by -`max_input_size` and the fetch by `max_processing_secs`. Opt in to internal +`max_input_size` and the fetch by `timeout`. Opt in to internal hosts with `allow_private_networks(true)`; restrict hosts with `allow_host(..)`. See the [threat model](knowledge/security/threat-model.md) and the diff --git a/crates/markkit/src/archive.rs b/crates/markkit/src/archive.rs index 5e7a85d..3166a02 100644 --- a/crates/markkit/src/archive.rs +++ b/crates/markkit/src/archive.rs @@ -41,11 +41,11 @@ impl<'a> ZipGuard<'a> { /// Open the archive and check the entry count. pub fn open(bytes: &'a [u8], config: &MarkkitBuilder) -> Result { let archive = ZipArchive::new(Cursor::new(bytes))?; - if archive.len() > config.max_archive_entries { + if archive.len() > config.limits.max_archive_entries { return Err(ArchiveError::Limit(format!( "archive has {} entries, limit is {}", archive.len(), - config.max_archive_entries + config.limits.max_archive_entries ))); } let names = archive @@ -55,8 +55,8 @@ impl<'a> ZipGuard<'a> { Ok(Self { archive, names, - max_entry: config.max_archive_entry_size, - max_total: config.max_archive_total_size, + max_entry: config.limits.max_archive_entry_size, + max_total: config.limits.max_archive_total_size, total: 0, }) } diff --git a/crates/markkit/src/config.rs b/crates/markkit/src/config.rs index 9f86747..cb5fa0d 100644 --- a/crates/markkit/src/config.rs +++ b/crates/markkit/src/config.rs @@ -1,12 +1,14 @@ +use std::time::Duration; + use crate::extract::Markkit; +use crate::limits::Limits; /// Builder for configuring a Markkit instance. #[derive(Debug, Clone)] pub struct MarkkitBuilder { - /// Maximum input size in bytes (default: 50MB). - pub max_input_size: usize, - /// Maximum processing time in seconds (default: 60). - pub max_processing_secs: u64, + /// Resource limits (input/output size, time, nesting, nodes, archives, + /// spreadsheets, redirects). See [`Limits`]. + pub limits: Limits, /// Whether to preserve link URLs in output. pub preserve_links: bool, /// Whether to preserve image references in output. @@ -17,19 +19,6 @@ pub struct MarkkitBuilder { pub include_selectors: Vec, /// CSS selectors for content to exclude. pub exclude_selectors: Vec, - /// Maximum number of entries in a ZIP container (DOCX/XLSX/ODS). Default: 10 000. - pub max_archive_entries: usize, - /// Maximum decompressed size of a single ZIP entry in bytes. Default: 64 MiB. - pub max_archive_entry_size: u64, - /// Maximum total decompressed size of a ZIP container in bytes. Default: 256 MiB. - pub max_archive_total_size: u64, - /// Maximum data rows rendered per sheet (header excluded). Default: 10 000. - pub max_sheet_rows: usize, - /// Maximum columns rendered per sheet. Default: 256. - pub max_sheet_cols: usize, - /// Maximum cell grid a workbook may expand to while parsing (XLS/ODS - /// pre-scan, summed over sheets). Default: 4 000 000. - pub max_spreadsheet_cells: u64, /// Render hidden / very hidden sheets. Default: false (skipped with a warning). pub include_hidden_sheets: bool, /// Let `extract_url` reach loopback, private, link-local and other @@ -38,15 +27,12 @@ pub struct MarkkitBuilder { /// Hosts `extract_url` may contact, matched exactly and case-insensitively /// on every hop (empty = any public host). pub allowed_hosts: Vec, - /// Maximum redirects `extract_url` follows. Default: 5. - pub max_redirects: usize, } impl Default for MarkkitBuilder { fn default() -> Self { Self { - max_input_size: 50 * 1024 * 1024, - max_processing_secs: 60, + limits: Limits::default(), preserve_links: true, preserve_images: true, extract_code_blocks: true, @@ -59,28 +45,50 @@ impl Default for MarkkitBuilder { "style".into(), "noscript".into(), ], - max_archive_entries: 10_000, - max_archive_entry_size: 64 * 1024 * 1024, - max_archive_total_size: 256 * 1024 * 1024, - max_sheet_rows: 10_000, - max_sheet_cols: 256, - max_spreadsheet_cells: 4_000_000, include_hidden_sheets: false, allow_private_networks: false, allowed_hosts: Vec::new(), - max_redirects: 5, } } } impl MarkkitBuilder { pub fn max_input_size(mut self, size: usize) -> Self { - self.max_input_size = size; + self.limits.max_input_size = size; + self + } + + /// Replace all limits at once. + pub fn limits(mut self, limits: Limits) -> Self { + self.limits = limits; self } + /// Wall-clock budget per call (fetch + parse + render). + pub fn timeout(mut self, timeout: Duration) -> Self { + self.limits.timeout = timeout; + self + } + + /// [`MarkkitBuilder::timeout`] in whole seconds. pub fn max_processing_secs(mut self, secs: u64) -> Self { - self.max_processing_secs = secs; + self.limits.timeout = Duration::from_secs(secs); + self + } + + pub fn max_output_size(mut self, bytes: usize) -> Self { + self.limits.max_output_size = bytes; + self + } + + /// Clamped to [`crate::limits::HARD_MAX_NESTING_DEPTH`]. + pub fn max_nesting_depth(mut self, depth: usize) -> Self { + self.limits.max_nesting_depth = depth; + self + } + + pub fn max_nodes(mut self, nodes: usize) -> Self { + self.limits.max_nodes = nodes; self } @@ -105,32 +113,32 @@ impl MarkkitBuilder { } pub fn max_archive_entries(mut self, n: usize) -> Self { - self.max_archive_entries = n; + self.limits.max_archive_entries = n; self } pub fn max_archive_entry_size(mut self, bytes: u64) -> Self { - self.max_archive_entry_size = bytes; + self.limits.max_archive_entry_size = bytes; self } pub fn max_archive_total_size(mut self, bytes: u64) -> Self { - self.max_archive_total_size = bytes; + self.limits.max_archive_total_size = bytes; self } pub fn max_sheet_rows(mut self, rows: usize) -> Self { - self.max_sheet_rows = rows; + self.limits.max_sheet_rows = rows; self } pub fn max_sheet_cols(mut self, cols: usize) -> Self { - self.max_sheet_cols = cols; + self.limits.max_sheet_cols = cols; self } pub fn max_spreadsheet_cells(mut self, cells: u64) -> Self { - self.max_spreadsheet_cells = cells; + self.limits.max_spreadsheet_cells = cells; self } @@ -153,7 +161,7 @@ impl MarkkitBuilder { } pub fn max_redirects(mut self, n: usize) -> Self { - self.max_redirects = n; + self.limits.max_redirects = n; self } diff --git a/crates/markkit/src/docx.rs b/crates/markkit/src/docx.rs index cf2795f..670dd7d 100644 --- a/crates/markkit/src/docx.rs +++ b/crates/markkit/src/docx.rs @@ -24,6 +24,7 @@ use crate::archive::{ArchiveError, ZipGuard}; use crate::config::MarkkitBuilder; use crate::error::Error; use crate::ir::{Document, ListItem, Node}; +use crate::limits::Budget; use crate::render::escape_table_cell; use crate::xml::{self, Element, XmlError}; @@ -51,10 +52,21 @@ fn xml_err(part: &str, e: XmlError) -> Error { } } -fn read_xml(zip: &mut ZipGuard<'_>, path: &str) -> Result, Error> { - match zip.read(path).map_err(archive_err)? { - Some(bytes) => xml::parse(&bytes).map(Some).map_err(|e| xml_err(path, e)), - None => Ok(None), +/// Bounded XML reader: archive limits, nesting depth, node count and deadline. +struct Parts<'z, 'b> { + zip: ZipGuard<'z>, + max_depth: usize, + budget: &'b Budget, +} + +impl Parts<'_, '_> { + fn read_xml(&mut self, path: &str) -> Result, Error> { + match self.zip.read(path).map_err(archive_err)? { + Some(bytes) => xml::parse(&bytes, self.max_depth, self.budget) + .map(Some) + .map_err(|e| xml_err(path, e)), + None => Ok(None), + } } } @@ -365,6 +377,7 @@ struct Ctx<'a> { rels: &'a HashMap, styles: &'a Styles, numbering: &'a Numbering, + budget: &'a Budget, } impl Ctx<'_> { @@ -665,6 +678,9 @@ fn build_list(entries: &[ListEntry], pos: &mut usize, level: u32) -> Node { impl Ctx<'_> { fn blocks(&self, container: &Element, body: &mut Body) { for child in container.elements() { + if self.budget.expired() { + return; + } match child.name.as_str() { "p" => self.paragraph(child, body), "tbl" => { @@ -863,10 +879,18 @@ fn table_cell(nodes: Vec) -> Vec { // ─── Entry point ──────────────────────────────────────────────────── -pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result { - let mut zip = ZipGuard::open(bytes, config).map_err(archive_err)?; +pub(crate) fn parse( + bytes: &[u8], + config: &MarkkitBuilder, + budget: &Budget, +) -> Result { + let mut zip = Parts { + zip: ZipGuard::open(bytes, config).map_err(archive_err)?, + max_depth: config.limits.nesting_depth(), + budget, + }; - let root_rels = parse_rels(read_xml(&mut zip, "_rels/.rels")?); + let root_rels = parse_rels(zip.read_xml("_rels/.rels")?); let main_path = rel_target_by_type(&root_rels, "/officeDocument") .map(|t| resolve_target("", t)) .unwrap_or_else(|| "word/document.xml".to_string()); @@ -874,10 +898,11 @@ pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result Result Result crate::Result { - if html.len() > self.config.max_input_size { - return Err(Error::ResourceLimit(format!( - "Input size {} exceeds limit {}", - html.len(), - self.config.max_input_size - ))); - } + self.extract_html_with(html, &Budget::new(&self.config.limits)) + } + + fn extract_html_with(&self, html: &str, budget: &Budget) -> crate::Result { + self.config.limits.check_input(html.len())?; #[cfg(feature = "html")] { - let parser = crate::html::HtmlParser::new(&self.config); + let parser = crate::html::HtmlParser::with_budget(&self.config, budget); let doc = parser.parse(html)?; - let mut renderer = crate::render::MarkdownRenderer::new(); - Ok(renderer.render(&doc)) + self.render(&doc, budget) } #[cfg(not(feature = "html"))] - Err(Error::UnsupportedFormat( - "HTML support requires the 'html' feature".to_string(), - )) + { + let _ = budget; + Err(Error::UnsupportedFormat( + "HTML support requires the 'html' feature".to_string(), + )) + } } /// Parse HTML into the IR without rendering. #[cfg(feature = "html")] pub fn parse_html(&self, html: &str) -> crate::Result { - if html.len() > self.config.max_input_size { - return Err(Error::ResourceLimit(format!( - "Input size {} exceeds limit {}", - html.len(), - self.config.max_input_size - ))); - } - let parser = crate::html::HtmlParser::new(&self.config); - parser.parse(html) + self.config.limits.check_input(html.len())?; + let budget = Budget::new(&self.config.limits); + crate::html::HtmlParser::with_budget(&self.config, &budget).parse(html) + } + + #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + fn render(&self, doc: &crate::ir::Document, budget: &Budget) -> crate::Result { + crate::render::MarkdownRenderer::new().render_bounded( + doc, + self.config.limits.max_output_size, + budget, + ) } /// Extract markdown from raw bytes, auto-detecting the format. @@ -157,7 +163,7 @@ impl Markkit { /// Synchronous [`Markkit::extract_document`]. /// - /// Errors: [`Error::ResourceLimit`] when input/archive/grid limits are + /// Errors: [`Error::ResourceLimit`] when any [`Limits`](crate::Limits) value is /// exceeded, [`Error::UnsupportedFormat`] for unknown formats or formats whose /// feature is disabled, and a format-specific variant for malformed input. pub fn extract_document_sync( @@ -165,23 +171,26 @@ impl Markkit { bytes: &[u8], hint: DocumentHint<'_>, ) -> crate::Result { - if bytes.len() > self.config.max_input_size { - return Err(Error::ResourceLimit(format!( - "Input size {} exceeds limit {}", - bytes.len(), - self.config.max_input_size - ))); - } + self.extract_document_with(bytes, hint, &Budget::new(&self.config.limits)) + } + + fn extract_document_with( + &self, + bytes: &[u8], + hint: DocumentHint<'_>, + budget: &Budget, + ) -> crate::Result { + self.config.limits.check_input(bytes.len())?; match Format::detect(bytes, hint) { Format::Html => { let text = String::from_utf8_lossy(bytes); - let md = self.extract_html_sync(&text)?; + let md = self.extract_html_with(&text, budget)?; Ok(Extraction::new(md, Format::Html)) } Format::Pdf => self.extract_pdf_sync(bytes), - Format::Docx => self.extract_docx(bytes), - Format::Xlsx | Format::Xls | Format::Ods => self.extract_spreadsheet(bytes), + Format::Docx => self.extract_docx(bytes, budget), + Format::Xlsx | Format::Xls | Format::Ods => self.extract_spreadsheet(bytes, budget), Format::Doc => Err(Error::UnsupportedFormat( "legacy Word 97-2003 (.doc) documents are not supported".into(), )), @@ -196,12 +205,11 @@ impl Markkit { } #[cfg(feature = "docx")] - fn extract_docx(&self, bytes: &[u8]) -> crate::Result { + fn extract_docx(&self, bytes: &[u8], budget: &Budget) -> crate::Result { let out = guarded("DOCX", Error::DocxExtract, || { - crate::docx::parse(bytes, &self.config) + crate::docx::parse(bytes, &self.config, budget) })?; - let mut renderer = crate::render::MarkdownRenderer::new(); - let mut e = Extraction::new(renderer.render(&out.doc), Format::Docx); + let mut e = Extraction::new(self.render(&out.doc, budget)?, Format::Docx); e.title = out.title; e.stats.paragraphs = Some(out.paragraphs); e.stats.tables = Some(out.tables); @@ -209,19 +217,18 @@ impl Markkit { } #[cfg(not(feature = "docx"))] - fn extract_docx(&self, _bytes: &[u8]) -> crate::Result { + fn extract_docx(&self, _bytes: &[u8], _budget: &Budget) -> crate::Result { Err(Error::UnsupportedFormat( "DOCX support requires the 'docx' feature".to_string(), )) } #[cfg(feature = "spreadsheet")] - fn extract_spreadsheet(&self, bytes: &[u8]) -> crate::Result { + fn extract_spreadsheet(&self, bytes: &[u8], budget: &Budget) -> crate::Result { let out = guarded("spreadsheet", Error::SpreadsheetExtract, || { - crate::spreadsheet::parse(bytes, &self.config) + crate::spreadsheet::parse(bytes, &self.config, budget) })?; - let mut renderer = crate::render::MarkdownRenderer::new(); - let mut e = Extraction::new(renderer.render(&out.doc), out.format); + let mut e = Extraction::new(self.render(&out.doc, budget)?, out.format); e.warnings = out.warnings; e.stats.sheets = Some(out.sheets); e.stats.rows = Some(out.rows); @@ -229,7 +236,7 @@ impl Markkit { } #[cfg(not(feature = "spreadsheet"))] - fn extract_spreadsheet(&self, _bytes: &[u8]) -> crate::Result { + fn extract_spreadsheet(&self, _bytes: &[u8], _budget: &Budget) -> crate::Result { Err(Error::UnsupportedFormat( "spreadsheet support requires the 'spreadsheet' feature".to_string(), )) @@ -242,13 +249,19 @@ impl Markkit { /// refused with [`Error::UrlBlocked`] on every hop (including after DNS /// resolution) unless [`MarkkitBuilder::allow_private_networks`] is set; /// [`MarkkitBuilder::allow_host`] restricts hosts; redirects capped by - /// `max_redirects`; body capped by `max_input_size`; whole fetch capped - /// by `max_processing_secs` ([`Error::ResourceLimit`]). + /// `max_redirects`; body capped by `max_input_size`; fetch, parse and + /// render together capped by `limits.timeout` ([`Error::ResourceLimit`]). #[cfg(feature = "http_client")] pub async fn extract_url(&self, url: &str) -> crate::Result { + let start = std::time::Instant::now(); let fetched = crate::fetch::fetch(url, &self.config).await?; - self.extract_bytes(&fetched.bytes, fetched.content_type.as_deref()) - .await + let budget = Budget::starting_at(&self.config.limits, start); + let hint = DocumentHint { + content_type: fetched.content_type.as_deref(), + file_name: None, + }; + self.extract_document_with(&fetched.bytes, hint, &budget) + .map(|e| e.markdown) } /// PDF extraction is not implemented yet; report it explicitly rather than @@ -274,7 +287,7 @@ mod tests { #[tokio::test] async fn test_builder_defaults() { let m = Markkit::builder().build(); - assert_eq!(m.config().max_input_size, 50 * 1024 * 1024); + assert_eq!(m.config().limits.max_input_size, 50 * 1024 * 1024); assert!(m.config().preserve_links); } diff --git a/crates/markkit/src/fetch.rs b/crates/markkit/src/fetch.rs index bc4a127..8e19772 100644 --- a/crates/markkit/src/fetch.rs +++ b/crates/markkit/src/fetch.rs @@ -12,7 +12,7 @@ //! layers 1-2 still apply, layer 3 cannot. Documented in the threat model. //! - Body bounded by `max_input_size` while streaming (declared //! `Content-Length` rejected up front); whole fetch bounded by -//! `max_processing_secs`. +//! `limits.timeout`. //! - Policy and resolver run inside reqwest and can only return opaque //! errors, so they record the block reason in a per-call slot that is //! turned into [`Error::UrlBlocked`] afterwards. @@ -47,7 +47,7 @@ pub(crate) async fn fetch(url: &str, cfg: &MarkkitBuilder) -> crate::Result crate::Result crate::Result crate::Result { - let max_redirects = cfg.max_redirects; + let max_redirects = cfg.limits.max_redirects; let policy_cfg = cfg.clone(); let policy_slot = blocked.clone(); let policy = reqwest::redirect::Policy::custom(move |attempt| { @@ -114,11 +114,11 @@ fn client(cfg: &MarkkitBuilder, blocked: &Blocked) -> crate::Result { + Owned(Budget), + Borrowed(&'a Budget), +} /// Parses HTML into the canonical IR. +/// +/// Limits: nesting past `limits.max_nesting_depth` is flattened to text; DOM +/// node count and the deadline are enforced ([`Error::ResourceLimit`]). pub struct HtmlParser<'a> { config: &'a MarkkitBuilder, + budget: BudgetRef<'a>, + max_depth: usize, } impl<'a> HtmlParser<'a> { + /// Parser with a fresh per-call budget (deadline starts now). pub fn new(config: &'a MarkkitBuilder) -> Self { - Self { config } + Self { + config, + budget: BudgetRef::Owned(Budget::new(&config.limits)), + max_depth: config.limits.nesting_depth(), + } + } + + pub(crate) fn with_budget(config: &'a MarkkitBuilder, budget: &'a Budget) -> Self { + Self { + config, + budget: BudgetRef::Borrowed(budget), + max_depth: config.limits.nesting_depth(), + } + } + + fn budget(&self) -> &Budget { + match &self.budget { + BudgetRef::Owned(b) => b, + BudgetRef::Borrowed(b) => b, + } } /// Parse an HTML string into a [`Document`]. pub fn parse(&self, html: &str) -> Result { let dom = Html::parse_document(html); + // html5ever is not interruptible: check right after it finishes. + self.budget().check_time()?; + self.budget().add_nodes(dom.tree.nodes().count())?; let root = dom.root_element(); let content_nodes: Vec> = @@ -50,6 +81,8 @@ impl<'a> HtmlParser<'a> { for content_root in content_nodes { self.convert_children(content_root, &exclude_selectors, &mut nodes, 0); } + // The walk stops early once the deadline passes; report it here. + self.budget().check_time()?; Ok(Document::new(normalize_blocks(nodes))) } @@ -65,7 +98,10 @@ impl<'a> HtmlParser<'a> { out: &mut Vec, depth: usize, ) { - if depth >= MAX_DEPTH { + if self.budget().expired() { + return; + } + if depth >= self.max_depth { // At max depth, just extract text let text: String = parent.text().collect(); let trimmed = text.trim(); @@ -110,7 +146,10 @@ impl<'a> HtmlParser<'a> { exclude: &[Selector], depth: usize, ) -> Option { - if depth >= MAX_DEPTH { + if self.budget().expired() { + return None; + } + if depth >= self.max_depth { let text: String = el.text().collect(); let trimmed = text.trim(); if trimmed.is_empty() { @@ -300,7 +339,10 @@ impl<'a> HtmlParser<'a> { depth: usize, ) -> Vec { let mut nodes = Vec::new(); - if depth >= MAX_DEPTH { + if self.budget().expired() { + return nodes; + } + if depth >= self.max_depth { let text: String = el.text().collect(); let trimmed = text.trim(); if !trimmed.is_empty() { diff --git a/crates/markkit/src/lib.rs b/crates/markkit/src/lib.rs index d6453e2..9b5adfd 100644 --- a/crates/markkit/src/lib.rs +++ b/crates/markkit/src/lib.rs @@ -53,6 +53,7 @@ pub mod format; #[cfg(feature = "html")] pub mod html; pub mod ir; +pub mod limits; pub mod render; #[cfg(feature = "spreadsheet")] mod spreadsheet; @@ -64,6 +65,7 @@ pub use error::Error; pub use extract::{Extraction, ExtractionStats, Markkit}; pub use format::{DocumentHint, Format}; pub use ir::Document; +pub use limits::Limits; pub use render::MarkdownRenderer; /// Result type alias for markkit operations. diff --git a/crates/markkit/src/limits.rs b/crates/markkit/src/limits.rs new file mode 100644 index 0000000..4725e96 --- /dev/null +++ b/crates/markkit/src/limits.rs @@ -0,0 +1,317 @@ +//! Resource limits for extraction. +//! +//! Every input to markkit is untrusted, so every resource an input can drive +//! (bytes in, bytes out, time, nesting, node count, decompression, grid size, +//! redirects) has a limit here. Defaults are generous: real documents should +//! never hit them; hostile ones always do. +//! +//! Decisions: +//! - One [`Limits`] struct (bashkit `ExecutionLimits` style) instead of loose +//! builder fields, so the CLI and the Python/Node bindings map one shape. +//! - Violations are hard errors ([`Error::ResourceLimit`]), never silently +//! truncated output. Exceptions, kept for compatibility: per-sheet rows and +//! columns truncate with a warning, and HTML nesting past +//! `max_nesting_depth` is flattened to plain text. +//! - Time is checked cooperatively ([`Budget`]) in parser and renderer loops. +//! Third-party parse steps (html5ever tokenizing, calamine cell decoding) are +//! not interruptible; they are bounded by input size and checked right after. +//! - `max_nesting_depth` is clamped to [`HARD_MAX_NESTING_DEPTH`] because the +//! HTML walker, DOCX conversion and IR drop recurse on the host stack. +//! +//! See `knowledge/security/threat-model.md` (T2, T6) for the threat mapping. + +use std::cell::Cell; +use std::time::{Duration, Instant}; + +use crate::error::Error; + +// THREAT[T2b]: recursion depth must stay safe on small host stacks whatever +// the embedder configures. Measured worst case (nested blockquote / inline +// tags): 1024 levels fit a 2 MiB thread in release; debug frames are ~6x +// larger (default 256 fits 2 MiB in debug). Test: limits::*hard_ceiling*. +/// Upper bound applied to [`Limits::max_nesting_depth`]. +pub const HARD_MAX_NESTING_DEPTH: usize = 1024; + +/// Resource limits applied to every extraction. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct Limits { + /// Maximum input size in bytes, also the cap on a fetched URL body. + /// Default: 50 MiB. + pub max_input_size: usize, + /// Maximum Markdown output size in bytes. Default: 128 MiB. + pub max_output_size: usize, + /// Wall-clock budget for one call: URL fetch, parsing and rendering. + /// Default: 60 s. + pub timeout: Duration, + /// Maximum element nesting. HTML deeper than this is flattened to text; + /// XML parts (DOCX) deeper than this are rejected. Clamped to + /// [`HARD_MAX_NESTING_DEPTH`]. Default: 256. + pub max_nesting_depth: usize, + /// Maximum parsed nodes (HTML DOM nodes; XML elements and text nodes summed + /// over DOCX parts). Default: 5 000 000. + pub max_nodes: usize, + /// Maximum number of entries in a ZIP container (DOCX/XLSX/ODS). Default: 10 000. + pub max_archive_entries: usize, + /// Maximum decompressed size of a single ZIP entry in bytes. Default: 64 MiB. + pub max_archive_entry_size: u64, + /// Maximum total decompressed size of a ZIP container in bytes. Default: 256 MiB. + pub max_archive_total_size: u64, + /// Maximum data rows rendered per sheet (header excluded); extra rows are + /// dropped with a warning. Default: 10 000. + pub max_sheet_rows: usize, + /// Maximum columns rendered per sheet; extra columns are dropped with a + /// warning. Default: 256. + pub max_sheet_cols: usize, + /// Maximum cell grid a workbook may expand to while parsing (XLS/ODS + /// pre-scan, summed over sheets). Default: 4 000 000. + pub max_spreadsheet_cells: u64, + /// Maximum redirects `extract_url` follows. Default: 5. + pub max_redirects: usize, +} + +impl Default for Limits { + fn default() -> Self { + Self { + max_input_size: 50 * 1024 * 1024, + max_output_size: 128 * 1024 * 1024, + timeout: Duration::from_secs(60), + max_nesting_depth: 256, + max_nodes: 5_000_000, + max_archive_entries: 10_000, + max_archive_entry_size: 64 * 1024 * 1024, + max_archive_total_size: 256 * 1024 * 1024, + max_sheet_rows: 10_000, + max_sheet_cols: 256, + max_spreadsheet_cells: 4_000_000, + max_redirects: 5, + } + } +} + +impl Limits { + /// Default limits. + pub fn new() -> Self { + Self::default() + } + + /// Effective nesting depth after the hard ceiling. + pub fn nesting_depth(&self) -> usize { + self.max_nesting_depth.min(HARD_MAX_NESTING_DEPTH) + } + + pub fn max_input_size(mut self, bytes: usize) -> Self { + self.max_input_size = bytes; + self + } + + pub fn max_output_size(mut self, bytes: usize) -> Self { + self.max_output_size = bytes; + self + } + + pub fn timeout(mut self, timeout: Duration) -> Self { + self.timeout = timeout; + self + } + + pub fn max_nesting_depth(mut self, depth: usize) -> Self { + self.max_nesting_depth = depth; + self + } + + pub fn max_nodes(mut self, nodes: usize) -> Self { + self.max_nodes = nodes; + self + } + + pub fn max_archive_entries(mut self, n: usize) -> Self { + self.max_archive_entries = n; + self + } + + pub fn max_archive_entry_size(mut self, bytes: u64) -> Self { + self.max_archive_entry_size = bytes; + self + } + + pub fn max_archive_total_size(mut self, bytes: u64) -> Self { + self.max_archive_total_size = bytes; + self + } + + pub fn max_sheet_rows(mut self, rows: usize) -> Self { + self.max_sheet_rows = rows; + self + } + + pub fn max_sheet_cols(mut self, cols: usize) -> Self { + self.max_sheet_cols = cols; + self + } + + pub fn max_spreadsheet_cells(mut self, cells: u64) -> Self { + self.max_spreadsheet_cells = cells; + self + } + + pub fn max_redirects(mut self, n: usize) -> Self { + self.max_redirects = n; + self + } + + /// Reject inputs larger than `max_input_size`. + pub(crate) fn check_input(&self, len: usize) -> crate::Result<()> { + if len > self.max_input_size { + return Err(Error::ResourceLimit(format!( + "input size {len} exceeds max_input_size {}", + self.max_input_size + ))); + } + Ok(()) + } +} + +// Budget is only driven by the feature-gated parsers. +/// How many [`Budget::expired`] calls share one clock read. +#[cfg_attr( + not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + allow(dead_code) +)] +const CLOCK_EVERY: u32 = 256; + +/// Per-call deadline and node counter, shared by reference through parsers and +/// the renderer (interior mutability so `&self` walkers can charge it). +#[derive(Debug)] +#[cfg_attr( + not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + allow(dead_code) +)] +pub(crate) struct Budget { + deadline: Option, + timeout: Duration, + ticks: Cell, + expired: Cell, + #[cfg_attr(not(any(feature = "html", feature = "docx")), allow(dead_code))] + nodes: Cell, + #[cfg_attr(not(any(feature = "html", feature = "docx")), allow(dead_code))] + max_nodes: usize, +} + +#[cfg_attr( + not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + allow(dead_code) +)] +impl Budget { + /// Budget starting now. + pub fn new(limits: &Limits) -> Self { + Self::starting_at(limits, Instant::now()) + } + + /// Budget whose clock started at `start` (e.g. before a URL fetch). + pub fn starting_at(limits: &Limits, start: Instant) -> Self { + Self { + // Overflow (e.g. Duration::MAX) means "no deadline". + deadline: start.checked_add(limits.timeout), + timeout: limits.timeout, + ticks: Cell::new(0), + expired: Cell::new(false), + nodes: Cell::new(0), + max_nodes: limits.max_nodes, + } + } + + pub fn deadline(&self) -> Option { + self.deadline + } + + /// Cheap periodic deadline check for hot loops; reads the clock every + /// [`CLOCK_EVERY`] calls. Sticky once expired. + pub fn expired(&self) -> bool { + if self.expired.get() { + return true; + } + let t = self.ticks.get().wrapping_add(1); + self.ticks.set(t); + if t.is_multiple_of(CLOCK_EVERY) { + self.read_clock(); + } + self.expired.get() + } + + fn read_clock(&self) { + if let Some(d) = self.deadline + && Instant::now() >= d + { + self.expired.set(true); + } + } + + /// Error if the deadline has passed (reads the clock). + pub fn check_time(&self) -> crate::Result<()> { + self.read_clock(); + if self.expired.get() { + return Err(self.timeout_error()); + } + Ok(()) + } + + pub fn timeout_error(&self) -> Error { + Error::ResourceLimit(format!("processing exceeded timeout of {:?}", self.timeout)) + } + + /// Charge `n` parsed nodes against `max_nodes`. + #[cfg_attr(not(any(feature = "html", feature = "docx")), allow(dead_code))] + pub fn add_nodes(&self, n: usize) -> crate::Result<()> { + let total = self.nodes.get().saturating_add(n); + self.nodes.set(total); + if total > self.max_nodes { + return Err(Error::ResourceLimit(format!( + "document has more than max_nodes {} nodes", + self.max_nodes + ))); + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn nesting_depth_clamped_to_hard_ceiling() { + let l = Limits::default().max_nesting_depth(usize::MAX); + assert_eq!(l.nesting_depth(), HARD_MAX_NESTING_DEPTH); + assert_eq!(Limits::default().nesting_depth(), 256); + } + + #[test] + fn zero_timeout_expires() { + let b = Budget::new(&Limits::default().timeout(Duration::ZERO)); + assert!(matches!(b.check_time(), Err(Error::ResourceLimit(_)))); + assert!((0..CLOCK_EVERY).any(|_| b.expired())); + } + + #[test] + fn huge_timeout_never_expires() { + let b = Budget::new(&Limits::default().timeout(Duration::MAX)); + assert!(b.check_time().is_ok()); + assert!(!(0..CLOCK_EVERY * 2).any(|_| b.expired())); + } + + #[test] + fn node_budget_accumulates() { + let b = Budget::new(&Limits::default().max_nodes(10)); + assert!(b.add_nodes(6).is_ok()); + assert!(b.add_nodes(4).is_ok()); + assert!(matches!(b.add_nodes(1), Err(Error::ResourceLimit(_)))); + } + + #[test] + fn input_check() { + let l = Limits::default().max_input_size(3); + assert!(l.check_input(3).is_ok()); + assert!(matches!(l.check_input(4), Err(Error::ResourceLimit(_)))); + } +} diff --git a/crates/markkit/src/render/markdown.rs b/crates/markkit/src/render/markdown.rs index eec02a1..5fee315 100644 --- a/crates/markkit/src/render/markdown.rs +++ b/crates/markkit/src/render/markdown.rs @@ -1,30 +1,102 @@ //! IR → Markdown renderer. +//! +//! Decision: output size and the deadline are checked per block, list item and +//! table row (not per byte). Tables pad every row to the widest row, so output +//! can be quadratic in input; checking per row keeps any buffer within +//! `max_output_size` plus one row. Nested renderers (blockquotes, list items) +//! inherit the limits and report a trip back to their parent. + +use std::time::Instant; use crate::ir::{Document, ListItem, Node}; +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Trip { + Output, + Time, +} + /// Renders a [`Document`] IR into Markdown text. pub struct MarkdownRenderer { /// Output buffer. buf: String, + max_output: usize, + deadline: Option, + tripped: Option, } impl MarkdownRenderer { pub fn new() -> Self { Self { buf: String::with_capacity(4096), + max_output: usize::MAX, + deadline: None, + tripped: None, } } - /// Render a document to a Markdown string. + /// Render a document to a Markdown string (no limits; the IR is trusted). pub fn render(&mut self, doc: &Document) -> String { self.buf.clear(); self.render_nodes(&doc.nodes, 0); normalize_output(&self.buf) } + /// Render with `max_output` bytes and the budget's deadline enforced. + #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + pub(crate) fn render_bounded( + &mut self, + doc: &Document, + max_output: usize, + budget: &crate::limits::Budget, + ) -> crate::Result { + self.max_output = max_output; + self.deadline = budget.deadline(); + self.tripped = None; + let out = self.render(doc); + match self.tripped { + Some(Trip::Time) => Err(budget.timeout_error()), + Some(Trip::Output) => Err(output_error(max_output)), + None if out.len() > max_output => Err(output_error(max_output)), + None => Ok(out), + } + } + + /// Nested renderer sharing this renderer's limits. + fn sub(&self) -> Self { + Self { + max_output: self.max_output, + deadline: self.deadline, + ..Self::new() + } + } + + /// Fold a nested renderer's trip into this one. + fn absorb(&mut self, sub: &Self) { + if self.tripped.is_none() { + self.tripped = sub.tripped; + } + } + + /// True once a limit is hit; rendering then stops at the next check. + fn over_limit(&mut self) -> bool { + if self.tripped.is_some() { + return true; + } + if self.buf.len() > self.max_output { + self.tripped = Some(Trip::Output); + } else if self.deadline.is_some_and(|d| Instant::now() >= d) { + self.tripped = Some(Trip::Time); + } + self.tripped.is_some() + } + /// Render a slice of nodes. fn render_nodes(&mut self, nodes: &[Node], depth: usize) { for (i, node) in nodes.iter().enumerate() { + if self.over_limit() { + return; + } if i > 0 && is_block(node) && is_block(&nodes[i - 1]) { self.buf.push('\n'); } @@ -61,8 +133,9 @@ impl MarkdownRenderer { Node::BlockQuote { children } => { let inner = { - let mut sub = MarkdownRenderer::new(); + let mut sub = self.sub(); sub.render_nodes(children, depth); + self.absorb(&sub); normalize_output(&sub.buf) }; for line in inner.lines() { @@ -117,6 +190,9 @@ impl MarkdownRenderer { let start_num = start.unwrap_or(1); for (i, item) in items.iter().enumerate() { + if self.over_limit() { + return; + } let marker = if ordered { format!("{}{}. ", indent, start_num + i as u64) } else { @@ -146,8 +222,9 @@ impl MarkdownRenderer { self.render_list(*sub_ord, *sub_start, sub_items, depth + 1); } _ => { - let mut sub = MarkdownRenderer::new(); + let mut sub = self.sub(); sub.render_node(child, depth + 1); + self.absorb(&sub); let text = sub.buf.trim_end().to_string(); self.buf.push_str(&text); self.buf.push('\n'); @@ -163,8 +240,9 @@ impl MarkdownRenderer { self.render_list(*sub_ord, *sub_start, sub_items, depth + 1); } _ => { - let mut sub = MarkdownRenderer::new(); + let mut sub = self.sub(); sub.render_node(child, depth + 1); + self.absorb(&sub); let inner = sub.buf.trim_end().to_string(); let sub_indent = " ".repeat(depth + 1); for line in inner.lines() { @@ -277,6 +355,9 @@ impl MarkdownRenderer { // Data rows for row in rows { + if self.over_limit() { + return; + } self.buf.push('|'); for i in 0..col_count { self.buf.push(' '); @@ -296,6 +377,11 @@ impl Default for MarkdownRenderer { } } +#[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] +fn output_error(max: usize) -> crate::Error { + crate::Error::ResourceLimit(format!("output exceeds max_output_size {max}")) +} + /// Trim leading and trailing whitespace from inline content rendered since `start`. fn trim_trailing_inline(buf: &mut String, start: usize) { // Trim trailing spaces diff --git a/crates/markkit/src/spreadsheet.rs b/crates/markkit/src/spreadsheet.rs index 2ca280c..5ff1383 100644 --- a/crates/markkit/src/spreadsheet.rs +++ b/crates/markkit/src/spreadsheet.rs @@ -29,6 +29,7 @@ use crate::config::MarkkitBuilder; use crate::error::Error; use crate::format::Format; use crate::ir::{Document, Node}; +use crate::limits::Budget; use crate::render::escape_table_cell; pub(crate) struct SheetOutput { @@ -62,7 +63,11 @@ enum Container { Xls, } -pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result { +pub(crate) fn parse( + bytes: &[u8], + config: &MarkkitBuilder, + budget: &Budget, +) -> Result { let container = if bytes.starts_with(b"PK") { let mut zip = ZipGuard::open(bytes, config).map_err(archive_err)?; let has = @@ -84,7 +89,7 @@ pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result Result Result Result<(), calamine::XlsxError> { let mut reader = wb.worksheet_cells_reader(&name)?; while let Some(cell) = reader.next_cell()? { + if budget.expired() { + break; + } let (r, c) = cell.get_position(); grid.add(r, c, fmt_ref(cell.get_value())); } Ok(()) })(); + budget.check_time()?; match res { Ok(()) => out.push_sheet(&name, grid), Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), @@ -121,13 +131,13 @@ pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result { let mut wb: Ods<_> = Ods::new(Cursor::new(bytes)).map_err(err)?; let sel = out.select(wb.sheets_metadata()); - read_ranges(&mut wb, sel, &mut out, config); + read_ranges(&mut wb, sel, &mut out, config, budget)?; out.finish(Format::Ods) } Container::Xls => { let mut wb: Xls<_> = Xls::new(Cursor::new(bytes)).map_err(err)?; let sel = out.select(wb.sheets_metadata()); - read_ranges(&mut wb, sel, &mut out, config); + read_ranges(&mut wb, sel, &mut out, config, budget)?; out.finish(Format::Xls) } } @@ -138,7 +148,9 @@ fn read_ranges( sel: Vec<(String, bool)>, out: &mut Collector, config: &MarkkitBuilder, -) where + budget: &Budget, +) -> Result<(), Error> +where RS: Read + Seek, R: Reader, R::Error: std::fmt::Display, @@ -147,17 +159,25 @@ fn read_ranges( if !include { continue; } - match wb.worksheet_range(&name) { + let range = wb.worksheet_range(&name); + // calamine decodes the whole sheet before returning: check after it. + budget.check_time()?; + match range { Ok(range) => { let mut grid = Grid::new(config); for (r, c, v) in range.used_cells() { + if budget.expired() { + break; + } grid.add(r as u32, c as u32, fmt_data(v)); } + budget.check_time()?; out.push_sheet(&name, grid); } Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), } } + Ok(()) } // ─── Cell formatting ──────────────────────────────────────────────── @@ -238,8 +258,8 @@ impl Grid { fn new(config: &MarkkitBuilder) -> Self { Self { rows: BTreeMap::new(), - max_rows: config.max_sheet_rows, - max_cols: config.max_sheet_cols.max(1), + max_rows: config.limits.max_sheet_rows, + max_cols: config.limits.max_sheet_cols.max(1), truncated_rows: false, truncated_cols: false, } @@ -437,7 +457,7 @@ fn prescan_xls(bytes: &[u8], config: &MarkkitBuilder) -> Result<(), Error> { .find(|n| comp.is_stream(n)) .ok_or_else(|| err("XLS has no Workbook stream"))?; let mut stream = Vec::new(); - let cap = config.max_archive_entry_size; + let cap = config.limits.max_archive_entry_size; comp.open_stream(name) .map_err(|e| err(format!("cannot open Workbook stream: {e}")))? .take(cap.saturating_add(1)) @@ -448,7 +468,7 @@ fn prescan_xls(bytes: &[u8], config: &MarkkitBuilder) -> Result<(), Error> { "XLS Workbook stream exceeds {cap} bytes" ))); } - scan_biff(&stream, config.max_spreadsheet_cells) + scan_biff(&stream, config.limits.max_spreadsheet_cells) } /// Estimate the cells calamine will allocate and reject inputs that would diff --git a/crates/markkit/src/xml.rs b/crates/markkit/src/xml.rs index a2881ba..45187af 100644 --- a/crates/markkit/src/xml.rs +++ b/crates/markkit/src/xml.rs @@ -5,13 +5,15 @@ //! matching on local names keeps us robust to unusual prefix choices without a //! full namespace resolver. quick-xml never expands DTD-declared entities, so //! billion-laughs / XXE payloads surface as unresolved references, which we drop. -//! Tree depth is capped and the tree is built iteratively (no recursion on input). +//! Tree depth (`limits.max_nesting_depth`), node count (`limits.max_nodes`, +//! shared across parts) and the deadline are enforced while building; the tree +//! is built iteratively (no recursion on input). use quick_xml::events::{BytesStart, Event}; use quick_xml::{Reader, XmlVersion}; -/// Maximum element nesting accepted in a single XML part. -pub(crate) const MAX_XML_DEPTH: usize = 256; +use crate::error::Error; +use crate::limits::Budget; #[derive(Debug)] pub(crate) enum XmlError { @@ -126,9 +128,16 @@ fn push_text(parent: &mut Element, text: &str) { } } +fn limit(e: Error) -> XmlError { + match e { + Error::ResourceLimit(m) => XmlError::Limit(m), + other => XmlError::Invalid(other.to_string()), + } +} + /// Parse an XML document into a tree. Returns a synthetic root whose children /// are the document's top-level nodes. -pub(crate) fn parse(bytes: &[u8]) -> Result { +pub(crate) fn parse(bytes: &[u8], max_depth: usize, budget: &Budget) -> Result { let mut reader = Reader::from_reader(bytes); reader.config_mut().check_end_names = false; let mut stack: Vec = vec![Element::default()]; @@ -137,11 +146,20 @@ pub(crate) fn parse(bytes: &[u8]) -> Result { let event = reader .read_event_into(&mut buf) .map_err(|e| XmlError::Invalid(format!("malformed XML: {e}")))?; + if matches!( + event, + Event::Start(_) | Event::Empty(_) | Event::Text(_) | Event::CData(_) + ) { + budget.add_nodes(1).map_err(limit)?; + if budget.expired() { + return Err(limit(budget.timeout_error())); + } + } match event { Event::Start(s) => { - if stack.len() > MAX_XML_DEPTH { + if stack.len() > max_depth { return Err(XmlError::Limit(format!( - "XML nesting deeper than {MAX_XML_DEPTH}" + "XML nesting deeper than max_nesting_depth {max_depth}" ))); } stack.push(element_from(&s)); @@ -202,6 +220,11 @@ pub(crate) fn parse(bytes: &[u8]) -> Result { #[cfg(test)] mod tests { use super::*; + use crate::limits::Limits; + + fn parse(bytes: &[u8]) -> Result { + super::parse(bytes, 256, &Budget::new(&Limits::default())) + } #[test] fn parses_local_names_text_and_refs() { @@ -216,8 +239,35 @@ mod tests { #[test] fn depth_limited() { - let deep = "".repeat(MAX_XML_DEPTH + 10); + let deep = "".repeat(256 + 10); assert!(matches!(parse(deep.as_bytes()), Err(XmlError::Limit(_)))); + let ok = format!("{}{}", "".repeat(100), "".repeat(100)); + assert!(parse(ok.as_bytes()).is_ok()); + let budget = Budget::new(&Limits::default()); + assert!(matches!( + super::parse(ok.as_bytes(), 50, &budget), + Err(XmlError::Limit(_)) + )); + } + + #[test] + fn node_count_limited() { + let many = format!("{}", "".repeat(100)); + let budget = Budget::new(&Limits::default().max_nodes(50)); + assert!(matches!( + super::parse(many.as_bytes(), 256, &budget), + Err(XmlError::Limit(m)) if m.contains("max_nodes") + )); + } + + #[test] + fn deadline_enforced() { + let many = format!("{}", "".repeat(1000)); + let budget = Budget::new(&Limits::default().timeout(std::time::Duration::ZERO)); + assert!(matches!( + super::parse(many.as_bytes(), 256, &budget), + Err(XmlError::Limit(m)) if m.contains("timeout") + )); } #[test] diff --git a/crates/markkit/tests/html_integration.rs b/crates/markkit/tests/html_integration.rs index afd18f6..8f961b5 100644 --- a/crates/markkit/tests/html_integration.rs +++ b/crates/markkit/tests/html_integration.rs @@ -593,7 +593,10 @@ fn code_block_with_backticks() { #[test] fn config_max_processing_secs() { let m = Markkit::builder().max_processing_secs(30).build(); - assert_eq!(m.config().max_processing_secs, 30); + assert_eq!( + m.config().limits.timeout, + std::time::Duration::from_secs(30) + ); } #[test] diff --git a/crates/markkit/tests/limits.rs b/crates/markkit/tests/limits.rs new file mode 100644 index 0000000..046240f --- /dev/null +++ b/crates/markkit/tests/limits.rs @@ -0,0 +1,269 @@ +//! Resource limits (threat model T2/T6): every limit in `markkit::Limits` is +//! enforced, defaults are generous, and the nesting hard ceiling keeps deep +//! input off the host stack. + +#![cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + +mod common; + +use std::time::Duration; + +use markkit::{DocumentHint, Error, Limits, Markkit}; + +#[allow(dead_code)] +const DOCX_CT: &str = "application/vnd.openxmlformats-officedocument.wordprocessingml.document"; +#[allow(dead_code)] +const XLSX_CT: &str = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; + +fn limit_msg(r: markkit::Result) -> String { + match r { + Err(Error::ResourceLimit(m)) => m, + other => panic!("expected ResourceLimit, got {other:?}"), + } +} + +#[test] +fn defaults_are_generous() { + let l = Limits::default(); + assert_eq!(l.max_input_size, 50 * 1024 * 1024); + assert_eq!(l.max_output_size, 128 * 1024 * 1024); + assert_eq!(l.timeout, Duration::from_secs(60)); + assert_eq!(l.max_nesting_depth, 256); + assert_eq!(l.max_nodes, 5_000_000); + assert_eq!(l.max_redirects, 5); + assert_eq!(Markkit::builder().build().config().limits, l); +} + +#[test] +fn builder_limits_replaces_all() { + let l = Limits::default().max_output_size(7).max_nodes(9); + let m = Markkit::builder().limits(l.clone()).build(); + assert_eq!(m.config().limits, l); + let m = Markkit::builder() + .max_processing_secs(3) + .max_nesting_depth(12) + .build(); + assert_eq!(m.config().limits.timeout, Duration::from_secs(3)); + assert_eq!(m.config().limits.max_nesting_depth, 12); +} + +/// Stack for hard-ceiling tests: 2 MiB (Rust's default thread stack) in +/// release; debug frames are ~6x larger, so debug runs get 8 MiB. +#[allow(dead_code)] +fn small_stack() -> usize { + if cfg!(debug_assertions) { + 8 << 20 + } else { + 2 << 20 + } +} + +// ─── HTML ─────────────────────────────────────────────────────────── + +#[cfg(feature = "html")] +mod html { + use super::*; + use markkit::limits::HARD_MAX_NESTING_DEPTH; + + fn deep(tag: &str, n: usize, inner: &str) -> String { + format!( + "{}{inner}{}", + format!("<{tag}>").repeat(n), + format!("").repeat(n) + ) + } + + #[test] + fn output_cap_rejects_large_output() { + let html = "

hello world

".repeat(1000); + let m = Markkit::builder().max_output_size(1000).build(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + let m = Markkit::builder().max_output_size(100_000).build(); + assert!(m.extract_html_sync(&html).is_ok()); + } + + #[test] + fn output_cap_stops_table_amplification() { + // One wide row pads every other row: output ~ rows x cols. + let wide = format!("{}", "x".repeat(2000)); + let html = format!( + "{wide}{}
", + "y".repeat(2000) + ); + let m = Markkit::builder().max_output_size(1 << 20).build(); + let start = std::time::Instant::now(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + assert!(start.elapsed() < Duration::from_secs(10)); + } + + #[test] + fn output_cap_applies_inside_nested_renderers() { + let wide = format!("{}", "x".repeat(500)); + let table = format!("{wide}{}
", "y".repeat(500)); + for html in [ + format!("
{table}
"), + format!("
  • a

    {table}
"), + ] { + let m = Markkit::builder().max_output_size(64 * 1024).build(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + } + } + + #[test] + fn zero_timeout_rejects() { + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(m.extract_html_sync("

x

")).contains("timeout")); + let r = m.extract_document_sync(b"

x

", DocumentHint::content_type("text/html")); + assert!(limit_msg(r).contains("timeout")); + } + + #[test] + fn node_cap_rejects() { + let html = "

a

".repeat(1000); + let m = Markkit::builder().max_nodes(500).build(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_nodes")); + assert!(limit_msg(m.parse_html(&html)).contains("max_nodes")); + let m = Markkit::builder().max_nodes(10_000).build(); + assert!(m.extract_html_sync(&html).is_ok()); + } + + #[test] + fn nesting_depth_configurable_flattens() { + let html = deep("blockquote", 20, "

Deep

"); + let full = Markkit::builder().build().extract_html_sync(&html).unwrap(); + assert!(full.contains(&"> ".repeat(20))); + let m = Markkit::builder().max_nesting_depth(5).build(); + let md = m.extract_html_sync(&html).unwrap(); + assert!(md.contains("Deep")); + assert!(!md.contains(&"> ".repeat(6))); + } + + #[test] + fn hard_ceiling_survives_small_stack() { + // Configured depth is clamped; deep nesting of recursive constructs + // must fit a small thread stack. + let inputs = [ + deep("blockquote", HARD_MAX_NESTING_DEPTH * 4, "

q

"), + deep("ul", HARD_MAX_NESTING_DEPTH * 4, "
  • l
  • "), + deep("div", HARD_MAX_NESTING_DEPTH * 4, "

    d

    "), + deep("b", HARD_MAX_NESTING_DEPTH * 4, "t"), + ]; + std::thread::Builder::new() + .stack_size(small_stack()) + .spawn(move || { + let m = Markkit::builder().max_nesting_depth(usize::MAX).build(); + for html in inputs { + m.extract_html_sync(&html).unwrap(); + } + }) + .unwrap() + .join() + .expect("deep input overflowed a small stack"); + } +} + +// ─── DOCX ─────────────────────────────────────────────────────────── + +#[cfg(feature = "docx")] +mod docx { + use super::*; + use common::*; + use markkit::limits::HARD_MAX_NESTING_DEPTH; + + fn run(m: &Markkit, bytes: &[u8]) -> markkit::Result { + m.extract_document_sync(bytes, DocumentHint::content_type(DOCX_CT)) + } + + #[test] + fn node_cap_rejects() { + let body: String = (0..200).map(|i| para(&format!("p{i}"))).collect(); + let bytes = docx(&body); + let m = Markkit::builder().max_nodes(100).build(); + assert!(limit_msg(run(&m, &bytes)).contains("max_nodes")); + assert!(run(&Markkit::builder().build(), &bytes).is_ok()); + } + + #[test] + fn nesting_depth_configurable() { + let body = format!( + "{}{}{}", + "".repeat(10), + para("x"), + "".repeat(10) + ); + let bytes = docx(&body); + assert!(run(&Markkit::builder().build(), &bytes).is_ok()); + let m = Markkit::builder().max_nesting_depth(8).build(); + assert!(limit_msg(run(&m, &bytes)).contains("max_nesting_depth")); + } + + #[test] + fn hard_ceiling_survives_small_stack() { + let n = HARD_MAX_NESTING_DEPTH / 2 - 8; + let body = format!( + "{}{}{}", + "".repeat(n), + para("x"), + "".repeat(n) + ); + let bytes = docx(&body); + let deeper = docx(&"".repeat(HARD_MAX_NESTING_DEPTH * 2)); + std::thread::Builder::new() + .stack_size(small_stack()) + .spawn(move || { + let m = Markkit::builder().max_nesting_depth(usize::MAX).build(); + assert!(run(&m, &bytes).unwrap().markdown.contains('x')); + assert!(limit_msg(run(&m, &deeper)).contains("max_nesting_depth")); + }) + .unwrap() + .join() + .expect("deep DOCX overflowed a small stack"); + } + + #[test] + fn zero_timeout_rejects() { + let bytes = docx(¶("x")); + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(run(&m, &bytes)).contains("timeout")); + } + + #[test] + fn output_cap_rejects() { + let body: String = (0..200).map(|i| para(&format!("paragraph {i}"))).collect(); + let bytes = docx(&body); + let m = Markkit::builder().max_output_size(500).build(); + assert!(limit_msg(run(&m, &bytes)).contains("max_output_size")); + } +} + +// ─── Spreadsheets ─────────────────────────────────────────────────── + +#[cfg(feature = "spreadsheet")] +mod spreadsheet { + use super::*; + use common::*; + + fn sheet() -> Vec { + let rows = (0..200) + .map(|i| vec![Cell::Str(format!("row {i}")), Cell::N(i as f64)]) + .collect(); + xlsx(&[Sheet::new("S", rows)]) + } + + fn run(m: &Markkit, bytes: &[u8]) -> markkit::Result { + m.extract_document_sync(bytes, DocumentHint::content_type(XLSX_CT)) + } + + #[test] + fn zero_timeout_rejects() { + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(run(&m, &sheet())).contains("timeout")); + } + + #[test] + fn output_cap_rejects() { + let m = Markkit::builder().max_output_size(500).build(); + assert!(limit_msg(run(&m, &sheet())).contains("max_output_size")); + assert!(run(&Markkit::builder().build(), &sheet()).is_ok()); + } +} diff --git a/examples/configuration.rs b/examples/configuration.rs index d9213a7..c307d9d 100644 --- a/examples/configuration.rs +++ b/examples/configuration.rs @@ -34,4 +34,17 @@ fn main() { Ok(_) => println!("Unexpected success"), Err(e) => println!("--- max_input_size(10) ---\nError: {e}"), } + + // All limits at once: output cap, timeout, nesting depth, node count, ... + let bounded = Markkit::builder() + .limits( + markkit::Limits::default() + .max_output_size(20) + .timeout(std::time::Duration::from_secs(5)), + ) + .build(); + match bounded.extract_html_sync(html) { + Ok(_) => println!("Unexpected success"), + Err(e) => println!("--- max_output_size(20) ---\nError: {e}"), + } } diff --git a/knowledge/formats/html-extraction.md b/knowledge/formats/html-extraction.md index c1bcc1a..e9c92d1 100644 --- a/knowledge/formats/html-extraction.md +++ b/knowledge/formats/html-extraction.md @@ -121,7 +121,7 @@ Trimming at render time: ## Depth Limiting -Constant: `MAX_DEPTH = 200` +Limit: `limits.max_nesting_depth` (default 256, clamped to `HARD_MAX_NESTING_DEPTH` = 1024) Behavior at depth limit: - `convert_children`: extracts all descendant text via `.text().collect()`, trims, emits as single `Text` node diff --git a/knowledge/foundations/architecture.md b/knowledge/foundations/architecture.md index 9781979..82c0c47 100644 --- a/knowledge/foundations/architecture.md +++ b/knowledge/foundations/architecture.md @@ -48,8 +48,10 @@ Input (bytes/URL/file) ## Configuration Via `MarkkitBuilder`: -- Input size limits -- Processing time limits +- Resource limits: one `Limits` struct (`limits.rs`, set whole via `.limits(..)` or per + field via builder setters): input/output size, timeout, nesting depth (hard max 1024), + node count, archive, spreadsheet and redirect caps. Violations are + `Error::ResourceLimit`; see [Threat Model](../security/threat-model.md) Resource Limits - CSS selectors for include/exclude - Link and image preservation - Office-document limits (archive entries/sizes, sheet rows/cols, grid cells, hidden sheets) diff --git a/knowledge/foundations/ir.md b/knowledge/foundations/ir.md index f861201..0c73b97 100644 --- a/knowledge/foundations/ir.md +++ b/knowledge/foundations/ir.md @@ -111,7 +111,7 @@ At render time, the `normalize_output` function: | No empty Text nodes after whitespace collapse | `collapse_whitespace` returns empty string for whitespace-only; parser skips empty | | Block lists hold no bare inline runs | `normalize_blocks` drops whitespace-only runs, wraps others in `Paragraph` | | Empty containers are elided | Parser returns `None` for headings, paragraphs, lists, etc. with no children | -| Depth bounded at 200 | `MAX_DEPTH` constant; at limit, subtree flattened to text | +| Depth bounded (`limits.max_nesting_depth`, default 256, hard max 1024) | HTML parser flattens the subtree to text at the limit | | No executable content in IR | Script, style, iframe, etc. stripped by parser | | `List.start` is `None` when `ordered=false` | Parser only reads `start` attr for `
      ` | diff --git a/knowledge/log.md b/knowledge/log.md index 1b2ff07..817b33f 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,6 +2,7 @@ ## 2026-10-26 +* **Resource limits**: all limits moved into one `Limits` struct (bashkit `ExecutionLimits` style). New: `timeout` now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB, stops table amplification); `max_nodes` (5M); `max_nesting_depth` configurable (256, was hardcoded 200 HTML / 256 XML) with a 1024 hard ceiling. Invariants S13-S15, `limits` tests. See [Threat Model](security/threat-model.md) T2. * **URL fetch hardening**: T3 SSRF is mitigated. `extract_url` blocks non-public addresses by default (URL, resolver and redirect layers), enforces scheme, allowlist, redirect, body and time limits; new invariant S12 and `url_security` tests. Proxy-resolved names are the documented residual. See [Threat Model](security/threat-model.md) and [Security Model](security/security-model.md). * **Dependency trim**: unused dependencies removed (lockfile shrinks by ~460 lines; the `instant`/`paste` unmaintained advisories and their `deny.toml` ignores are gone with the stub bench deps). quick-xml moves to 0.42, one version shared with calamine; markkit's XML walkers now read `&str` names and values. See [Threat Model](security/threat-model.md) T6d and [Maintenance](operations/maintenance.md). * **Inter-block whitespace**: indented HTML lost blank lines between blocks and leaked a leading space into every following line, because whitespace-only text between blocks stayed in the block list as `Text(" ")`. The parser now normalizes block-context lists; golden fixtures, which had recorded the broken output, are regenerated. See [HTML Extraction](formats/html-extraction.md) and [IR Specification](foundations/ir.md). diff --git a/knowledge/operations/testing.md b/knowledge/operations/testing.md index 49bdfba..0069cd2 100644 --- a/knowledge/operations/testing.md +++ b/knowledge/operations/testing.md @@ -21,6 +21,7 @@ tags: | `crates/markkit/tests/docx_integration.rs`, `spreadsheet_integration.rs` | Office mapping | | `crates/markkit/tests/document_detection.rs` | Format detection order and explicit unsupported errors | | `crates/markkit/tests/document_security.rs` | Zip bombs, grid amplification, XML entities, malformed containers | +| `crates/markkit/tests/limits.rs` | Every `Limits` field enforced for HTML, DOCX, spreadsheets; small-stack depth ceiling | | `crates/markkit/tests/common/` | In-memory DOCX/XLSX/XLS/ODS fixture builders | | `crates/markkit-cli/tests/cli_integration.rs` | CLI flags and output | | `crates/markkit/benches/extraction.rs` | Criterion throughput benchmarks | diff --git a/knowledge/security/security-model.md b/knowledge/security/security-model.md index a5086a6..70ffd08 100644 --- a/knowledge/security/security-model.md +++ b/knowledge/security/security-model.md @@ -25,7 +25,7 @@ Markkit processes untrusted input from the internet. Key threats: ### TM-SSRF: Server-side request forgery (URL fetching) - **Mitigation**: Non-public addresses refused by default, after DNS resolution and on every redirect (opt in with `allow_private_networks`) -- **Mitigation**: http/https only, optional host allowlist, redirect limit, body bounded by `max_input_size`, fetch bounded by `max_processing_secs` +- **Mitigation**: http/https only, optional host allowlist, redirect limit, body bounded by `max_input_size`, fetch bounded by `timeout` - **Residual**: an environment proxy resolves names itself; see [Threat Model](threat-model.md) T3 ### TM-DOC: Malicious office documents diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 49d78f1..913e529 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -59,19 +59,34 @@ tags: **Sub-threats**: +All limits live in one `Limits` struct (`crates/markkit/src/limits.rs`, +`MarkkitBuilder::limits`); violations return `Error::ResourceLimit` naming the limit. + #### T2a: Oversized Input - Input size checked before parsing against `max_input_size` - Default: 50 MB - Returns `Error::ResourceLimit` on violation #### T2b: Deep Nesting (Stack Overflow) -- `MAX_DEPTH=200` limits recursion in HTML parser -- At depth limit, subtree flattened to plain text (no further recursion) -- Tested with 1000 levels of nesting - -#### T2c: Many Elements (CPU Exhaustion) -- `max_processing_secs` configurable (default: 60s) -- Tested with 1000 paragraph elements +- `max_nesting_depth` (default 256) bounds recursion: HTML flattens deeper subtrees to + plain text; DOCX XML parts deeper than it are rejected +- Clamped to `HARD_MAX_NESTING_DEPTH` (1024) whatever the embedder sets: 1024 levels + of the worst recursive constructs fit a 2 MiB stack in release (debug frames are ~6x) +- Tested with 4096 levels under a small thread stack + +#### T2c: Many Elements / Slow Input (CPU Exhaustion) +- `timeout` (default 60 s) is one wall-clock budget per call covering URL fetch, + parsing and rendering. Checked cooperatively in the HTML walk, XML tree build, DOCX + conversion, spreadsheet cell loops and renderer. html5ever tokenizing and calamine + sheet decoding are not interruptible; they are linear in the (capped) input and the + deadline is checked right after them +- `max_nodes` (default 5 000 000) caps HTML DOM nodes and DOCX XML nodes (summed over parts) + +#### T2e: Output Amplification +- GFM tables pad every row to the widest row, so output can be rows x columns of input +- `max_output_size` (default 128 MiB) checked per block, list item and table row, + including nested renderers (blockquotes, list items), so no buffer grows past the + limit by more than one row #### T2d: Billion Laughs / Entity Expansion - html5ever handles entity expansion internally @@ -85,6 +100,9 @@ tags: - `html_security::resource_limit_bytes` - `html_integration::resource_limit_enforced` - `html_proptest::resource_limit_enforced` +- `limits::*` (every `Limits` field: output cap incl. table amplification, timeout, + node cap, configurable depth, hard depth ceiling on a small stack, for HTML, DOCX and + spreadsheets) ### T3: Server-Side Request Forgery (SSRF) @@ -115,7 +133,7 @@ and `http_client` is a **default** feature. - Redirects capped by `max_redirects` (default 5) - Response body bounded by `max_input_size`: declared `Content-Length` rejected before reading, undeclared bodies stopped while streaming (`ResourceLimit`) -- Whole fetch (connect, headers, body) bounded by `max_processing_secs` +- Whole fetch (connect, headers, body) bounded by `timeout`; parse and render share the remaining budget - Opt-out: embedders that fetch content themselves build with `default-features = false` and never link reqwest (CI asserts this for `docx,spreadsheet`) @@ -145,6 +163,7 @@ the proxy, or use `allow_host`. **Mitigations**: - Input size limit (pre-parse) - Depth limit prevents unbounded tree growth +- `max_nodes` caps parsed tree size; `max_output_size` caps the Markdown buffer - `String::with_capacity(4096)` for renderer output buffer (bounded initial allocation) **Test Coverage**: @@ -236,17 +255,17 @@ process, or smuggling content past a human reviewer. | Resource | Default | Configurable | Enforcement Point | |----------|---------|--------------|-------------------| | Input size | 50 MB | `max_input_size(usize)` | Before parsing | -| Processing time | 60 s | `max_processing_secs(u64)` | Whole `extract_url` fetch (parsing itself TBD) | +| Output size | 128 MiB | `max_output_size(usize)` | Renderer, per block / list item / table row | +| Processing time | 60 s | `timeout(Duration)`, `max_processing_secs(u64)` | One budget per call: fetch, parse, render (cooperative) | +| Parsed nodes | 5 000 000 | `max_nodes(usize)` | After html5ever (DOM count); while building DOCX XML trees | +| Nesting depth | 256 (hard max 1024) | `max_nesting_depth(usize)` | HTML walk (flatten); DOCX XML build (reject) | | Redirects (`extract_url`) | 5 | `max_redirects(usize)` | Every redirect hop | | Fetched body (`extract_url`) | = input size | `max_input_size(usize)` | `Content-Length` and while streaming | -| Nesting depth | 200 | Hardcoded `MAX_DEPTH` | During HTML tree walk | -| XML nesting depth (DOCX) | 256 | Hardcoded `MAX_XML_DEPTH` | While building the XML tree | | ZIP entries | 10 000 | `max_archive_entries(usize)` | On archive open | | ZIP entry size (decompressed) | 64 MiB | `max_archive_entry_size(u64)` | While decompressing | | ZIP total size (decompressed) | 256 MiB | `max_archive_total_size(u64)` | While decompressing | | Spreadsheet grid (XLS/ODS) | 4 000 000 cells | `max_spreadsheet_cells(u64)` | Pre-scan before calamine | | Rows / columns per sheet | 10 000 / 256 | `max_sheet_rows`, `max_sheet_cols` | While collecting cells (truncate + warn) | -| Output size | Unbounded | N/A | Proportional to input size | ## Malicious Input Categories @@ -257,7 +276,7 @@ process, or smuggling content past a human reviewer. | JavaScript URLs | `` | URL preserved as-is in markdown (no execution context) | | Data URI payloads | `` | Rendered as markdown image (no execution) | | Style expressions | `style="background:url(javascript:...)"` | Attribute not read | -| Deep nesting | 1000+ nested `
      ` | Flattened to text at depth 200 | +| Deep nesting | 1000+ nested `
      ` | Flattened to text at `max_nesting_depth` (256) | | Huge text nodes | 100KB+ single text node | Passed through (bounded by input limit) | | Null bytes | `\0` in content | Replaced with U+FFFD by html5ever | | Unclosed tags | `

      text

      more` | html5ever auto-closes per spec | @@ -285,6 +304,9 @@ process, or smuggling content past a human reviewer. | S10 | Spreadsheet parsing never allocates a grid beyond `max_spreadsheet_cells` | `document_security::xls_*_rejected`, `ods_repeated_rows_and_columns_rejected` | | S11 | Unsupported formats fail explicitly (no empty success) | `document_detection::pdf_is_explicitly_unsupported_not_empty` | | S12 | Default `extract_url` never connects to a non-public address | `url_security::*_blocked*`, `fetch::tests::non_public_addresses_forbidden` | +| S13 | Returned Markdown never exceeds `max_output_size` | `limits::*output_cap*` | +| S14 | No configuration of `max_nesting_depth` can overflow the stack | `limits::*hard_ceiling_survives_small_stack` | +| S15 | Extraction stops with `ResourceLimit` once `timeout` passes (checked at least after each non-interruptible third-party step) | `limits::*zero_timeout_rejects`, `limits::tests::*` | ## See also From cac90317dff668cfe3b879c77e34a96770f3437f Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Mon, 26 Oct 2026 23:49:26 +0000 Subject: [PATCH 2/5] fix(limits): drop private intra-doc link to Budget --- crates/markkit/src/limits.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/markkit/src/limits.rs b/crates/markkit/src/limits.rs index 4725e96..4ad8ac9 100644 --- a/crates/markkit/src/limits.rs +++ b/crates/markkit/src/limits.rs @@ -12,7 +12,7 @@ //! truncated output. Exceptions, kept for compatibility: per-sheet rows and //! columns truncate with a warning, and HTML nesting past //! `max_nesting_depth` is flattened to plain text. -//! - Time is checked cooperatively ([`Budget`]) in parser and renderer loops. +//! - Time is checked cooperatively (`Budget`) in parser and renderer loops. //! Third-party parse steps (html5ever tokenizing, calamine cell decoding) are //! not interruptible; they are bounded by input size and checked right after. //! - `max_nesting_depth` is clamped to [`HARD_MAX_NESTING_DEPTH`] because the From e0f8b40e0520ab6a96f52607471890e29264e876 Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Mon, 26 Oct 2026 23:52:29 +0000 Subject: [PATCH 3/5] fix(limits): lower nesting hard ceiling to 512 for release stack headroom --- CHANGELOG.md | 2 +- README.md | 2 +- crates/markkit/src/limits.rs | 9 +++++---- knowledge/formats/html-extraction.md | 2 +- knowledge/foundations/architecture.md | 2 +- knowledge/foundations/ir.md | 2 +- knowledge/log.md | 2 +- knowledge/security/threat-model.md | 7 ++++--- 8 files changed, 15 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ade409..31d1c96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/). per-field setters). New: `timeout` (default 60 s) now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB) stops output amplification (e.g. one wide table row padding thousands of rows); `max_nodes` (5 000 000) caps parsed HTML DOM / DOCX XML nodes; - `max_nesting_depth` (256) is configurable, with a hard ceiling of 1024 that keeps deep input off + `max_nesting_depth` (256) is configurable, with a hard ceiling of 512 that keeps deep input off the host stack. Violations return `Error::ResourceLimit` naming the limit. - `extract_url` (CLI, Python and Node `extract_url` too) now refuses non-public destinations by default: diff --git a/README.md b/README.md index 9faf2e0..798a0ea 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,7 @@ generous; violations return `Error::ResourceLimit` naming the limit. | `max_input_size` | 50 MiB | Input bytes, fetched URL body | | `max_output_size` | 128 MiB | Markdown output | | `timeout` | 60 s | Fetch + parse + render, per call | -| `max_nesting_depth` | 256 (hard max 1024) | HTML flattens deeper content; deeper DOCX XML is rejected | +| `max_nesting_depth` | 256 (hard max 512) | HTML flattens deeper content; deeper DOCX XML is rejected | | `max_nodes` | 5 000 000 | HTML DOM nodes, DOCX XML nodes | | `max_archive_entries` | 10 000 | ZIP entries (DOCX/XLSX/ODS) | | `max_archive_entry_size` | 64 MiB | Decompressed bytes per ZIP entry | diff --git a/crates/markkit/src/limits.rs b/crates/markkit/src/limits.rs index 4ad8ac9..49d1c3a 100644 --- a/crates/markkit/src/limits.rs +++ b/crates/markkit/src/limits.rs @@ -26,11 +26,12 @@ use std::time::{Duration, Instant}; use crate::error::Error; // THREAT[T2b]: recursion depth must stay safe on small host stacks whatever -// the embedder configures. Measured worst case (nested blockquote / inline -// tags): 1024 levels fit a 2 MiB thread in release; debug frames are ~6x -// larger (default 256 fits 2 MiB in debug). Test: limits::*hard_ceiling*. +// the embedder configures. Measured worst case (nested blockquotes, renderer +// recursion): 768 levels fit a 2 MiB thread in release, 1024 do not; 512 keeps +// ~2x headroom. Debug frames are ~6x larger (default 256 fits 2 MiB in debug). +// Test: limits::*hard_ceiling*. /// Upper bound applied to [`Limits::max_nesting_depth`]. -pub const HARD_MAX_NESTING_DEPTH: usize = 1024; +pub const HARD_MAX_NESTING_DEPTH: usize = 512; /// Resource limits applied to every extraction. #[derive(Debug, Clone, PartialEq, Eq)] diff --git a/knowledge/formats/html-extraction.md b/knowledge/formats/html-extraction.md index adf19ef..8ad911b 100644 --- a/knowledge/formats/html-extraction.md +++ b/knowledge/formats/html-extraction.md @@ -129,7 +129,7 @@ Trimming at render time: ## Depth Limiting -Limit: `limits.max_nesting_depth` (default 256, clamped to `HARD_MAX_NESTING_DEPTH` = 1024) +Limit: `limits.max_nesting_depth` (default 256, clamped to `HARD_MAX_NESTING_DEPTH` = 512) Behavior at depth limit: - `convert_children`: extracts all descendant text via `.text().collect()`, trims, emits as single `Text` node diff --git a/knowledge/foundations/architecture.md b/knowledge/foundations/architecture.md index 82c0c47..57d51aa 100644 --- a/knowledge/foundations/architecture.md +++ b/knowledge/foundations/architecture.md @@ -49,7 +49,7 @@ Input (bytes/URL/file) Via `MarkkitBuilder`: - Resource limits: one `Limits` struct (`limits.rs`, set whole via `.limits(..)` or per - field via builder setters): input/output size, timeout, nesting depth (hard max 1024), + field via builder setters): input/output size, timeout, nesting depth (hard max 512), node count, archive, spreadsheet and redirect caps. Violations are `Error::ResourceLimit`; see [Threat Model](../security/threat-model.md) Resource Limits - CSS selectors for include/exclude diff --git a/knowledge/foundations/ir.md b/knowledge/foundations/ir.md index 3c0d937..253ea35 100644 --- a/knowledge/foundations/ir.md +++ b/knowledge/foundations/ir.md @@ -138,7 +138,7 @@ At render time, the `normalize_output` function: | No empty Text nodes after whitespace collapse | `collapse_whitespace` returns empty string for whitespace-only; parser skips empty | | Block lists hold no bare inline runs | `normalize_blocks` drops whitespace-only runs, wraps others in `Paragraph` | | Empty containers are elided | Parser returns `None` for headings, paragraphs, lists, etc. with no children | -| Depth bounded (`limits.max_nesting_depth`, default 256, hard max 1024) | HTML parser flattens the subtree to text at the limit | +| Depth bounded (`limits.max_nesting_depth`, default 256, hard max 512) | HTML parser flattens the subtree to text at the limit | | No executable content in IR | Script, style, iframe, etc. stripped by parser | | `List.start` is `None` when `ordered=false` | Parser only reads `start` attr for `

        ` | | Table spans ≥ 1 | `TableCell::with_span` clamps 0 to 1; HTML parser clamps to 1000 / 65534 | diff --git a/knowledge/log.md b/knowledge/log.md index e8d5716..af349b0 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,7 +2,7 @@ ## 2026-10-26 -* **Resource limits**: all limits moved into one `Limits` struct (bashkit `ExecutionLimits` style). New: `timeout` now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB, stops table amplification); `max_nodes` (5M); `max_nesting_depth` configurable (256, was hardcoded 200 HTML / 256 XML) with a 1024 hard ceiling. Invariants S14-S16, `limits` tests. See [Threat Model](security/threat-model.md) T2. +* **Resource limits**: all limits moved into one `Limits` struct (bashkit `ExecutionLimits` style). New: `timeout` now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB, stops table amplification); `max_nodes` (5M); `max_nesting_depth` configurable (256, was hardcoded 200 HTML / 256 XML) with a 512 hard ceiling. Invariants S14-S16, `limits` tests. See [Threat Model](security/threat-model.md) T2. * **IR v2**: footnotes, definition lists, task lists, figures, table alignment and spans; `Document.ir_version` (`IR_VERSION = 2`), `#[non_exhaustive]` IR types, `Node::is_block`/`is_inline`. HTML maps GitHub/Pandoc footnotes, `
        `, checkbox items, captioned figures, `colspan`/`rowspan`/alignment and row-header ``. New threat T2e and invariant S13 (span grid budget, footnote label sanitizing). See [IR Specification](foundations/ir.md), [HTML Extraction](formats/html-extraction.md), [Threat Model](security/threat-model.md). * **URL fetch hardening**: T3 SSRF is mitigated. `extract_url` blocks non-public addresses by default (URL, resolver and redirect layers), enforces scheme, allowlist, redirect, body and time limits; new invariant S12 and `url_security` tests. Proxy-resolved names are the documented residual. See [Threat Model](security/threat-model.md) and [Security Model](security/security-model.md). diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index f48730b..ab2480f 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -70,8 +70,9 @@ All limits live in one `Limits` struct (`crates/markkit/src/limits.rs`, #### T2b: Deep Nesting (Stack Overflow) - `max_nesting_depth` (default 256) bounds recursion: HTML flattens deeper subtrees to plain text; DOCX XML parts deeper than it are rejected -- Clamped to `HARD_MAX_NESTING_DEPTH` (1024) whatever the embedder sets: 1024 levels - of the worst recursive constructs fit a 2 MiB stack in release (debug frames are ~6x) +- Clamped to `HARD_MAX_NESTING_DEPTH` (512) whatever the embedder sets: 512 levels + of the worst recursive construct (nested blockquotes) fit a 2 MiB stack in release + with ~2x headroom (1024 did not after IR v2); debug frames are ~6x larger - Tested with 4096 levels under a small thread stack #### T2c: Many Elements / Slow Input (CPU Exhaustion) @@ -266,7 +267,7 @@ process, or smuggling content past a human reviewer. | Output size | 128 MiB | `max_output_size(usize)` | Renderer, per block / list item / table row | | Processing time | 60 s | `timeout(Duration)`, `max_processing_secs(u64)` | One budget per call: fetch, parse, render (cooperative) | | Parsed nodes | 5 000 000 | `max_nodes(usize)` | After html5ever (DOM count); while building DOCX XML trees | -| Nesting depth | 256 (hard max 1024) | `max_nesting_depth(usize)` | HTML walk (flatten); DOCX XML build (reject) | +| Nesting depth | 256 (hard max 512) | `max_nesting_depth(usize)` | HTML walk (flatten); DOCX XML build (reject) | | Redirects (`extract_url`) | 5 | `max_redirects(usize)` | Every redirect hop | | Fetched body (`extract_url`) | = input size | `max_input_size(usize)` | `Content-Length` and while streaming | | Table span grid | 1 000 000 slots | Hardcoded `MAX_TABLE_GRID_CELLS` | Renderer table layout (spans dropped past it) | From dd0be45ca1d21a22d198053876976a914a5dd766 Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Tue, 27 Oct 2026 00:56:49 +0000 Subject: [PATCH 4/5] feat(bindings): expose every resource limit in CLI, Python, Node and WASM CLI: --max-output-size, --timeout, --max-nesting-depth, --max-nodes plus archive, sheet and redirect flags under a Limits help heading. Python: markkit.Limits passed as limits= to every function. Node: MarkkitOptions gains timeout (seconds), maxOutputSize, maxNodes, maxNestingDepth. WASM Options gains maxOutputSize, maxNodes, maxNestingDepth (no timeout: no clock on wasm32). --- CHANGELOG.md | 6 +- README.md | 4 +- crates/markkit-cli/src/main.rs | 143 ++++++++++++- crates/markkit-cli/tests/cli_integration.rs | 21 ++ crates/markkit-js/README.md | 4 +- crates/markkit-js/__test__/options.test.cjs | 12 ++ crates/markkit-js/src/lib.rs | 36 +++- crates/markkit-python/README.md | 9 + crates/markkit-python/src/lib.rs | 201 +++++++++++++++--- crates/markkit-wasm/README.md | 5 +- .../__test__/markkit-wasm.test.mjs | 9 + crates/markkit-wasm/js/index.d.ts | 6 + crates/markkit-wasm/src/lib.rs | 9 + docs/getting-started.md | 6 + knowledge/log.md | 3 +- knowledge/runtimes/node-package.md | 6 +- knowledge/runtimes/python-package.md | 8 + knowledge/runtimes/wasm-package.md | 3 +- 18 files changed, 446 insertions(+), 45 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8cf3d53..8a550f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,6 +48,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/). the article body of a web page, Readability-style, dropping navigation, sidebars, comments, share bars and related-link blocks. Off by default; include selectors take precedence; falls back to the full page when no article-like block is found. +- Every resource limit is configurable outside Rust: CLI flags (`--max-output-size`, `--timeout`, + `--max-nesting-depth`, `--max-nodes`, archive / sheet / redirect flags; `--max-size` kept as an + alias of `--max-input-size`), Python `markkit.Limits(...)` passed as `limits=` to every function, + and flat Node.js and WASM option keys (`maxOutputSize`, `maxNodes`, `maxNestingDepth`, plus + `timeout` in Node.js). ### Security @@ -57,7 +62,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/). thousands of rows); `max_nodes` (5 000 000) caps parsed HTML DOM / DOCX XML nodes; `max_nesting_depth` (256) is configurable, with a hard ceiling of 512 that keeps deep input off the host stack. Violations return `Error::ResourceLimit` naming the limit. - - `extract_url` (CLI, Python and Node `extract_url` too) now refuses non-public destinations by default: loopback, private, link-local / cloud metadata and other reserved addresses, checked on the URL, after DNS resolution and on every redirect. Only `http`/`https` are fetched. Redirects are capped diff --git a/README.md b/README.md index 768ec74..e8a273d 100644 --- a/README.md +++ b/README.md @@ -223,7 +223,9 @@ let markkit = Markkit::builder() .build(); ``` -Each limit also has a builder setter (`.max_nodes(..)`, `.timeout(..)`, ...). +Each limit also has a builder setter (`.max_nodes(..)`, `.timeout(..)`, ...), a CLI +flag (`--max-nodes`, `--timeout`, ...; see `markkit extract --help`), a +`markkit.Limits(...)` argument in Python and flat option keys in Node.js and WASM. Other options: `include_hidden_sheets`, `allow_private_networks`, `allow_host`. ## Performance diff --git a/crates/markkit-cli/src/main.rs b/crates/markkit-cli/src/main.rs index 36abb6d..dfef440 100644 --- a/crates/markkit-cli/src/main.rs +++ b/crates/markkit-cli/src/main.rs @@ -1,8 +1,10 @@ use std::io::Read; use anyhow::Result; -use clap::{Parser, Subcommand}; -use markkit::Markkit; +use std::time::Duration; + +use clap::{Args, Parser, Subcommand}; +use markkit::{Limits, Markkit}; #[derive(Parser)] #[command( @@ -38,10 +40,6 @@ enum Commands { #[arg(long = "exclude", short = 'e')] exclude_selectors: Vec, - /// Maximum input size in bytes - #[arg(long, default_value = "52428800")] - max_size: usize, - /// Output as JSON with metadata #[arg(long)] json: bool, @@ -54,9 +52,82 @@ enum Commands { /// Keep only the detected main content (article body) of HTML pages #[arg(long)] main_content: bool, + + #[command(flatten)] + limits: LimitArgs, }, } +/// Resource limits; unset flags keep the library defaults (see `markkit::Limits`). +#[derive(Args, Debug, Default)] +#[command(next_help_heading = "Limits")] +struct LimitArgs { + /// Maximum input (and fetched body) size in bytes [default: 50 MiB] + #[arg(long, alias = "max-size", value_name = "BYTES")] + max_input_size: Option, + /// Maximum Markdown output size in bytes [default: 128 MiB] + #[arg(long, value_name = "BYTES")] + max_output_size: Option, + /// Time budget for fetch + parse + render, in seconds [default: 60] + #[arg(long, value_name = "SECS", value_parser = parse_secs)] + timeout: Option, + /// Maximum element nesting, hard-capped at 512 [default: 256] + #[arg(long, value_name = "N")] + max_nesting_depth: Option, + /// Maximum parsed HTML DOM / DOCX XML nodes [default: 5000000] + #[arg(long, value_name = "N")] + max_nodes: Option, + /// Maximum entries in a ZIP container [default: 10000] + #[arg(long, value_name = "N")] + max_archive_entries: Option, + /// Maximum decompressed bytes per ZIP entry [default: 64 MiB] + #[arg(long, value_name = "BYTES")] + max_archive_entry_size: Option, + /// Maximum decompressed bytes per ZIP container [default: 256 MiB] + #[arg(long, value_name = "BYTES")] + max_archive_total_size: Option, + /// Maximum data rows rendered per sheet [default: 10000] + #[arg(long, value_name = "N")] + max_sheet_rows: Option, + /// Maximum columns rendered per sheet [default: 256] + #[arg(long, value_name = "N")] + max_sheet_cols: Option, + /// Maximum XLS/ODS cell grid [default: 4000000] + #[arg(long, value_name = "N")] + max_spreadsheet_cells: Option, + /// Maximum redirects followed when fetching a URL [default: 5] + #[arg(long, value_name = "N")] + max_redirects: Option, +} + +impl LimitArgs { + fn apply(&self, mut l: Limits) -> Limits { + macro_rules! set { + ($($f:ident),*) => {$( if let Some(v) = self.$f { l.$f = v; } )*}; + } + set!( + max_input_size, + max_output_size, + timeout, + max_nesting_depth, + max_nodes, + max_archive_entries, + max_archive_entry_size, + max_archive_total_size, + max_sheet_rows, + max_sheet_cols, + max_spreadsheet_cells, + max_redirects + ); + l + } +} + +fn parse_secs(s: &str) -> Result { + let secs: f64 = s.parse().map_err(|e| format!("{e}"))?; + Duration::try_from_secs_f64(secs).map_err(|e| format!("{e}")) +} + #[tokio::main] async fn main() -> Result<()> { let cli = Cli::parse(); @@ -68,7 +139,7 @@ async fn main() -> Result<()> { stdin, include_selectors, exclude_selectors, - max_size, + limits, json: output_json, base_url, main_content, @@ -80,7 +151,7 @@ async fn main() -> Result<()> { } let mut builder = Markkit::builder() - .max_input_size(max_size) + .limits(limits.apply(Limits::default())) .main_content(main_content); for sel in &include_selectors { @@ -141,4 +212,60 @@ mod tests { fn verify_cli() { Cli::command().debug_assert(); } + + fn limits_from(args: &[&str]) -> Limits { + let cli = Cli::try_parse_from(["markkit", "extract", "x"].iter().chain(args)).unwrap(); + let Commands::Extract { limits, .. } = cli.command; + limits.apply(Limits::default()) + } + + #[test] + fn unset_limit_flags_keep_defaults() { + assert_eq!(limits_from(&[]), Limits::default()); + } + + #[test] + fn every_limit_flag_maps() { + let l = limits_from(&[ + "--max-input-size=1", + "--max-output-size=2", + "--timeout=0.5", + "--max-nesting-depth=3", + "--max-nodes=4", + "--max-archive-entries=5", + "--max-archive-entry-size=6", + "--max-archive-total-size=7", + "--max-sheet-rows=8", + "--max-sheet-cols=9", + "--max-spreadsheet-cells=10", + "--max-redirects=11", + ]); + let want = Limits::default() + .max_input_size(1) + .max_output_size(2) + .timeout(Duration::from_millis(500)) + .max_nesting_depth(3) + .max_nodes(4) + .max_archive_entries(5) + .max_archive_entry_size(6) + .max_archive_total_size(7) + .max_sheet_rows(8) + .max_sheet_cols(9) + .max_spreadsheet_cells(10) + .max_redirects(11); + assert_eq!(l, want); + } + + #[test] + fn legacy_max_size_alias() { + assert_eq!(limits_from(&["--max-size=9"]).max_input_size, 9); + } + + #[test] + fn bad_timeout_rejected() { + for bad in ["-1", "nan", "abc"] { + let arg = format!("--timeout={bad}"); + assert!(Cli::try_parse_from(["markkit", "extract", "x", &arg]).is_err()); + } + } } diff --git a/crates/markkit-cli/tests/cli_integration.rs b/crates/markkit-cli/tests/cli_integration.rs index 2d50944..1b2e9c8 100644 --- a/crates/markkit-cli/tests/cli_integration.rs +++ b/crates/markkit-cli/tests/cli_integration.rs @@ -199,3 +199,24 @@ fn missing_source_errors() { assert!(!output.status.success()); } + +#[test] +fn limit_flags_enforced() { + let mut child = markkit_bin() + .args(["extract", "--stdin", "--max-output-size", "5"]) + .stdin(std::process::Stdio::piped()) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .expect("failed to spawn markkit"); + child + .stdin + .take() + .unwrap() + .write_all(b"

        hello world, this is long

        ") + .unwrap(); + let output = child.wait_with_output().unwrap(); + assert!(!output.status.success()); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains("max_output_size"), "stderr: {stderr}"); +} diff --git a/crates/markkit-js/README.md b/crates/markkit-js/README.md index a8d07e7..4cd98be 100644 --- a/crates/markkit-js/README.md +++ b/crates/markkit-js/README.md @@ -33,7 +33,9 @@ const doc = extractDocumentSync(fs.readFileSync('report.docx'), { fileName: 'rep console.log(doc.title, doc.format, doc.warnings, doc.markdown); ``` -Fields: `maxInputSize`, `maxProcessingSecs`, `preserveLinks`, `preserveImages`, +Fields: `maxInputSize`, `maxOutputSize`, `timeout` (seconds, covers fetch, parsing and +rendering; `maxProcessingSecs` is the whole-second form), `maxNodes`, `maxNestingDepth`, +`preserveLinks`, `preserveImages`, `includeSelectors`, `excludeSelectors` (added to the defaults), `maxArchiveEntries`, `maxArchiveEntrySize`, `maxArchiveTotalSize`, `maxSheetRows`, `maxSheetCols`, `maxSpreadsheetCells`, `includeHiddenSheets`, `allowPrivateNetworks` (trusted URLs only), diff --git a/crates/markkit-js/__test__/options.test.cjs b/crates/markkit-js/__test__/options.test.cjs index 1d393a8..81923a7 100644 --- a/crates/markkit-js/__test__/options.test.cjs +++ b/crates/markkit-js/__test__/options.test.cjs @@ -24,9 +24,21 @@ test("input size limit", () => { assert.throws(() => m.extractHtmlSync("

        too long

        ", { maxInputSize: 4 }), /Resource limit/); }); +test("resource limit options", () => { + const html = "

        hello world

        ".repeat(200); + assert.throws(() => m.extractHtmlSync(html, { maxOutputSize: 100 }), /max_output_size/); + assert.throws(() => m.extractHtmlSync(html, { maxNodes: 50 }), /max_nodes/); + assert.throws(() => m.extractHtmlSync(html, { timeout: 0 }), /timeout/); + assert.ok(m.extractHtmlSync(html, { timeout: 1e300, maxOutputSize: 1 << 20, maxNodes: 10000 }).includes("hello")); + const deep = "
        ".repeat(20) + "

        Deep

        " + "
        ".repeat(20); + assert.ok(!m.extractHtmlSync(deep, { maxNestingDepth: 5 }).includes("> ".repeat(6))); +}); + test("invalid numeric options are rejected", () => { assert.throws(() => m.extractHtmlSync("

        x

        ", { maxArchiveTotalSize: -1 }), /invalid options/); assert.throws(() => m.extractHtmlSync("

        x

        ", { maxSpreadsheetCells: 1.5 }), /invalid options/); + assert.throws(() => m.extractHtmlSync("

        x

        ", { timeout: -1 }), /invalid options/); + assert.throws(() => m.extractHtmlSync("

        x

        ", { timeout: NaN }), /invalid options/); }); test("extractDocument returns metadata", async () => { diff --git a/crates/markkit-js/src/lib.rs b/crates/markkit-js/src/lib.rs index d7d5ea6..a8e43ad 100644 --- a/crates/markkit-js/src/lib.rs +++ b/crates/markkit-js/src/lib.rs @@ -17,8 +17,17 @@ use napi_derive::napi; pub struct MarkkitOptions { /// Maximum input size in bytes (default 50 MiB). pub max_input_size: Option, - /// Upper bound for `extractUrl` in seconds (default 60). + /// Wall-clock budget per call in whole seconds (default 60). Prefer `timeout`. pub max_processing_secs: Option, + /// Wall-clock budget per call in seconds, fractions allowed (default 60); + /// covers URL fetch, parsing and rendering. Wins over `maxProcessingSecs`. + pub timeout: Option, + /// Maximum Markdown output in bytes (default 128 MiB). + pub max_output_size: Option, + /// Maximum parsed nodes (default 5 000 000). + pub max_nodes: Option, + /// Maximum element nesting (default 256, capped at 512). + pub max_nesting_depth: Option, pub preserve_links: Option, pub preserve_images: Option, /// CSS selectors to include (empty = whole document). @@ -98,6 +107,19 @@ fn to_u64(v: f64, field: &str) -> napi::Result { } } +/// Seconds as a JS number; non-negative (values too large for `Duration` mean no deadline). +fn to_duration(v: f64) -> napi::Result { + std::time::Duration::try_from_secs_f64(v).or_else(|_| { + if v.is_nan() || v < 0.0 { + Err(napi::Error::from_reason( + "invalid options: `timeout` must be a non-negative number of seconds", + )) + } else { + Ok(std::time::Duration::MAX) + } + }) +} + fn markkit(options: Option) -> napi::Result { let o = options.unwrap_or_default(); let mut b = markkit::Markkit::builder(); @@ -107,6 +129,18 @@ fn markkit(options: Option) -> napi::Result { if let Some(v) = o.max_processing_secs { b = b.max_processing_secs(v.into()); } + if let Some(v) = o.timeout { + b = b.timeout(to_duration(v)?); + } + if let Some(v) = o.max_output_size { + b = b.max_output_size(v as usize); + } + if let Some(v) = o.max_nodes { + b = b.max_nodes(v as usize); + } + if let Some(v) = o.max_nesting_depth { + b = b.max_nesting_depth(v as usize); + } if let Some(v) = o.preserve_links { b = b.preserve_links(v); } diff --git a/crates/markkit-python/README.md b/crates/markkit-python/README.md index 3b4ca3b..437da4c 100644 --- a/crates/markkit-python/README.md +++ b/crates/markkit-python/README.md @@ -17,8 +17,17 @@ import markkit print(markkit.extract_html_sync("

        Hello

        ")) # # Hello + +# Resource limits (all optional, generous defaults); breaches raise RuntimeError +limits = markkit.Limits(timeout=10.0, max_output_size=8 << 20, max_nesting_depth=128) +markkit.extract_html_sync("

        Hello

        ", limits=limits) ``` +`Limits` fields: `max_input_size`, `max_output_size`, `timeout` (seconds), +`max_nesting_depth`, `max_nodes`, `max_archive_entries`, `max_archive_entry_size`, +`max_archive_total_size`, `max_sheet_rows`, `max_sheet_cols`, +`max_spreadsheet_cells`, `max_redirects`. Every function accepts `limits=`. + Publication verifies wheel metadata, installs the actual wheel into a clean virtual environment and exercises native extraction before upload. diff --git a/crates/markkit-python/src/lib.rs b/crates/markkit-python/src/lib.rs index 7f3c744..674bff2 100644 --- a/crates/markkit-python/src/lib.rs +++ b/crates/markkit-python/src/lib.rs @@ -1,22 +1,162 @@ +use std::time::Duration; + +use pyo3::exceptions::{PyRuntimeError, PyValueError}; use pyo3::prelude::*; +/// Resource limits (mirrors `markkit::Limits`). Every argument is optional and +/// defaults to the library default; violations raise `RuntimeError` naming +/// the limit. +/// +/// Keyword arguments: max_input_size, max_output_size, timeout (seconds, +/// float), max_nesting_depth, max_nodes, max_archive_entries, +/// max_archive_entry_size, max_archive_total_size, max_sheet_rows, +/// max_sheet_cols, max_spreadsheet_cells, max_redirects. +#[pyclass(name = "Limits", module = "markkit", frozen, get_all)] +#[derive(Clone)] +struct Limits { + max_input_size: usize, + max_output_size: usize, + timeout: f64, + max_nesting_depth: usize, + max_nodes: usize, + max_archive_entries: usize, + max_archive_entry_size: u64, + max_archive_total_size: u64, + max_sheet_rows: usize, + max_sheet_cols: usize, + max_spreadsheet_cells: u64, + max_redirects: usize, +} + +#[pymethods] +impl Limits { + #[new] + #[pyo3(signature = ( + *, + max_input_size=None, + max_output_size=None, + timeout=None, + max_nesting_depth=None, + max_nodes=None, + max_archive_entries=None, + max_archive_entry_size=None, + max_archive_total_size=None, + max_sheet_rows=None, + max_sheet_cols=None, + max_spreadsheet_cells=None, + max_redirects=None, + ))] + #[allow(clippy::too_many_arguments)] + fn new( + max_input_size: Option, + max_output_size: Option, + timeout: Option, + max_nesting_depth: Option, + max_nodes: Option, + max_archive_entries: Option, + max_archive_entry_size: Option, + max_archive_total_size: Option, + max_sheet_rows: Option, + max_sheet_cols: Option, + max_spreadsheet_cells: Option, + max_redirects: Option, + ) -> PyResult { + let d = markkit::Limits::default(); + if let Some(t) = timeout { + Duration::try_from_secs_f64(t) + .map_err(|e| PyValueError::new_err(format!("timeout: {e}")))?; + } + Ok(Self { + max_input_size: max_input_size.unwrap_or(d.max_input_size), + max_output_size: max_output_size.unwrap_or(d.max_output_size), + timeout: timeout.unwrap_or(d.timeout.as_secs_f64()), + max_nesting_depth: max_nesting_depth.unwrap_or(d.max_nesting_depth), + max_nodes: max_nodes.unwrap_or(d.max_nodes), + max_archive_entries: max_archive_entries.unwrap_or(d.max_archive_entries), + max_archive_entry_size: max_archive_entry_size.unwrap_or(d.max_archive_entry_size), + max_archive_total_size: max_archive_total_size.unwrap_or(d.max_archive_total_size), + max_sheet_rows: max_sheet_rows.unwrap_or(d.max_sheet_rows), + max_sheet_cols: max_sheet_cols.unwrap_or(d.max_sheet_cols), + max_spreadsheet_cells: max_spreadsheet_cells.unwrap_or(d.max_spreadsheet_cells), + max_redirects: max_redirects.unwrap_or(d.max_redirects), + }) + } + + fn __repr__(&self) -> String { + format!( + "Limits(max_input_size={}, max_output_size={}, timeout={}, max_nesting_depth={}, \ + max_nodes={}, max_archive_entries={}, max_archive_entry_size={}, \ + max_archive_total_size={}, max_sheet_rows={}, max_sheet_cols={}, \ + max_spreadsheet_cells={}, max_redirects={})", + self.max_input_size, + self.max_output_size, + self.timeout, + self.max_nesting_depth, + self.max_nodes, + self.max_archive_entries, + self.max_archive_entry_size, + self.max_archive_total_size, + self.max_sheet_rows, + self.max_sheet_cols, + self.max_spreadsheet_cells, + self.max_redirects + ) + } +} + +impl Limits { + fn to_core(&self) -> markkit::Limits { + markkit::Limits::default() + .max_input_size(self.max_input_size) + .max_output_size(self.max_output_size) + // Validated in `new`. + .timeout(Duration::try_from_secs_f64(self.timeout).unwrap_or(Duration::ZERO)) + .max_nesting_depth(self.max_nesting_depth) + .max_nodes(self.max_nodes) + .max_archive_entries(self.max_archive_entries) + .max_archive_entry_size(self.max_archive_entry_size) + .max_archive_total_size(self.max_archive_total_size) + .max_sheet_rows(self.max_sheet_rows) + .max_sheet_cols(self.max_sheet_cols) + .max_spreadsheet_cells(self.max_spreadsheet_cells) + .max_redirects(self.max_redirects) + } +} + +fn builder(limits: Option<&Limits>) -> markkit::MarkkitBuilder { + let b = markkit::Markkit::builder(); + match limits { + Some(l) => b.limits(l.to_core()), + None => b, + } +} + +fn err(e: markkit::Error) -> PyErr { + PyRuntimeError::new_err(e.to_string()) +} + /// Extract markdown from an HTML string (async). #[pyfunction] -fn extract_html(py: Python<'_>, html: String) -> PyResult> { +#[pyo3(signature = (html, *, limits=None))] +fn extract_html<'py>( + py: Python<'py>, + html: String, + limits: Option>, +) -> PyResult> { + let m = builder(limits.as_deref()).build(); pyo3_async_runtimes::tokio::future_into_py(py, async move { - let m = markkit::Markkit::builder().build(); - m.extract_html(&html) - .await - .map_err(|e| PyErr::new::(e.to_string())) + m.extract_html(&html).await.map_err(err) }) } /// Extract markdown from an HTML string (synchronous). #[pyfunction] -fn extract_html_sync(html: String) -> PyResult { - let m = markkit::Markkit::builder().build(); - m.extract_html_sync(&html) - .map_err(|e| PyErr::new::(e.to_string())) +#[pyo3(signature = (html, *, limits=None))] +fn extract_html_sync(html: String, limits: Option>) -> PyResult { + builder(limits.as_deref()) + .build() + .extract_html_sync(&html) + .map_err(err) } /// Extract markdown from an HTML string with configuration options (synchronous). @@ -24,16 +164,18 @@ fn extract_html_sync(html: String) -> PyResult { /// Keyword arguments: /// include_selectors: list of CSS selectors to include /// exclude_selectors: list of CSS selectors to exclude -/// max_input_size: maximum input size in bytes +/// max_input_size: maximum input size in bytes (overrides limits.max_input_size) +/// limits: markkit.Limits #[pyfunction] -#[pyo3(signature = (html, *, include_selectors=None, exclude_selectors=None, max_input_size=None))] +#[pyo3(signature = (html, *, include_selectors=None, exclude_selectors=None, max_input_size=None, limits=None))] fn extract_html_sync_with_options( html: String, include_selectors: Option>, exclude_selectors: Option>, max_input_size: Option, + limits: Option>, ) -> PyResult { - let mut builder = markkit::Markkit::builder(); + let mut builder = builder(limits.as_deref()); if let Some(size) = max_input_size { builder = builder.max_input_size(size); } @@ -47,40 +189,45 @@ fn extract_html_sync_with_options( builder = builder.exclude_selector(sel); } } - let m = builder.build(); - m.extract_html_sync(&html) - .map_err(|e| PyErr::new::(e.to_string())) + builder.build().extract_html_sync(&html).map_err(err) } /// Extract markdown from a URL. #[pyfunction] -fn extract_url(py: Python<'_>, url: String) -> PyResult> { - pyo3_async_runtimes::tokio::future_into_py(py, async move { - let m = markkit::Markkit::builder().build(); - m.extract_url(&url) - .await - .map_err(|e| PyErr::new::(e.to_string())) - }) +#[pyo3(signature = (url, *, limits=None))] +fn extract_url<'py>( + py: Python<'py>, + url: String, + limits: Option>, +) -> PyResult> { + let m = builder(limits.as_deref()).build(); + pyo3_async_runtimes::tokio::future_into_py( + py, + async move { m.extract_url(&url).await.map_err(err) }, + ) } /// Extract markdown from bytes with optional content type. #[pyfunction] -fn extract_bytes( - py: Python<'_>, +#[pyo3(signature = (data, content_type=None, *, limits=None))] +fn extract_bytes<'py>( + py: Python<'py>, data: Vec, content_type: Option, -) -> PyResult> { + limits: Option>, +) -> PyResult> { + let m = builder(limits.as_deref()).build(); pyo3_async_runtimes::tokio::future_into_py(py, async move { - let m = markkit::Markkit::builder().build(); m.extract_bytes(&data, content_type.as_deref()) .await - .map_err(|e| PyErr::new::(e.to_string())) + .map_err(err) }) } #[pymodule] #[pyo3(name = "markkit")] fn markkit_module(m: &Bound<'_, PyModule>) -> PyResult<()> { + m.add_class::()?; m.add_function(wrap_pyfunction!(extract_html, m)?)?; m.add_function(wrap_pyfunction!(extract_html_sync, m)?)?; m.add_function(wrap_pyfunction!(extract_html_sync_with_options, m)?)?; diff --git a/crates/markkit-wasm/README.md b/crates/markkit-wasm/README.md index 76fe4c6..88c2461 100644 --- a/crates/markkit-wasm/README.md +++ b/crates/markkit-wasm/README.md @@ -37,10 +37,11 @@ console.log(doc.title, doc.format, doc.markdown); ## Options -`maxInputSize`, `preserveLinks`, `preserveImages`, `includeSelectors`, +`maxInputSize`, `maxOutputSize`, `maxNodes`, `maxNestingDepth`, `preserveLinks`, `preserveImages`, `includeSelectors`, `excludeSelectors` (added to the defaults), `maxArchiveEntries`, `maxArchiveEntrySize`, `maxArchiveTotalSize`, `maxSheetRows`, `maxSheetCols`, -`maxSpreadsheetCells`, `includeHiddenSheets`. Unknown keys throw. +`maxSpreadsheetCells`, `includeHiddenSheets`. Unknown keys throw. There is no +`timeout`: wasm has no clock, so bound time in the host (e.g. a Worker timeout). ## Agent tool diff --git a/crates/markkit-wasm/__test__/markkit-wasm.test.mjs b/crates/markkit-wasm/__test__/markkit-wasm.test.mjs index 0f73332..c62bdca 100644 --- a/crates/markkit-wasm/__test__/markkit-wasm.test.mjs +++ b/crates/markkit-wasm/__test__/markkit-wasm.test.mjs @@ -38,6 +38,15 @@ test("input size limit", () => { assert.throws(() => mk.extractHtml("

        too long

        "), /Resource limit/); }); +test("output, node and depth limits", () => { + const html = "

        hello world

        ".repeat(200); + assert.throws(() => new Markkit({ maxOutputSize: 100 }).extractHtml(html), /max_output_size/); + assert.throws(() => new Markkit({ maxNodes: 50 }).extractHtml(html), /max_nodes/); + const deep = "
        ".repeat(20) + "

        Deep

        " + "
        ".repeat(20); + assert.ok(!new Markkit({ maxNestingDepth: 5 }).extractHtml(deep).includes("> ".repeat(6))); + assert.throws(() => new Markkit({ timeout: 1 }), /invalid options/); +}); + test("extractDocument returns metadata", () => { const mk = new Markkit(); const out = mk.extractDocument(new TextEncoder().encode("

        bytes

        "), { fileName: "a.html" }); diff --git a/crates/markkit-wasm/js/index.d.ts b/crates/markkit-wasm/js/index.d.ts index 0fb6808..8fbeb33 100644 --- a/crates/markkit-wasm/js/index.d.ts +++ b/crates/markkit-wasm/js/index.d.ts @@ -10,6 +10,12 @@ export function version(): string; export interface MarkkitOptions { /** Maximum input size in bytes (default 50 MiB). */ maxInputSize?: number; + /** Maximum Markdown output in bytes (default 128 MiB). */ + maxOutputSize?: number; + /** Maximum parsed nodes (default 5 000 000). */ + maxNodes?: number; + /** Maximum element nesting (default 256, capped at 512). */ + maxNestingDepth?: number; preserveLinks?: boolean; preserveImages?: boolean; /** CSS selectors to include (empty = whole document). */ diff --git a/crates/markkit-wasm/src/lib.rs b/crates/markkit-wasm/src/lib.rs index c49ef35..2e78363 100644 --- a/crates/markkit-wasm/src/lib.rs +++ b/crates/markkit-wasm/src/lib.rs @@ -6,6 +6,8 @@ //! so typos fail loudly instead of silently using defaults. //! - Bytes in, Markdown out. No URL fetching in wasm: the host fetches. //! - Errors are thrown as `Error` with the Rust `Display` text. +//! - No `timeout` option: wasm32-unknown-unknown has no clock, so the core +//! budget has no deadline here; hosts bound time (Worker timeout, CPU limit). use serde::{Deserialize, Serialize}; use wasm_bindgen::prelude::*; @@ -18,6 +20,10 @@ use markkit::{DocumentHint, MarkkitBuilder}; #[serde(rename_all = "camelCase", deny_unknown_fields)] struct Options { max_input_size: Option, + max_output_size: Option, + max_nodes: Option, + /// Capped at 512. + max_nesting_depth: Option, preserve_links: Option, preserve_images: Option, /// Added to the default selector set. @@ -43,6 +49,9 @@ impl Options { } set!( max_input_size, + max_output_size, + max_nodes, + max_nesting_depth, preserve_links, preserve_images, max_archive_entries, diff --git a/docs/getting-started.md b/docs/getting-started.md index ab06b78..cae9b8f 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -77,6 +77,12 @@ let markkit = Markkit::builder() .build(); ``` +Every resource an input can drive (input/output size, time, nesting, node count, +archives, spreadsheets, redirects) is bounded by `markkit::Limits` with generous +defaults. The same limits are available as CLI flags (`markkit extract --help`, +"Limits" section), as `markkit.Limits(...)` in Python and as flat option keys in +Node.js and WASM. + ## Next Steps - [HTML extraction](../knowledge/formats/html-extraction.md): element mapping and stripping rules diff --git a/knowledge/log.md b/knowledge/log.md index 293615e..68d7709 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -27,4 +27,5 @@ * **Accuracy pass**: [Python Package](runtimes/python-package.md) now names the shipped `markkit` module; [Examples](operations/examples.md) lists the four real examples; [Evaluation Harness](operations/eval.md) is marked as a stub; [Threat Model](security/threat-model.md) T3 records that `http_client` is a default feature and SSRF mitigations are still open. * **Maintenance contract**: [Maintenance](operations/maintenance.md) adopts the analyze, fix, ship invocation contract and a workflow credential audit. * **Benchmarks**: `markkit-bench` is no longer a stub. It measures latency, ops/s, throughput and per-op heap allocations (count, bytes, peak) plus process peak RSS over a generated corpus, and `just bench` saves JSON + Markdown runs to `crates/markkit-bench/results/`, like bashkit. Criterion moves to `just bench-criterion`. See [Performance Results](operations/performance-results.md). -* **Resource limits**: all limits moved into one `Limits` struct (bashkit `ExecutionLimits` style). New: `timeout` now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB, stops table amplification); `max_nodes` (5M); `max_nesting_depth` configurable (256, was hardcoded 200 HTML / 256 XML) with a 512 hard ceiling. Threat T2h, invariants S14-S16, `limits` tests. See [Threat Model](security/threat-model.md) T2. +* **Resource limits**: all limits moved into one `Limits` struct (bashkit `ExecutionLimits` style). New: `timeout` now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB, stops table amplification); `max_nodes` (5M); `max_nesting_depth` configurable (256, was hardcoded 200 HTML / 256 XML) with a 512 hard ceiling. Threat T2g, invariants S14-S16, `limits` tests. See [Threat Model](security/threat-model.md) T2. +* **Limits in CLI and bindings**: every `Limits` field is a CLI flag and a Python `markkit.Limits` argument; Node `MarkkitOptions` and WASM `Options` gain flat `timeout`, `maxOutputSize`, `maxNodes`, `maxNestingDepth` keys; parity table updated. See [Node.js Package](runtimes/node-package.md), [Python Package](runtimes/python-package.md), [WASM Package](runtimes/wasm-package.md). diff --git a/knowledge/runtimes/node-package.md b/knowledge/runtimes/node-package.md index 9de5657..aa3da61 100644 --- a/knowledge/runtimes/node-package.md +++ b/knowledge/runtimes/node-package.md @@ -30,9 +30,10 @@ const doc = m.extractDocumentSync(buf, { fileName: 'report.docx' }); Every function takes an optional trailing `options` object (`MarkkitOptions`): the camelCase builder fields shared with the [WASM Package](wasm-package.md) -(`maxInputSize`, `preserveLinks`, `preserveImages`, `includeSelectors`, +(`maxInputSize`, `maxOutputSize`, `maxNodes`, `maxNestingDepth`, `preserveLinks`, `preserveImages`, `includeSelectors`, `excludeSelectors`, archive and sheet limits, `includeHiddenSheets`) plus the URL -policy fields `maxProcessingSecs`, `allowPrivateNetworks`, `allowedHosts`, +policy fields `timeout` (seconds, float; whole call), `maxProcessingSecs` +(whole-second alias, `timeout` wins), `allowPrivateNetworks`, `allowedHosts`, `maxRedirects`. Selectors add to the defaults. 64-bit limits arrive as JS numbers and must be whole and non-negative. Unlike WASM, unknown keys are ignored (napi objects do not report them). @@ -45,6 +46,7 @@ Errors reject (or throw) with the Rust error's `Display` text as the reason. |---|---|---|---| | HTML async / sync | yes | yes | sync | | HTML with options | `extract_html_sync_with_options` (selectors, size) | all builder options | all builder options | +| Resource limits (`Limits`) | `limits=markkit.Limits(...)` on every function | flat option keys, incl. `timeout` | flat option keys, no `timeout` (no clock) | | URL | yes, defaults only | yes, with URL policy options | no (host fetches) | | Bytes | yes | yes | yes | | `extract_document` (metadata) | no | yes | yes | diff --git a/knowledge/runtimes/python-package.md b/knowledge/runtimes/python-package.md index 087a2d5..63a181e 100644 --- a/knowledge/runtimes/python-package.md +++ b/knowledge/runtimes/python-package.md @@ -34,8 +34,16 @@ markdown = markkit.extract_html_sync_with_options( exclude_selectors=["nav"], max_input_size=1_000_000, ) + +# Resource limits: every function takes keyword-only `limits=` +limits = markkit.Limits(timeout=10.0, max_output_size=8 << 20) +markdown = markkit.extract_html_sync(html, limits=limits) ``` +`markkit.Limits` mirrors `markkit::Limits` field for field (`timeout` in float +seconds); unset arguments keep library defaults and are readable as attributes. +Invalid values raise `ValueError` / `OverflowError` at construction. + Errors surface as `RuntimeError` carrying the Rust error's `Display` text. ## Build diff --git a/knowledge/runtimes/wasm-package.md b/knowledge/runtimes/wasm-package.md index 33bbb45..2b9504c 100644 --- a/knowledge/runtimes/wasm-package.md +++ b/knowledge/runtimes/wasm-package.md @@ -37,7 +37,8 @@ const tool = new MarkkitTool({ locale, maxOutputChars }); tool.toolDefinition(); await tool.execute(args); ``` -Options are camelCase builder fields (`maxInputSize`, `preserveLinks`, +Options are camelCase builder fields (`maxInputSize`, `maxOutputSize`, +`maxNodes`, `maxNestingDepth`, `preserveLinks`, `preserveImages`, `includeSelectors`, `excludeSelectors`, archive and sheet limits, `includeHiddenSheets`). Selectors add to the defaults, as in Python. Unknown keys and wrong types throw `invalid options`: values go through From ece5be1777fc2c12696b2357b4dbd321c1e09ef3 Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Tue, 27 Oct 2026 01:24:31 +0000 Subject: [PATCH 5/5] feat(bindings): base_url and main_content in Node, WASM and Python options --- CHANGELOG.md | 2 ++ crates/markkit-js/README.md | 2 +- crates/markkit-js/__test__/options.test.cjs | 8 ++++++++ crates/markkit-js/src/lib.rs | 11 +++++++++++ crates/markkit-python/src/lib.rs | 11 +++++++++-- crates/markkit-wasm/README.md | 2 +- crates/markkit-wasm/__test__/markkit-wasm.test.mjs | 8 ++++++++ crates/markkit-wasm/js/index.d.ts | 4 ++++ crates/markkit-wasm/src/lib.rs | 5 +++++ knowledge/runtimes/node-package.md | 4 ++-- knowledge/runtimes/python-package.md | 2 ++ knowledge/runtimes/wasm-package.md | 2 +- 12 files changed, 54 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a550f8..5807dff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/). alias of `--max-input-size`), Python `markkit.Limits(...)` passed as `limits=` to every function, and flat Node.js and WASM option keys (`maxOutputSize`, `maxNodes`, `maxNestingDepth`, plus `timeout` in Node.js). +- Node.js and WASM options gain `baseUrl` and `mainContent`; Python `extract_html_sync_with_options` + gains `base_url=` and `main_content=`, matching the core builder. ### Security diff --git a/crates/markkit-js/README.md b/crates/markkit-js/README.md index 4cd98be..0f194a3 100644 --- a/crates/markkit-js/README.md +++ b/crates/markkit-js/README.md @@ -35,7 +35,7 @@ console.log(doc.title, doc.format, doc.warnings, doc.markdown); Fields: `maxInputSize`, `maxOutputSize`, `timeout` (seconds, covers fetch, parsing and rendering; `maxProcessingSecs` is the whole-second form), `maxNodes`, `maxNestingDepth`, -`preserveLinks`, `preserveImages`, +`baseUrl`, `mainContent`, `preserveLinks`, `preserveImages`, `includeSelectors`, `excludeSelectors` (added to the defaults), `maxArchiveEntries`, `maxArchiveEntrySize`, `maxArchiveTotalSize`, `maxSheetRows`, `maxSheetCols`, `maxSpreadsheetCells`, `includeHiddenSheets`, `allowPrivateNetworks` (trusted URLs only), diff --git a/crates/markkit-js/__test__/options.test.cjs b/crates/markkit-js/__test__/options.test.cjs index 81923a7..aa60a0c 100644 --- a/crates/markkit-js/__test__/options.test.cjs +++ b/crates/markkit-js/__test__/options.test.cjs @@ -34,6 +34,14 @@ test("resource limit options", () => { assert.ok(!m.extractHtmlSync(deep, { maxNestingDepth: 5 }).includes("> ".repeat(6))); }); +test("baseUrl and mainContent options", () => { + const page = '

        T

        ' + "

        Long article text, with commas, and more words here.

        ".repeat(8) + '

        rel

        '; + const md = m.extractHtmlSync(page, { baseUrl: "https://x.test/d/", mainContent: true }); + assert.ok(md.includes("(https://x.test/a)")); + assert.ok(!md.includes("side")); + assert.ok(m.extractHtmlSync(page).includes("side")); +}); + test("invalid numeric options are rejected", () => { assert.throws(() => m.extractHtmlSync("

        x

        ", { maxArchiveTotalSize: -1 }), /invalid options/); assert.throws(() => m.extractHtmlSync("

        x

        ", { maxSpreadsheetCells: 1.5 }), /invalid options/); diff --git a/crates/markkit-js/src/lib.rs b/crates/markkit-js/src/lib.rs index a8e43ad..9f1f3bf 100644 --- a/crates/markkit-js/src/lib.rs +++ b/crates/markkit-js/src/lib.rs @@ -28,6 +28,11 @@ pub struct MarkkitOptions { pub max_nodes: Option, /// Maximum element nesting (default 256, capped at 512). pub max_nesting_depth: Option, + /// Absolute URL relative links and images resolve against (`extractUrl` + /// uses the final response URL). + pub base_url: Option, + /// Keep only the detected main content (article body) of HTML pages. + pub main_content: Option, pub preserve_links: Option, pub preserve_images: Option, /// CSS selectors to include (empty = whole document). @@ -141,6 +146,12 @@ fn markkit(options: Option) -> napi::Result { if let Some(v) = o.max_nesting_depth { b = b.max_nesting_depth(v as usize); } + if let Some(v) = o.base_url { + b = b.base_url(v); + } + if let Some(v) = o.main_content { + b = b.main_content(v); + } if let Some(v) = o.preserve_links { b = b.preserve_links(v); } diff --git a/crates/markkit-python/src/lib.rs b/crates/markkit-python/src/lib.rs index 674bff2..d28c0f6 100644 --- a/crates/markkit-python/src/lib.rs +++ b/crates/markkit-python/src/lib.rs @@ -165,17 +165,24 @@ fn extract_html_sync(html: String, limits: Option>) -> PyResul /// include_selectors: list of CSS selectors to include /// exclude_selectors: list of CSS selectors to exclude /// max_input_size: maximum input size in bytes (overrides limits.max_input_size) +/// base_url: absolute URL relative links and images resolve against +/// main_content: keep only the detected article body (default False) /// limits: markkit.Limits #[pyfunction] -#[pyo3(signature = (html, *, include_selectors=None, exclude_selectors=None, max_input_size=None, limits=None))] +#[pyo3(signature = (html, *, include_selectors=None, exclude_selectors=None, max_input_size=None, base_url=None, main_content=false, limits=None))] fn extract_html_sync_with_options( html: String, include_selectors: Option>, exclude_selectors: Option>, max_input_size: Option, + base_url: Option, + main_content: bool, limits: Option>, ) -> PyResult { - let mut builder = builder(limits.as_deref()); + let mut builder = builder(limits.as_deref()).main_content(main_content); + if let Some(base) = base_url { + builder = builder.base_url(base); + } if let Some(size) = max_input_size { builder = builder.max_input_size(size); } diff --git a/crates/markkit-wasm/README.md b/crates/markkit-wasm/README.md index 88c2461..30c020a 100644 --- a/crates/markkit-wasm/README.md +++ b/crates/markkit-wasm/README.md @@ -37,7 +37,7 @@ console.log(doc.title, doc.format, doc.markdown); ## Options -`maxInputSize`, `maxOutputSize`, `maxNodes`, `maxNestingDepth`, `preserveLinks`, `preserveImages`, `includeSelectors`, +`maxInputSize`, `maxOutputSize`, `maxNodes`, `maxNestingDepth`, `baseUrl`, `mainContent`, `preserveLinks`, `preserveImages`, `includeSelectors`, `excludeSelectors` (added to the defaults), `maxArchiveEntries`, `maxArchiveEntrySize`, `maxArchiveTotalSize`, `maxSheetRows`, `maxSheetCols`, `maxSpreadsheetCells`, `includeHiddenSheets`. Unknown keys throw. There is no diff --git a/crates/markkit-wasm/__test__/markkit-wasm.test.mjs b/crates/markkit-wasm/__test__/markkit-wasm.test.mjs index c62bdca..3c80df0 100644 --- a/crates/markkit-wasm/__test__/markkit-wasm.test.mjs +++ b/crates/markkit-wasm/__test__/markkit-wasm.test.mjs @@ -47,6 +47,14 @@ test("output, node and depth limits", () => { assert.throws(() => new Markkit({ timeout: 1 }), /invalid options/); }); +test("baseUrl and mainContent options", () => { + const page = '

        T

        ' + "

        Long article text, with commas, and more words here.

        ".repeat(8) + '

        rel

        '; + const md = new Markkit({ baseUrl: "https://x.test/d/", mainContent: true }).extractHtml(page); + assert.ok(md.includes("(https://x.test/a)")); + assert.ok(!md.includes("side")); + assert.ok(new Markkit().extractHtml(page).includes("side")); +}); + test("extractDocument returns metadata", () => { const mk = new Markkit(); const out = mk.extractDocument(new TextEncoder().encode("

        bytes

        "), { fileName: "a.html" }); diff --git a/crates/markkit-wasm/js/index.d.ts b/crates/markkit-wasm/js/index.d.ts index 8fbeb33..b465432 100644 --- a/crates/markkit-wasm/js/index.d.ts +++ b/crates/markkit-wasm/js/index.d.ts @@ -16,6 +16,10 @@ export interface MarkkitOptions { maxNodes?: number; /** Maximum element nesting (default 256, capped at 512). */ maxNestingDepth?: number; + /** Absolute URL relative links and images resolve against. */ + baseUrl?: string; + /** Keep only the detected main content (article body) of HTML pages. */ + mainContent?: boolean; preserveLinks?: boolean; preserveImages?: boolean; /** CSS selectors to include (empty = whole document). */ diff --git a/crates/markkit-wasm/src/lib.rs b/crates/markkit-wasm/src/lib.rs index 2e78363..fe279da 100644 --- a/crates/markkit-wasm/src/lib.rs +++ b/crates/markkit-wasm/src/lib.rs @@ -24,6 +24,9 @@ struct Options { max_nodes: Option, /// Capped at 512. max_nesting_depth: Option, + /// Absolute URL relative links and images resolve against. + base_url: Option, + main_content: Option, preserve_links: Option, preserve_images: Option, /// Added to the default selector set. @@ -52,6 +55,8 @@ impl Options { max_output_size, max_nodes, max_nesting_depth, + base_url, + main_content, preserve_links, preserve_images, max_archive_entries, diff --git a/knowledge/runtimes/node-package.md b/knowledge/runtimes/node-package.md index aa3da61..7343ced 100644 --- a/knowledge/runtimes/node-package.md +++ b/knowledge/runtimes/node-package.md @@ -30,7 +30,7 @@ const doc = m.extractDocumentSync(buf, { fileName: 'report.docx' }); Every function takes an optional trailing `options` object (`MarkkitOptions`): the camelCase builder fields shared with the [WASM Package](wasm-package.md) -(`maxInputSize`, `maxOutputSize`, `maxNodes`, `maxNestingDepth`, `preserveLinks`, `preserveImages`, `includeSelectors`, +(`maxInputSize`, `maxOutputSize`, `maxNodes`, `maxNestingDepth`, `baseUrl`, `mainContent`, `preserveLinks`, `preserveImages`, `includeSelectors`, `excludeSelectors`, archive and sheet limits, `includeHiddenSheets`) plus the URL policy fields `timeout` (seconds, float; whole call), `maxProcessingSecs` (whole-second alias, `timeout` wins), `allowPrivateNetworks`, `allowedHosts`, @@ -45,7 +45,7 @@ Errors reject (or throw) with the Rust error's `Display` text as the reason. | Capability | Python | Node.js | WASM | |---|---|---|---| | HTML async / sync | yes | yes | sync | -| HTML with options | `extract_html_sync_with_options` (selectors, size) | all builder options | all builder options | +| HTML with options | `extract_html_sync_with_options` (selectors, size, `base_url`, `main_content`) | all builder options | all builder options | | Resource limits (`Limits`) | `limits=markkit.Limits(...)` on every function | flat option keys, incl. `timeout` | flat option keys, no `timeout` (no clock) | | URL | yes, defaults only | yes, with URL policy options | no (host fetches) | | Bytes | yes | yes | yes | diff --git a/knowledge/runtimes/python-package.md b/knowledge/runtimes/python-package.md index 63a181e..87d8485 100644 --- a/knowledge/runtimes/python-package.md +++ b/knowledge/runtimes/python-package.md @@ -33,6 +33,8 @@ markdown = markkit.extract_html_sync_with_options( include_selectors=["article"], exclude_selectors=["nav"], max_input_size=1_000_000, + base_url="https://example.com/docs/", + main_content=True, ) # Resource limits: every function takes keyword-only `limits=` diff --git a/knowledge/runtimes/wasm-package.md b/knowledge/runtimes/wasm-package.md index 2b9504c..3887068 100644 --- a/knowledge/runtimes/wasm-package.md +++ b/knowledge/runtimes/wasm-package.md @@ -38,7 +38,7 @@ tool.toolDefinition(); await tool.execute(args); ``` Options are camelCase builder fields (`maxInputSize`, `maxOutputSize`, -`maxNodes`, `maxNestingDepth`, `preserveLinks`, +`maxNodes`, `maxNestingDepth`, `baseUrl`, `mainContent`, `preserveLinks`, `preserveImages`, `includeSelectors`, `excludeSelectors`, archive and sheet limits, `includeHiddenSheets`). Selectors add to the defaults, as in Python. Unknown keys and wrong types throw `invalid options`: values go through