From d2131b38e117714483afe30b1a4e94274f9fd6ca Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Mon, 26 Oct 2026 23:42:45 +0000 Subject: [PATCH 1/3] feat(limits): unified Limits with timeout, output, node and depth caps Move every resource limit into one markkit::Limits struct (bashkit ExecutionLimits style) on MarkkitBuilder. New limits, generous defaults: - timeout (60s) now bounds parsing and rendering, not only URL fetch - max_output_size (128 MiB), checked per block/row; stops table amplification - max_nodes (5M) for HTML DOM and DOCX XML nodes - max_nesting_depth (256) configurable, hard ceiling 1024 for stack safety Breaking: limit fields moved to builder.limits; max_processing_secs field is now limits.timeout (setter kept). --- CHANGELOG.md | 25 ++ README.md | 36 ++- crates/markkit/src/archive.rs | 8 +- crates/markkit/src/config.rs | 82 +++--- crates/markkit/src/docx.rs | 52 +++- crates/markkit/src/extract.rs | 109 ++++---- crates/markkit/src/fetch.rs | 16 +- crates/markkit/src/html/parser.rs | 54 +++- crates/markkit/src/lib.rs | 2 + crates/markkit/src/limits.rs | 317 +++++++++++++++++++++++ crates/markkit/src/render/markdown.rs | 94 ++++++- crates/markkit/src/spreadsheet.rs | 40 ++- crates/markkit/src/xml.rs | 64 ++++- crates/markkit/tests/html_integration.rs | 5 +- crates/markkit/tests/limits.rs | 269 +++++++++++++++++++ examples/configuration.rs | 13 + knowledge/formats/html-extraction.md | 2 +- knowledge/foundations/architecture.md | 6 +- knowledge/foundations/ir.md | 2 +- knowledge/log.md | 1 + knowledge/operations/testing.md | 1 + knowledge/security/security-model.md | 2 +- knowledge/security/threat-model.md | 48 +++- 23 files changed, 1088 insertions(+), 160 deletions(-) create mode 100644 crates/markkit/src/limits.rs create mode 100644 crates/markkit/tests/limits.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b78190..a1297a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Security +- Resource limits on every aspect, in one `markkit::Limits` struct (`MarkkitBuilder::limits(..)` or + per-field setters). New: `timeout` (default 60 s) now bounds parsing and rendering, not just URL + fetch; `max_output_size` (128 MiB) stops output amplification (e.g. one wide table row padding + thousands of rows); `max_nodes` (5 000 000) caps parsed HTML DOM / DOCX XML nodes; + `max_nesting_depth` (256) is configurable, with a hard ceiling of 1024 that keeps deep input off + the host stack. Violations return `Error::ResourceLimit` naming the limit. + - `extract_url` (CLI, Python and Node `extract_url` too) now refuses non-public destinations by default: loopback, private, link-local / cloud metadata and other reserved addresses, checked on the URL, after DNS resolution and on every redirect. Only `http`/`https` are fetched. Redirects are capped @@ -18,6 +25,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` + (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is + now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, + `max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth + 256 (was 200). + - Design docs moved from `specs/` to an OKF v0.2 knowledge bundle in `knowledge/`, validated in CI. - CI workflows declare read-only default permissions and do not persist checkout credentials. - CI runs every example; `just bench` now runs the criterion benchmarks. @@ -48,6 +61,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` + (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is + now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, + `max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth + 256 (was 200). + - Release automation: merging `chore(release): prepare vX.Y.Z` to main tags, creates the GitHub Release and publishes to crates.io (idempotent); manual dispatch from main also works @@ -94,6 +113,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` + (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is + now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, + `max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth + 256 (was 200). + - `docx` feature now uses `zip` + `quick-xml` directly; `docx-rs` dependency removed - `calamine` upgraded to 0.36 (pulls quick-xml 0.41, clearing RUSTSEC-2026-0194/0195); `zip` upgraded to 8 (deflate only), shared with calamine - PDF extraction returns `Error::UnsupportedFormat` instead of an empty string diff --git a/README.md b/README.md index 0d9e55a..9faf2e0 100644 --- a/README.md +++ b/README.md @@ -166,10 +166,36 @@ markkit = { version = "0.1", default-features = false, features = ["docx", "spre # add "html" if you also convert HTML ``` -Office-document limits are configurable on the builder: `max_archive_entries`, -`max_archive_entry_size`, `max_archive_total_size`, `max_spreadsheet_cells`, -`max_sheet_rows`, `max_sheet_cols`, `include_hidden_sheets`. URL fetching: -`allow_private_networks`, `allow_host`, `max_redirects`. +### Resource limits + +Every resource an input can drive is bounded by `markkit::Limits`. Defaults are +generous; violations return `Error::ResourceLimit` naming the limit. + +| Limit | Default | Bounds | +|-------|---------|--------| +| `max_input_size` | 50 MiB | Input bytes, fetched URL body | +| `max_output_size` | 128 MiB | Markdown output | +| `timeout` | 60 s | Fetch + parse + render, per call | +| `max_nesting_depth` | 256 (hard max 1024) | HTML flattens deeper content; deeper DOCX XML is rejected | +| `max_nodes` | 5 000 000 | HTML DOM nodes, DOCX XML nodes | +| `max_archive_entries` | 10 000 | ZIP entries (DOCX/XLSX/ODS) | +| `max_archive_entry_size` | 64 MiB | Decompressed bytes per ZIP entry | +| `max_archive_total_size` | 256 MiB | Decompressed bytes per ZIP | +| `max_spreadsheet_cells` | 4 000 000 | XLS/ODS grid before parsing | +| `max_sheet_rows` / `max_sheet_cols` | 10 000 / 256 | Rendered per sheet (truncated with a warning) | +| `max_redirects` | 5 | `extract_url` redirects | + +```rust +use markkit::{Limits, Markkit}; +use std::time::Duration; + +let markkit = Markkit::builder() + .limits(Limits::default().timeout(Duration::from_secs(10)).max_output_size(8 << 20)) + .build(); +``` + +Each limit also has a builder setter (`.max_nodes(..)`, `.timeout(..)`, ...). +Other options: `include_hidden_sheets`, `allow_private_networks`, `allow_host`. ## Security @@ -180,7 +206,7 @@ XML entities are never expanded; macros and formulas are never executed; hidden sheets are skipped by default. `extract_url` accepts only http/https, refuses loopback, private, link-local and cloud-metadata addresses (also after DNS resolution and on every redirect), caps redirects, and bounds the body by -`max_input_size` and the fetch by `max_processing_secs`. Opt in to internal +`max_input_size` and the fetch by `timeout`. Opt in to internal hosts with `allow_private_networks(true)`; restrict hosts with `allow_host(..)`. See the [threat model](knowledge/security/threat-model.md) and the diff --git a/crates/markkit/src/archive.rs b/crates/markkit/src/archive.rs index 5e7a85d..3166a02 100644 --- a/crates/markkit/src/archive.rs +++ b/crates/markkit/src/archive.rs @@ -41,11 +41,11 @@ impl<'a> ZipGuard<'a> { /// Open the archive and check the entry count. pub fn open(bytes: &'a [u8], config: &MarkkitBuilder) -> Result { let archive = ZipArchive::new(Cursor::new(bytes))?; - if archive.len() > config.max_archive_entries { + if archive.len() > config.limits.max_archive_entries { return Err(ArchiveError::Limit(format!( "archive has {} entries, limit is {}", archive.len(), - config.max_archive_entries + config.limits.max_archive_entries ))); } let names = archive @@ -55,8 +55,8 @@ impl<'a> ZipGuard<'a> { Ok(Self { archive, names, - max_entry: config.max_archive_entry_size, - max_total: config.max_archive_total_size, + max_entry: config.limits.max_archive_entry_size, + max_total: config.limits.max_archive_total_size, total: 0, }) } diff --git a/crates/markkit/src/config.rs b/crates/markkit/src/config.rs index 9f86747..cb5fa0d 100644 --- a/crates/markkit/src/config.rs +++ b/crates/markkit/src/config.rs @@ -1,12 +1,14 @@ +use std::time::Duration; + use crate::extract::Markkit; +use crate::limits::Limits; /// Builder for configuring a Markkit instance. #[derive(Debug, Clone)] pub struct MarkkitBuilder { - /// Maximum input size in bytes (default: 50MB). - pub max_input_size: usize, - /// Maximum processing time in seconds (default: 60). - pub max_processing_secs: u64, + /// Resource limits (input/output size, time, nesting, nodes, archives, + /// spreadsheets, redirects). See [`Limits`]. + pub limits: Limits, /// Whether to preserve link URLs in output. pub preserve_links: bool, /// Whether to preserve image references in output. @@ -17,19 +19,6 @@ pub struct MarkkitBuilder { pub include_selectors: Vec, /// CSS selectors for content to exclude. pub exclude_selectors: Vec, - /// Maximum number of entries in a ZIP container (DOCX/XLSX/ODS). Default: 10 000. - pub max_archive_entries: usize, - /// Maximum decompressed size of a single ZIP entry in bytes. Default: 64 MiB. - pub max_archive_entry_size: u64, - /// Maximum total decompressed size of a ZIP container in bytes. Default: 256 MiB. - pub max_archive_total_size: u64, - /// Maximum data rows rendered per sheet (header excluded). Default: 10 000. - pub max_sheet_rows: usize, - /// Maximum columns rendered per sheet. Default: 256. - pub max_sheet_cols: usize, - /// Maximum cell grid a workbook may expand to while parsing (XLS/ODS - /// pre-scan, summed over sheets). Default: 4 000 000. - pub max_spreadsheet_cells: u64, /// Render hidden / very hidden sheets. Default: false (skipped with a warning). pub include_hidden_sheets: bool, /// Let `extract_url` reach loopback, private, link-local and other @@ -38,15 +27,12 @@ pub struct MarkkitBuilder { /// Hosts `extract_url` may contact, matched exactly and case-insensitively /// on every hop (empty = any public host). pub allowed_hosts: Vec, - /// Maximum redirects `extract_url` follows. Default: 5. - pub max_redirects: usize, } impl Default for MarkkitBuilder { fn default() -> Self { Self { - max_input_size: 50 * 1024 * 1024, - max_processing_secs: 60, + limits: Limits::default(), preserve_links: true, preserve_images: true, extract_code_blocks: true, @@ -59,28 +45,50 @@ impl Default for MarkkitBuilder { "style".into(), "noscript".into(), ], - max_archive_entries: 10_000, - max_archive_entry_size: 64 * 1024 * 1024, - max_archive_total_size: 256 * 1024 * 1024, - max_sheet_rows: 10_000, - max_sheet_cols: 256, - max_spreadsheet_cells: 4_000_000, include_hidden_sheets: false, allow_private_networks: false, allowed_hosts: Vec::new(), - max_redirects: 5, } } } impl MarkkitBuilder { pub fn max_input_size(mut self, size: usize) -> Self { - self.max_input_size = size; + self.limits.max_input_size = size; + self + } + + /// Replace all limits at once. + pub fn limits(mut self, limits: Limits) -> Self { + self.limits = limits; self } + /// Wall-clock budget per call (fetch + parse + render). + pub fn timeout(mut self, timeout: Duration) -> Self { + self.limits.timeout = timeout; + self + } + + /// [`MarkkitBuilder::timeout`] in whole seconds. pub fn max_processing_secs(mut self, secs: u64) -> Self { - self.max_processing_secs = secs; + self.limits.timeout = Duration::from_secs(secs); + self + } + + pub fn max_output_size(mut self, bytes: usize) -> Self { + self.limits.max_output_size = bytes; + self + } + + /// Clamped to [`crate::limits::HARD_MAX_NESTING_DEPTH`]. + pub fn max_nesting_depth(mut self, depth: usize) -> Self { + self.limits.max_nesting_depth = depth; + self + } + + pub fn max_nodes(mut self, nodes: usize) -> Self { + self.limits.max_nodes = nodes; self } @@ -105,32 +113,32 @@ impl MarkkitBuilder { } pub fn max_archive_entries(mut self, n: usize) -> Self { - self.max_archive_entries = n; + self.limits.max_archive_entries = n; self } pub fn max_archive_entry_size(mut self, bytes: u64) -> Self { - self.max_archive_entry_size = bytes; + self.limits.max_archive_entry_size = bytes; self } pub fn max_archive_total_size(mut self, bytes: u64) -> Self { - self.max_archive_total_size = bytes; + self.limits.max_archive_total_size = bytes; self } pub fn max_sheet_rows(mut self, rows: usize) -> Self { - self.max_sheet_rows = rows; + self.limits.max_sheet_rows = rows; self } pub fn max_sheet_cols(mut self, cols: usize) -> Self { - self.max_sheet_cols = cols; + self.limits.max_sheet_cols = cols; self } pub fn max_spreadsheet_cells(mut self, cells: u64) -> Self { - self.max_spreadsheet_cells = cells; + self.limits.max_spreadsheet_cells = cells; self } @@ -153,7 +161,7 @@ impl MarkkitBuilder { } pub fn max_redirects(mut self, n: usize) -> Self { - self.max_redirects = n; + self.limits.max_redirects = n; self } diff --git a/crates/markkit/src/docx.rs b/crates/markkit/src/docx.rs index cf2795f..670dd7d 100644 --- a/crates/markkit/src/docx.rs +++ b/crates/markkit/src/docx.rs @@ -24,6 +24,7 @@ use crate::archive::{ArchiveError, ZipGuard}; use crate::config::MarkkitBuilder; use crate::error::Error; use crate::ir::{Document, ListItem, Node}; +use crate::limits::Budget; use crate::render::escape_table_cell; use crate::xml::{self, Element, XmlError}; @@ -51,10 +52,21 @@ fn xml_err(part: &str, e: XmlError) -> Error { } } -fn read_xml(zip: &mut ZipGuard<'_>, path: &str) -> Result, Error> { - match zip.read(path).map_err(archive_err)? { - Some(bytes) => xml::parse(&bytes).map(Some).map_err(|e| xml_err(path, e)), - None => Ok(None), +/// Bounded XML reader: archive limits, nesting depth, node count and deadline. +struct Parts<'z, 'b> { + zip: ZipGuard<'z>, + max_depth: usize, + budget: &'b Budget, +} + +impl Parts<'_, '_> { + fn read_xml(&mut self, path: &str) -> Result, Error> { + match self.zip.read(path).map_err(archive_err)? { + Some(bytes) => xml::parse(&bytes, self.max_depth, self.budget) + .map(Some) + .map_err(|e| xml_err(path, e)), + None => Ok(None), + } } } @@ -365,6 +377,7 @@ struct Ctx<'a> { rels: &'a HashMap, styles: &'a Styles, numbering: &'a Numbering, + budget: &'a Budget, } impl Ctx<'_> { @@ -665,6 +678,9 @@ fn build_list(entries: &[ListEntry], pos: &mut usize, level: u32) -> Node { impl Ctx<'_> { fn blocks(&self, container: &Element, body: &mut Body) { for child in container.elements() { + if self.budget.expired() { + return; + } match child.name.as_str() { "p" => self.paragraph(child, body), "tbl" => { @@ -863,10 +879,18 @@ fn table_cell(nodes: Vec) -> Vec { // ─── Entry point ──────────────────────────────────────────────────── -pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result { - let mut zip = ZipGuard::open(bytes, config).map_err(archive_err)?; +pub(crate) fn parse( + bytes: &[u8], + config: &MarkkitBuilder, + budget: &Budget, +) -> Result { + let mut zip = Parts { + zip: ZipGuard::open(bytes, config).map_err(archive_err)?, + max_depth: config.limits.nesting_depth(), + budget, + }; - let root_rels = parse_rels(read_xml(&mut zip, "_rels/.rels")?); + let root_rels = parse_rels(zip.read_xml("_rels/.rels")?); let main_path = rel_target_by_type(&root_rels, "/officeDocument") .map(|t| resolve_target("", t)) .unwrap_or_else(|| "word/document.xml".to_string()); @@ -874,10 +898,11 @@ pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result Result Result crate::Result { - if html.len() > self.config.max_input_size { - return Err(Error::ResourceLimit(format!( - "Input size {} exceeds limit {}", - html.len(), - self.config.max_input_size - ))); - } + self.extract_html_with(html, &Budget::new(&self.config.limits)) + } + + fn extract_html_with(&self, html: &str, budget: &Budget) -> crate::Result { + self.config.limits.check_input(html.len())?; #[cfg(feature = "html")] { - let parser = crate::html::HtmlParser::new(&self.config); + let parser = crate::html::HtmlParser::with_budget(&self.config, budget); let doc = parser.parse(html)?; - let mut renderer = crate::render::MarkdownRenderer::new(); - Ok(renderer.render(&doc)) + self.render(&doc, budget) } #[cfg(not(feature = "html"))] - Err(Error::UnsupportedFormat( - "HTML support requires the 'html' feature".to_string(), - )) + { + let _ = budget; + Err(Error::UnsupportedFormat( + "HTML support requires the 'html' feature".to_string(), + )) + } } /// Parse HTML into the IR without rendering. #[cfg(feature = "html")] pub fn parse_html(&self, html: &str) -> crate::Result { - if html.len() > self.config.max_input_size { - return Err(Error::ResourceLimit(format!( - "Input size {} exceeds limit {}", - html.len(), - self.config.max_input_size - ))); - } - let parser = crate::html::HtmlParser::new(&self.config); - parser.parse(html) + self.config.limits.check_input(html.len())?; + let budget = Budget::new(&self.config.limits); + crate::html::HtmlParser::with_budget(&self.config, &budget).parse(html) + } + + #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + fn render(&self, doc: &crate::ir::Document, budget: &Budget) -> crate::Result { + crate::render::MarkdownRenderer::new().render_bounded( + doc, + self.config.limits.max_output_size, + budget, + ) } /// Extract markdown from raw bytes, auto-detecting the format. @@ -157,7 +163,7 @@ impl Markkit { /// Synchronous [`Markkit::extract_document`]. /// - /// Errors: [`Error::ResourceLimit`] when input/archive/grid limits are + /// Errors: [`Error::ResourceLimit`] when any [`Limits`](crate::Limits) value is /// exceeded, [`Error::UnsupportedFormat`] for unknown formats or formats whose /// feature is disabled, and a format-specific variant for malformed input. pub fn extract_document_sync( @@ -165,23 +171,26 @@ impl Markkit { bytes: &[u8], hint: DocumentHint<'_>, ) -> crate::Result { - if bytes.len() > self.config.max_input_size { - return Err(Error::ResourceLimit(format!( - "Input size {} exceeds limit {}", - bytes.len(), - self.config.max_input_size - ))); - } + self.extract_document_with(bytes, hint, &Budget::new(&self.config.limits)) + } + + fn extract_document_with( + &self, + bytes: &[u8], + hint: DocumentHint<'_>, + budget: &Budget, + ) -> crate::Result { + self.config.limits.check_input(bytes.len())?; match Format::detect(bytes, hint) { Format::Html => { let text = String::from_utf8_lossy(bytes); - let md = self.extract_html_sync(&text)?; + let md = self.extract_html_with(&text, budget)?; Ok(Extraction::new(md, Format::Html)) } Format::Pdf => self.extract_pdf_sync(bytes), - Format::Docx => self.extract_docx(bytes), - Format::Xlsx | Format::Xls | Format::Ods => self.extract_spreadsheet(bytes), + Format::Docx => self.extract_docx(bytes, budget), + Format::Xlsx | Format::Xls | Format::Ods => self.extract_spreadsheet(bytes, budget), Format::Doc => Err(Error::UnsupportedFormat( "legacy Word 97-2003 (.doc) documents are not supported".into(), )), @@ -196,12 +205,11 @@ impl Markkit { } #[cfg(feature = "docx")] - fn extract_docx(&self, bytes: &[u8]) -> crate::Result { + fn extract_docx(&self, bytes: &[u8], budget: &Budget) -> crate::Result { let out = guarded("DOCX", Error::DocxExtract, || { - crate::docx::parse(bytes, &self.config) + crate::docx::parse(bytes, &self.config, budget) })?; - let mut renderer = crate::render::MarkdownRenderer::new(); - let mut e = Extraction::new(renderer.render(&out.doc), Format::Docx); + let mut e = Extraction::new(self.render(&out.doc, budget)?, Format::Docx); e.title = out.title; e.stats.paragraphs = Some(out.paragraphs); e.stats.tables = Some(out.tables); @@ -209,19 +217,18 @@ impl Markkit { } #[cfg(not(feature = "docx"))] - fn extract_docx(&self, _bytes: &[u8]) -> crate::Result { + fn extract_docx(&self, _bytes: &[u8], _budget: &Budget) -> crate::Result { Err(Error::UnsupportedFormat( "DOCX support requires the 'docx' feature".to_string(), )) } #[cfg(feature = "spreadsheet")] - fn extract_spreadsheet(&self, bytes: &[u8]) -> crate::Result { + fn extract_spreadsheet(&self, bytes: &[u8], budget: &Budget) -> crate::Result { let out = guarded("spreadsheet", Error::SpreadsheetExtract, || { - crate::spreadsheet::parse(bytes, &self.config) + crate::spreadsheet::parse(bytes, &self.config, budget) })?; - let mut renderer = crate::render::MarkdownRenderer::new(); - let mut e = Extraction::new(renderer.render(&out.doc), out.format); + let mut e = Extraction::new(self.render(&out.doc, budget)?, out.format); e.warnings = out.warnings; e.stats.sheets = Some(out.sheets); e.stats.rows = Some(out.rows); @@ -229,7 +236,7 @@ impl Markkit { } #[cfg(not(feature = "spreadsheet"))] - fn extract_spreadsheet(&self, _bytes: &[u8]) -> crate::Result { + fn extract_spreadsheet(&self, _bytes: &[u8], _budget: &Budget) -> crate::Result { Err(Error::UnsupportedFormat( "spreadsheet support requires the 'spreadsheet' feature".to_string(), )) @@ -242,13 +249,19 @@ impl Markkit { /// refused with [`Error::UrlBlocked`] on every hop (including after DNS /// resolution) unless [`MarkkitBuilder::allow_private_networks`] is set; /// [`MarkkitBuilder::allow_host`] restricts hosts; redirects capped by - /// `max_redirects`; body capped by `max_input_size`; whole fetch capped - /// by `max_processing_secs` ([`Error::ResourceLimit`]). + /// `max_redirects`; body capped by `max_input_size`; fetch, parse and + /// render together capped by `limits.timeout` ([`Error::ResourceLimit`]). #[cfg(feature = "http_client")] pub async fn extract_url(&self, url: &str) -> crate::Result { + let start = std::time::Instant::now(); let fetched = crate::fetch::fetch(url, &self.config).await?; - self.extract_bytes(&fetched.bytes, fetched.content_type.as_deref()) - .await + let budget = Budget::starting_at(&self.config.limits, start); + let hint = DocumentHint { + content_type: fetched.content_type.as_deref(), + file_name: None, + }; + self.extract_document_with(&fetched.bytes, hint, &budget) + .map(|e| e.markdown) } /// PDF extraction is not implemented yet; report it explicitly rather than @@ -274,7 +287,7 @@ mod tests { #[tokio::test] async fn test_builder_defaults() { let m = Markkit::builder().build(); - assert_eq!(m.config().max_input_size, 50 * 1024 * 1024); + assert_eq!(m.config().limits.max_input_size, 50 * 1024 * 1024); assert!(m.config().preserve_links); } diff --git a/crates/markkit/src/fetch.rs b/crates/markkit/src/fetch.rs index bc4a127..8e19772 100644 --- a/crates/markkit/src/fetch.rs +++ b/crates/markkit/src/fetch.rs @@ -12,7 +12,7 @@ //! layers 1-2 still apply, layer 3 cannot. Documented in the threat model. //! - Body bounded by `max_input_size` while streaming (declared //! `Content-Length` rejected up front); whole fetch bounded by -//! `max_processing_secs`. +//! `limits.timeout`. //! - Policy and resolver run inside reqwest and can only return opaque //! errors, so they record the block reason in a per-call slot that is //! turned into [`Error::UrlBlocked`] afterwards. @@ -47,7 +47,7 @@ pub(crate) async fn fetch(url: &str, cfg: &MarkkitBuilder) -> crate::Result crate::Result crate::Result crate::Result { - let max_redirects = cfg.max_redirects; + let max_redirects = cfg.limits.max_redirects; let policy_cfg = cfg.clone(); let policy_slot = blocked.clone(); let policy = reqwest::redirect::Policy::custom(move |attempt| { @@ -114,11 +114,11 @@ fn client(cfg: &MarkkitBuilder, blocked: &Blocked) -> crate::Result { + Owned(Budget), + Borrowed(&'a Budget), +} /// Parses HTML into the canonical IR. +/// +/// Limits: nesting past `limits.max_nesting_depth` is flattened to text; DOM +/// node count and the deadline are enforced ([`Error::ResourceLimit`]). pub struct HtmlParser<'a> { config: &'a MarkkitBuilder, + budget: BudgetRef<'a>, + max_depth: usize, } impl<'a> HtmlParser<'a> { + /// Parser with a fresh per-call budget (deadline starts now). pub fn new(config: &'a MarkkitBuilder) -> Self { - Self { config } + Self { + config, + budget: BudgetRef::Owned(Budget::new(&config.limits)), + max_depth: config.limits.nesting_depth(), + } + } + + pub(crate) fn with_budget(config: &'a MarkkitBuilder, budget: &'a Budget) -> Self { + Self { + config, + budget: BudgetRef::Borrowed(budget), + max_depth: config.limits.nesting_depth(), + } + } + + fn budget(&self) -> &Budget { + match &self.budget { + BudgetRef::Owned(b) => b, + BudgetRef::Borrowed(b) => b, + } } /// Parse an HTML string into a [`Document`]. pub fn parse(&self, html: &str) -> Result { let dom = Html::parse_document(html); + // html5ever is not interruptible: check right after it finishes. + self.budget().check_time()?; + self.budget().add_nodes(dom.tree.nodes().count())?; let root = dom.root_element(); let content_nodes: Vec> = @@ -50,6 +81,8 @@ impl<'a> HtmlParser<'a> { for content_root in content_nodes { self.convert_children(content_root, &exclude_selectors, &mut nodes, 0); } + // The walk stops early once the deadline passes; report it here. + self.budget().check_time()?; Ok(Document::new(normalize_blocks(nodes))) } @@ -65,7 +98,10 @@ impl<'a> HtmlParser<'a> { out: &mut Vec, depth: usize, ) { - if depth >= MAX_DEPTH { + if self.budget().expired() { + return; + } + if depth >= self.max_depth { // At max depth, just extract text let text: String = parent.text().collect(); let trimmed = text.trim(); @@ -110,7 +146,10 @@ impl<'a> HtmlParser<'a> { exclude: &[Selector], depth: usize, ) -> Option { - if depth >= MAX_DEPTH { + if self.budget().expired() { + return None; + } + if depth >= self.max_depth { let text: String = el.text().collect(); let trimmed = text.trim(); if trimmed.is_empty() { @@ -300,7 +339,10 @@ impl<'a> HtmlParser<'a> { depth: usize, ) -> Vec { let mut nodes = Vec::new(); - if depth >= MAX_DEPTH { + if self.budget().expired() { + return nodes; + } + if depth >= self.max_depth { let text: String = el.text().collect(); let trimmed = text.trim(); if !trimmed.is_empty() { diff --git a/crates/markkit/src/lib.rs b/crates/markkit/src/lib.rs index d6453e2..9b5adfd 100644 --- a/crates/markkit/src/lib.rs +++ b/crates/markkit/src/lib.rs @@ -53,6 +53,7 @@ pub mod format; #[cfg(feature = "html")] pub mod html; pub mod ir; +pub mod limits; pub mod render; #[cfg(feature = "spreadsheet")] mod spreadsheet; @@ -64,6 +65,7 @@ pub use error::Error; pub use extract::{Extraction, ExtractionStats, Markkit}; pub use format::{DocumentHint, Format}; pub use ir::Document; +pub use limits::Limits; pub use render::MarkdownRenderer; /// Result type alias for markkit operations. diff --git a/crates/markkit/src/limits.rs b/crates/markkit/src/limits.rs new file mode 100644 index 0000000..4725e96 --- /dev/null +++ b/crates/markkit/src/limits.rs @@ -0,0 +1,317 @@ +//! Resource limits for extraction. +//! +//! Every input to markkit is untrusted, so every resource an input can drive +//! (bytes in, bytes out, time, nesting, node count, decompression, grid size, +//! redirects) has a limit here. Defaults are generous: real documents should +//! never hit them; hostile ones always do. +//! +//! Decisions: +//! - One [`Limits`] struct (bashkit `ExecutionLimits` style) instead of loose +//! builder fields, so the CLI and the Python/Node bindings map one shape. +//! - Violations are hard errors ([`Error::ResourceLimit`]), never silently +//! truncated output. Exceptions, kept for compatibility: per-sheet rows and +//! columns truncate with a warning, and HTML nesting past +//! `max_nesting_depth` is flattened to plain text. +//! - Time is checked cooperatively ([`Budget`]) in parser and renderer loops. +//! Third-party parse steps (html5ever tokenizing, calamine cell decoding) are +//! not interruptible; they are bounded by input size and checked right after. +//! - `max_nesting_depth` is clamped to [`HARD_MAX_NESTING_DEPTH`] because the +//! HTML walker, DOCX conversion and IR drop recurse on the host stack. +//! +//! See `knowledge/security/threat-model.md` (T2, T6) for the threat mapping. + +use std::cell::Cell; +use std::time::{Duration, Instant}; + +use crate::error::Error; + +// THREAT[T2b]: recursion depth must stay safe on small host stacks whatever +// the embedder configures. Measured worst case (nested blockquote / inline +// tags): 1024 levels fit a 2 MiB thread in release; debug frames are ~6x +// larger (default 256 fits 2 MiB in debug). Test: limits::*hard_ceiling*. +/// Upper bound applied to [`Limits::max_nesting_depth`]. +pub const HARD_MAX_NESTING_DEPTH: usize = 1024; + +/// Resource limits applied to every extraction. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct Limits { + /// Maximum input size in bytes, also the cap on a fetched URL body. + /// Default: 50 MiB. + pub max_input_size: usize, + /// Maximum Markdown output size in bytes. Default: 128 MiB. + pub max_output_size: usize, + /// Wall-clock budget for one call: URL fetch, parsing and rendering. + /// Default: 60 s. + pub timeout: Duration, + /// Maximum element nesting. HTML deeper than this is flattened to text; + /// XML parts (DOCX) deeper than this are rejected. Clamped to + /// [`HARD_MAX_NESTING_DEPTH`]. Default: 256. + pub max_nesting_depth: usize, + /// Maximum parsed nodes (HTML DOM nodes; XML elements and text nodes summed + /// over DOCX parts). Default: 5 000 000. + pub max_nodes: usize, + /// Maximum number of entries in a ZIP container (DOCX/XLSX/ODS). Default: 10 000. + pub max_archive_entries: usize, + /// Maximum decompressed size of a single ZIP entry in bytes. Default: 64 MiB. + pub max_archive_entry_size: u64, + /// Maximum total decompressed size of a ZIP container in bytes. Default: 256 MiB. + pub max_archive_total_size: u64, + /// Maximum data rows rendered per sheet (header excluded); extra rows are + /// dropped with a warning. Default: 10 000. + pub max_sheet_rows: usize, + /// Maximum columns rendered per sheet; extra columns are dropped with a + /// warning. Default: 256. + pub max_sheet_cols: usize, + /// Maximum cell grid a workbook may expand to while parsing (XLS/ODS + /// pre-scan, summed over sheets). Default: 4 000 000. + pub max_spreadsheet_cells: u64, + /// Maximum redirects `extract_url` follows. Default: 5. + pub max_redirects: usize, +} + +impl Default for Limits { + fn default() -> Self { + Self { + max_input_size: 50 * 1024 * 1024, + max_output_size: 128 * 1024 * 1024, + timeout: Duration::from_secs(60), + max_nesting_depth: 256, + max_nodes: 5_000_000, + max_archive_entries: 10_000, + max_archive_entry_size: 64 * 1024 * 1024, + max_archive_total_size: 256 * 1024 * 1024, + max_sheet_rows: 10_000, + max_sheet_cols: 256, + max_spreadsheet_cells: 4_000_000, + max_redirects: 5, + } + } +} + +impl Limits { + /// Default limits. + pub fn new() -> Self { + Self::default() + } + + /// Effective nesting depth after the hard ceiling. + pub fn nesting_depth(&self) -> usize { + self.max_nesting_depth.min(HARD_MAX_NESTING_DEPTH) + } + + pub fn max_input_size(mut self, bytes: usize) -> Self { + self.max_input_size = bytes; + self + } + + pub fn max_output_size(mut self, bytes: usize) -> Self { + self.max_output_size = bytes; + self + } + + pub fn timeout(mut self, timeout: Duration) -> Self { + self.timeout = timeout; + self + } + + pub fn max_nesting_depth(mut self, depth: usize) -> Self { + self.max_nesting_depth = depth; + self + } + + pub fn max_nodes(mut self, nodes: usize) -> Self { + self.max_nodes = nodes; + self + } + + pub fn max_archive_entries(mut self, n: usize) -> Self { + self.max_archive_entries = n; + self + } + + pub fn max_archive_entry_size(mut self, bytes: u64) -> Self { + self.max_archive_entry_size = bytes; + self + } + + pub fn max_archive_total_size(mut self, bytes: u64) -> Self { + self.max_archive_total_size = bytes; + self + } + + pub fn max_sheet_rows(mut self, rows: usize) -> Self { + self.max_sheet_rows = rows; + self + } + + pub fn max_sheet_cols(mut self, cols: usize) -> Self { + self.max_sheet_cols = cols; + self + } + + pub fn max_spreadsheet_cells(mut self, cells: u64) -> Self { + self.max_spreadsheet_cells = cells; + self + } + + pub fn max_redirects(mut self, n: usize) -> Self { + self.max_redirects = n; + self + } + + /// Reject inputs larger than `max_input_size`. + pub(crate) fn check_input(&self, len: usize) -> crate::Result<()> { + if len > self.max_input_size { + return Err(Error::ResourceLimit(format!( + "input size {len} exceeds max_input_size {}", + self.max_input_size + ))); + } + Ok(()) + } +} + +// Budget is only driven by the feature-gated parsers. +/// How many [`Budget::expired`] calls share one clock read. +#[cfg_attr( + not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + allow(dead_code) +)] +const CLOCK_EVERY: u32 = 256; + +/// Per-call deadline and node counter, shared by reference through parsers and +/// the renderer (interior mutability so `&self` walkers can charge it). +#[derive(Debug)] +#[cfg_attr( + not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + allow(dead_code) +)] +pub(crate) struct Budget { + deadline: Option, + timeout: Duration, + ticks: Cell, + expired: Cell, + #[cfg_attr(not(any(feature = "html", feature = "docx")), allow(dead_code))] + nodes: Cell, + #[cfg_attr(not(any(feature = "html", feature = "docx")), allow(dead_code))] + max_nodes: usize, +} + +#[cfg_attr( + not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + allow(dead_code) +)] +impl Budget { + /// Budget starting now. + pub fn new(limits: &Limits) -> Self { + Self::starting_at(limits, Instant::now()) + } + + /// Budget whose clock started at `start` (e.g. before a URL fetch). + pub fn starting_at(limits: &Limits, start: Instant) -> Self { + Self { + // Overflow (e.g. Duration::MAX) means "no deadline". + deadline: start.checked_add(limits.timeout), + timeout: limits.timeout, + ticks: Cell::new(0), + expired: Cell::new(false), + nodes: Cell::new(0), + max_nodes: limits.max_nodes, + } + } + + pub fn deadline(&self) -> Option { + self.deadline + } + + /// Cheap periodic deadline check for hot loops; reads the clock every + /// [`CLOCK_EVERY`] calls. Sticky once expired. + pub fn expired(&self) -> bool { + if self.expired.get() { + return true; + } + let t = self.ticks.get().wrapping_add(1); + self.ticks.set(t); + if t.is_multiple_of(CLOCK_EVERY) { + self.read_clock(); + } + self.expired.get() + } + + fn read_clock(&self) { + if let Some(d) = self.deadline + && Instant::now() >= d + { + self.expired.set(true); + } + } + + /// Error if the deadline has passed (reads the clock). + pub fn check_time(&self) -> crate::Result<()> { + self.read_clock(); + if self.expired.get() { + return Err(self.timeout_error()); + } + Ok(()) + } + + pub fn timeout_error(&self) -> Error { + Error::ResourceLimit(format!("processing exceeded timeout of {:?}", self.timeout)) + } + + /// Charge `n` parsed nodes against `max_nodes`. + #[cfg_attr(not(any(feature = "html", feature = "docx")), allow(dead_code))] + pub fn add_nodes(&self, n: usize) -> crate::Result<()> { + let total = self.nodes.get().saturating_add(n); + self.nodes.set(total); + if total > self.max_nodes { + return Err(Error::ResourceLimit(format!( + "document has more than max_nodes {} nodes", + self.max_nodes + ))); + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn nesting_depth_clamped_to_hard_ceiling() { + let l = Limits::default().max_nesting_depth(usize::MAX); + assert_eq!(l.nesting_depth(), HARD_MAX_NESTING_DEPTH); + assert_eq!(Limits::default().nesting_depth(), 256); + } + + #[test] + fn zero_timeout_expires() { + let b = Budget::new(&Limits::default().timeout(Duration::ZERO)); + assert!(matches!(b.check_time(), Err(Error::ResourceLimit(_)))); + assert!((0..CLOCK_EVERY).any(|_| b.expired())); + } + + #[test] + fn huge_timeout_never_expires() { + let b = Budget::new(&Limits::default().timeout(Duration::MAX)); + assert!(b.check_time().is_ok()); + assert!(!(0..CLOCK_EVERY * 2).any(|_| b.expired())); + } + + #[test] + fn node_budget_accumulates() { + let b = Budget::new(&Limits::default().max_nodes(10)); + assert!(b.add_nodes(6).is_ok()); + assert!(b.add_nodes(4).is_ok()); + assert!(matches!(b.add_nodes(1), Err(Error::ResourceLimit(_)))); + } + + #[test] + fn input_check() { + let l = Limits::default().max_input_size(3); + assert!(l.check_input(3).is_ok()); + assert!(matches!(l.check_input(4), Err(Error::ResourceLimit(_)))); + } +} diff --git a/crates/markkit/src/render/markdown.rs b/crates/markkit/src/render/markdown.rs index eec02a1..5fee315 100644 --- a/crates/markkit/src/render/markdown.rs +++ b/crates/markkit/src/render/markdown.rs @@ -1,30 +1,102 @@ //! IR → Markdown renderer. +//! +//! Decision: output size and the deadline are checked per block, list item and +//! table row (not per byte). Tables pad every row to the widest row, so output +//! can be quadratic in input; checking per row keeps any buffer within +//! `max_output_size` plus one row. Nested renderers (blockquotes, list items) +//! inherit the limits and report a trip back to their parent. + +use std::time::Instant; use crate::ir::{Document, ListItem, Node}; +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Trip { + Output, + Time, +} + /// Renders a [`Document`] IR into Markdown text. pub struct MarkdownRenderer { /// Output buffer. buf: String, + max_output: usize, + deadline: Option, + tripped: Option, } impl MarkdownRenderer { pub fn new() -> Self { Self { buf: String::with_capacity(4096), + max_output: usize::MAX, + deadline: None, + tripped: None, } } - /// Render a document to a Markdown string. + /// Render a document to a Markdown string (no limits; the IR is trusted). pub fn render(&mut self, doc: &Document) -> String { self.buf.clear(); self.render_nodes(&doc.nodes, 0); normalize_output(&self.buf) } + /// Render with `max_output` bytes and the budget's deadline enforced. + #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + pub(crate) fn render_bounded( + &mut self, + doc: &Document, + max_output: usize, + budget: &crate::limits::Budget, + ) -> crate::Result { + self.max_output = max_output; + self.deadline = budget.deadline(); + self.tripped = None; + let out = self.render(doc); + match self.tripped { + Some(Trip::Time) => Err(budget.timeout_error()), + Some(Trip::Output) => Err(output_error(max_output)), + None if out.len() > max_output => Err(output_error(max_output)), + None => Ok(out), + } + } + + /// Nested renderer sharing this renderer's limits. + fn sub(&self) -> Self { + Self { + max_output: self.max_output, + deadline: self.deadline, + ..Self::new() + } + } + + /// Fold a nested renderer's trip into this one. + fn absorb(&mut self, sub: &Self) { + if self.tripped.is_none() { + self.tripped = sub.tripped; + } + } + + /// True once a limit is hit; rendering then stops at the next check. + fn over_limit(&mut self) -> bool { + if self.tripped.is_some() { + return true; + } + if self.buf.len() > self.max_output { + self.tripped = Some(Trip::Output); + } else if self.deadline.is_some_and(|d| Instant::now() >= d) { + self.tripped = Some(Trip::Time); + } + self.tripped.is_some() + } + /// Render a slice of nodes. fn render_nodes(&mut self, nodes: &[Node], depth: usize) { for (i, node) in nodes.iter().enumerate() { + if self.over_limit() { + return; + } if i > 0 && is_block(node) && is_block(&nodes[i - 1]) { self.buf.push('\n'); } @@ -61,8 +133,9 @@ impl MarkdownRenderer { Node::BlockQuote { children } => { let inner = { - let mut sub = MarkdownRenderer::new(); + let mut sub = self.sub(); sub.render_nodes(children, depth); + self.absorb(&sub); normalize_output(&sub.buf) }; for line in inner.lines() { @@ -117,6 +190,9 @@ impl MarkdownRenderer { let start_num = start.unwrap_or(1); for (i, item) in items.iter().enumerate() { + if self.over_limit() { + return; + } let marker = if ordered { format!("{}{}. ", indent, start_num + i as u64) } else { @@ -146,8 +222,9 @@ impl MarkdownRenderer { self.render_list(*sub_ord, *sub_start, sub_items, depth + 1); } _ => { - let mut sub = MarkdownRenderer::new(); + let mut sub = self.sub(); sub.render_node(child, depth + 1); + self.absorb(&sub); let text = sub.buf.trim_end().to_string(); self.buf.push_str(&text); self.buf.push('\n'); @@ -163,8 +240,9 @@ impl MarkdownRenderer { self.render_list(*sub_ord, *sub_start, sub_items, depth + 1); } _ => { - let mut sub = MarkdownRenderer::new(); + let mut sub = self.sub(); sub.render_node(child, depth + 1); + self.absorb(&sub); let inner = sub.buf.trim_end().to_string(); let sub_indent = " ".repeat(depth + 1); for line in inner.lines() { @@ -277,6 +355,9 @@ impl MarkdownRenderer { // Data rows for row in rows { + if self.over_limit() { + return; + } self.buf.push('|'); for i in 0..col_count { self.buf.push(' '); @@ -296,6 +377,11 @@ impl Default for MarkdownRenderer { } } +#[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] +fn output_error(max: usize) -> crate::Error { + crate::Error::ResourceLimit(format!("output exceeds max_output_size {max}")) +} + /// Trim leading and trailing whitespace from inline content rendered since `start`. fn trim_trailing_inline(buf: &mut String, start: usize) { // Trim trailing spaces diff --git a/crates/markkit/src/spreadsheet.rs b/crates/markkit/src/spreadsheet.rs index 2ca280c..5ff1383 100644 --- a/crates/markkit/src/spreadsheet.rs +++ b/crates/markkit/src/spreadsheet.rs @@ -29,6 +29,7 @@ use crate::config::MarkkitBuilder; use crate::error::Error; use crate::format::Format; use crate::ir::{Document, Node}; +use crate::limits::Budget; use crate::render::escape_table_cell; pub(crate) struct SheetOutput { @@ -62,7 +63,11 @@ enum Container { Xls, } -pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result { +pub(crate) fn parse( + bytes: &[u8], + config: &MarkkitBuilder, + budget: &Budget, +) -> Result { let container = if bytes.starts_with(b"PK") { let mut zip = ZipGuard::open(bytes, config).map_err(archive_err)?; let has = @@ -84,7 +89,7 @@ pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result Result Result Result<(), calamine::XlsxError> { let mut reader = wb.worksheet_cells_reader(&name)?; while let Some(cell) = reader.next_cell()? { + if budget.expired() { + break; + } let (r, c) = cell.get_position(); grid.add(r, c, fmt_ref(cell.get_value())); } Ok(()) })(); + budget.check_time()?; match res { Ok(()) => out.push_sheet(&name, grid), Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), @@ -121,13 +131,13 @@ pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result { let mut wb: Ods<_> = Ods::new(Cursor::new(bytes)).map_err(err)?; let sel = out.select(wb.sheets_metadata()); - read_ranges(&mut wb, sel, &mut out, config); + read_ranges(&mut wb, sel, &mut out, config, budget)?; out.finish(Format::Ods) } Container::Xls => { let mut wb: Xls<_> = Xls::new(Cursor::new(bytes)).map_err(err)?; let sel = out.select(wb.sheets_metadata()); - read_ranges(&mut wb, sel, &mut out, config); + read_ranges(&mut wb, sel, &mut out, config, budget)?; out.finish(Format::Xls) } } @@ -138,7 +148,9 @@ fn read_ranges( sel: Vec<(String, bool)>, out: &mut Collector, config: &MarkkitBuilder, -) where + budget: &Budget, +) -> Result<(), Error> +where RS: Read + Seek, R: Reader, R::Error: std::fmt::Display, @@ -147,17 +159,25 @@ fn read_ranges( if !include { continue; } - match wb.worksheet_range(&name) { + let range = wb.worksheet_range(&name); + // calamine decodes the whole sheet before returning: check after it. + budget.check_time()?; + match range { Ok(range) => { let mut grid = Grid::new(config); for (r, c, v) in range.used_cells() { + if budget.expired() { + break; + } grid.add(r as u32, c as u32, fmt_data(v)); } + budget.check_time()?; out.push_sheet(&name, grid); } Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), } } + Ok(()) } // ─── Cell formatting ──────────────────────────────────────────────── @@ -238,8 +258,8 @@ impl Grid { fn new(config: &MarkkitBuilder) -> Self { Self { rows: BTreeMap::new(), - max_rows: config.max_sheet_rows, - max_cols: config.max_sheet_cols.max(1), + max_rows: config.limits.max_sheet_rows, + max_cols: config.limits.max_sheet_cols.max(1), truncated_rows: false, truncated_cols: false, } @@ -437,7 +457,7 @@ fn prescan_xls(bytes: &[u8], config: &MarkkitBuilder) -> Result<(), Error> { .find(|n| comp.is_stream(n)) .ok_or_else(|| err("XLS has no Workbook stream"))?; let mut stream = Vec::new(); - let cap = config.max_archive_entry_size; + let cap = config.limits.max_archive_entry_size; comp.open_stream(name) .map_err(|e| err(format!("cannot open Workbook stream: {e}")))? .take(cap.saturating_add(1)) @@ -448,7 +468,7 @@ fn prescan_xls(bytes: &[u8], config: &MarkkitBuilder) -> Result<(), Error> { "XLS Workbook stream exceeds {cap} bytes" ))); } - scan_biff(&stream, config.max_spreadsheet_cells) + scan_biff(&stream, config.limits.max_spreadsheet_cells) } /// Estimate the cells calamine will allocate and reject inputs that would diff --git a/crates/markkit/src/xml.rs b/crates/markkit/src/xml.rs index a2881ba..45187af 100644 --- a/crates/markkit/src/xml.rs +++ b/crates/markkit/src/xml.rs @@ -5,13 +5,15 @@ //! matching on local names keeps us robust to unusual prefix choices without a //! full namespace resolver. quick-xml never expands DTD-declared entities, so //! billion-laughs / XXE payloads surface as unresolved references, which we drop. -//! Tree depth is capped and the tree is built iteratively (no recursion on input). +//! Tree depth (`limits.max_nesting_depth`), node count (`limits.max_nodes`, +//! shared across parts) and the deadline are enforced while building; the tree +//! is built iteratively (no recursion on input). use quick_xml::events::{BytesStart, Event}; use quick_xml::{Reader, XmlVersion}; -/// Maximum element nesting accepted in a single XML part. -pub(crate) const MAX_XML_DEPTH: usize = 256; +use crate::error::Error; +use crate::limits::Budget; #[derive(Debug)] pub(crate) enum XmlError { @@ -126,9 +128,16 @@ fn push_text(parent: &mut Element, text: &str) { } } +fn limit(e: Error) -> XmlError { + match e { + Error::ResourceLimit(m) => XmlError::Limit(m), + other => XmlError::Invalid(other.to_string()), + } +} + /// Parse an XML document into a tree. Returns a synthetic root whose children /// are the document's top-level nodes. -pub(crate) fn parse(bytes: &[u8]) -> Result { +pub(crate) fn parse(bytes: &[u8], max_depth: usize, budget: &Budget) -> Result { let mut reader = Reader::from_reader(bytes); reader.config_mut().check_end_names = false; let mut stack: Vec = vec![Element::default()]; @@ -137,11 +146,20 @@ pub(crate) fn parse(bytes: &[u8]) -> Result { let event = reader .read_event_into(&mut buf) .map_err(|e| XmlError::Invalid(format!("malformed XML: {e}")))?; + if matches!( + event, + Event::Start(_) | Event::Empty(_) | Event::Text(_) | Event::CData(_) + ) { + budget.add_nodes(1).map_err(limit)?; + if budget.expired() { + return Err(limit(budget.timeout_error())); + } + } match event { Event::Start(s) => { - if stack.len() > MAX_XML_DEPTH { + if stack.len() > max_depth { return Err(XmlError::Limit(format!( - "XML nesting deeper than {MAX_XML_DEPTH}" + "XML nesting deeper than max_nesting_depth {max_depth}" ))); } stack.push(element_from(&s)); @@ -202,6 +220,11 @@ pub(crate) fn parse(bytes: &[u8]) -> Result { #[cfg(test)] mod tests { use super::*; + use crate::limits::Limits; + + fn parse(bytes: &[u8]) -> Result { + super::parse(bytes, 256, &Budget::new(&Limits::default())) + } #[test] fn parses_local_names_text_and_refs() { @@ -216,8 +239,35 @@ mod tests { #[test] fn depth_limited() { - let deep = "".repeat(MAX_XML_DEPTH + 10); + let deep = "".repeat(256 + 10); assert!(matches!(parse(deep.as_bytes()), Err(XmlError::Limit(_)))); + let ok = format!("{}{}", "".repeat(100), "".repeat(100)); + assert!(parse(ok.as_bytes()).is_ok()); + let budget = Budget::new(&Limits::default()); + assert!(matches!( + super::parse(ok.as_bytes(), 50, &budget), + Err(XmlError::Limit(_)) + )); + } + + #[test] + fn node_count_limited() { + let many = format!("{}", "".repeat(100)); + let budget = Budget::new(&Limits::default().max_nodes(50)); + assert!(matches!( + super::parse(many.as_bytes(), 256, &budget), + Err(XmlError::Limit(m)) if m.contains("max_nodes") + )); + } + + #[test] + fn deadline_enforced() { + let many = format!("{}", "".repeat(1000)); + let budget = Budget::new(&Limits::default().timeout(std::time::Duration::ZERO)); + assert!(matches!( + super::parse(many.as_bytes(), 256, &budget), + Err(XmlError::Limit(m)) if m.contains("timeout") + )); } #[test] diff --git a/crates/markkit/tests/html_integration.rs b/crates/markkit/tests/html_integration.rs index afd18f6..8f961b5 100644 --- a/crates/markkit/tests/html_integration.rs +++ b/crates/markkit/tests/html_integration.rs @@ -593,7 +593,10 @@ fn code_block_with_backticks() { #[test] fn config_max_processing_secs() { let m = Markkit::builder().max_processing_secs(30).build(); - assert_eq!(m.config().max_processing_secs, 30); + assert_eq!( + m.config().limits.timeout, + std::time::Duration::from_secs(30) + ); } #[test] diff --git a/crates/markkit/tests/limits.rs b/crates/markkit/tests/limits.rs new file mode 100644 index 0000000..046240f --- /dev/null +++ b/crates/markkit/tests/limits.rs @@ -0,0 +1,269 @@ +//! Resource limits (threat model T2/T6): every limit in `markkit::Limits` is +//! enforced, defaults are generous, and the nesting hard ceiling keeps deep +//! input off the host stack. + +#![cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + +mod common; + +use std::time::Duration; + +use markkit::{DocumentHint, Error, Limits, Markkit}; + +#[allow(dead_code)] +const DOCX_CT: &str = "application/vnd.openxmlformats-officedocument.wordprocessingml.document"; +#[allow(dead_code)] +const XLSX_CT: &str = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; + +fn limit_msg(r: markkit::Result) -> String { + match r { + Err(Error::ResourceLimit(m)) => m, + other => panic!("expected ResourceLimit, got {other:?}"), + } +} + +#[test] +fn defaults_are_generous() { + let l = Limits::default(); + assert_eq!(l.max_input_size, 50 * 1024 * 1024); + assert_eq!(l.max_output_size, 128 * 1024 * 1024); + assert_eq!(l.timeout, Duration::from_secs(60)); + assert_eq!(l.max_nesting_depth, 256); + assert_eq!(l.max_nodes, 5_000_000); + assert_eq!(l.max_redirects, 5); + assert_eq!(Markkit::builder().build().config().limits, l); +} + +#[test] +fn builder_limits_replaces_all() { + let l = Limits::default().max_output_size(7).max_nodes(9); + let m = Markkit::builder().limits(l.clone()).build(); + assert_eq!(m.config().limits, l); + let m = Markkit::builder() + .max_processing_secs(3) + .max_nesting_depth(12) + .build(); + assert_eq!(m.config().limits.timeout, Duration::from_secs(3)); + assert_eq!(m.config().limits.max_nesting_depth, 12); +} + +/// Stack for hard-ceiling tests: 2 MiB (Rust's default thread stack) in +/// release; debug frames are ~6x larger, so debug runs get 8 MiB. +#[allow(dead_code)] +fn small_stack() -> usize { + if cfg!(debug_assertions) { + 8 << 20 + } else { + 2 << 20 + } +} + +// ─── HTML ─────────────────────────────────────────────────────────── + +#[cfg(feature = "html")] +mod html { + use super::*; + use markkit::limits::HARD_MAX_NESTING_DEPTH; + + fn deep(tag: &str, n: usize, inner: &str) -> String { + format!( + "{}{inner}{}", + format!("<{tag}>").repeat(n), + format!("").repeat(n) + ) + } + + #[test] + fn output_cap_rejects_large_output() { + let html = "

hello world

".repeat(1000); + let m = Markkit::builder().max_output_size(1000).build(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + let m = Markkit::builder().max_output_size(100_000).build(); + assert!(m.extract_html_sync(&html).is_ok()); + } + + #[test] + fn output_cap_stops_table_amplification() { + // One wide row pads every other row: output ~ rows x cols. + let wide = format!("{}", "x".repeat(2000)); + let html = format!( + "{wide}{}
", + "y".repeat(2000) + ); + let m = Markkit::builder().max_output_size(1 << 20).build(); + let start = std::time::Instant::now(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + assert!(start.elapsed() < Duration::from_secs(10)); + } + + #[test] + fn output_cap_applies_inside_nested_renderers() { + let wide = format!("{}", "x".repeat(500)); + let table = format!("{wide}{}
", "y".repeat(500)); + for html in [ + format!("
{table}
"), + format!("

  • a

    {table}
"), + ] { + let m = Markkit::builder().max_output_size(64 * 1024).build(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + } + } + + #[test] + fn zero_timeout_rejects() { + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(m.extract_html_sync("

x

")).contains("timeout")); + let r = m.extract_document_sync(b"

x

", DocumentHint::content_type("text/html")); + assert!(limit_msg(r).contains("timeout")); + } + + #[test] + fn node_cap_rejects() { + let html = "

a

".repeat(1000); + let m = Markkit::builder().max_nodes(500).build(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_nodes")); + assert!(limit_msg(m.parse_html(&html)).contains("max_nodes")); + let m = Markkit::builder().max_nodes(10_000).build(); + assert!(m.extract_html_sync(&html).is_ok()); + } + + #[test] + fn nesting_depth_configurable_flattens() { + let html = deep("blockquote", 20, "

Deep

"); + let full = Markkit::builder().build().extract_html_sync(&html).unwrap(); + assert!(full.contains(&"> ".repeat(20))); + let m = Markkit::builder().max_nesting_depth(5).build(); + let md = m.extract_html_sync(&html).unwrap(); + assert!(md.contains("Deep")); + assert!(!md.contains(&"> ".repeat(6))); + } + + #[test] + fn hard_ceiling_survives_small_stack() { + // Configured depth is clamped; deep nesting of recursive constructs + // must fit a small thread stack. + let inputs = [ + deep("blockquote", HARD_MAX_NESTING_DEPTH * 4, "

q

"), + deep("ul", HARD_MAX_NESTING_DEPTH * 4, "
  • l
    • "), + deep("div", HARD_MAX_NESTING_DEPTH * 4, "

    d

    "), + deep("b", HARD_MAX_NESTING_DEPTH * 4, "t"), + ]; + std::thread::Builder::new() + .stack_size(small_stack()) + .spawn(move || { + let m = Markkit::builder().max_nesting_depth(usize::MAX).build(); + for html in inputs { + m.extract_html_sync(&html).unwrap(); + } + }) + .unwrap() + .join() + .expect("deep input overflowed a small stack"); + } +} + +// ─── DOCX ─────────────────────────────────────────────────────────── + +#[cfg(feature = "docx")] +mod docx { + use super::*; + use common::*; + use markkit::limits::HARD_MAX_NESTING_DEPTH; + + fn run(m: &Markkit, bytes: &[u8]) -> markkit::Result { + m.extract_document_sync(bytes, DocumentHint::content_type(DOCX_CT)) + } + + #[test] + fn node_cap_rejects() { + let body: String = (0..200).map(|i| para(&format!("p{i}"))).collect(); + let bytes = docx(&body); + let m = Markkit::builder().max_nodes(100).build(); + assert!(limit_msg(run(&m, &bytes)).contains("max_nodes")); + assert!(run(&Markkit::builder().build(), &bytes).is_ok()); + } + + #[test] + fn nesting_depth_configurable() { + let body = format!( + "{}{}{}", + "".repeat(10), + para("x"), + "".repeat(10) + ); + let bytes = docx(&body); + assert!(run(&Markkit::builder().build(), &bytes).is_ok()); + let m = Markkit::builder().max_nesting_depth(8).build(); + assert!(limit_msg(run(&m, &bytes)).contains("max_nesting_depth")); + } + + #[test] + fn hard_ceiling_survives_small_stack() { + let n = HARD_MAX_NESTING_DEPTH / 2 - 8; + let body = format!( + "{}{}{}", + "".repeat(n), + para("x"), + "".repeat(n) + ); + let bytes = docx(&body); + let deeper = docx(&"".repeat(HARD_MAX_NESTING_DEPTH * 2)); + std::thread::Builder::new() + .stack_size(small_stack()) + .spawn(move || { + let m = Markkit::builder().max_nesting_depth(usize::MAX).build(); + assert!(run(&m, &bytes).unwrap().markdown.contains('x')); + assert!(limit_msg(run(&m, &deeper)).contains("max_nesting_depth")); + }) + .unwrap() + .join() + .expect("deep DOCX overflowed a small stack"); + } + + #[test] + fn zero_timeout_rejects() { + let bytes = docx(¶("x")); + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(run(&m, &bytes)).contains("timeout")); + } + + #[test] + fn output_cap_rejects() { + let body: String = (0..200).map(|i| para(&format!("paragraph {i}"))).collect(); + let bytes = docx(&body); + let m = Markkit::builder().max_output_size(500).build(); + assert!(limit_msg(run(&m, &bytes)).contains("max_output_size")); + } +} + +// ─── Spreadsheets ─────────────────────────────────────────────────── + +#[cfg(feature = "spreadsheet")] +mod spreadsheet { + use super::*; + use common::*; + + fn sheet() -> Vec { + let rows = (0..200) + .map(|i| vec![Cell::Str(format!("row {i}")), Cell::N(i as f64)]) + .collect(); + xlsx(&[Sheet::new("S", rows)]) + } + + fn run(m: &Markkit, bytes: &[u8]) -> markkit::Result { + m.extract_document_sync(bytes, DocumentHint::content_type(XLSX_CT)) + } + + #[test] + fn zero_timeout_rejects() { + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(run(&m, &sheet())).contains("timeout")); + } + + #[test] + fn output_cap_rejects() { + let m = Markkit::builder().max_output_size(500).build(); + assert!(limit_msg(run(&m, &sheet())).contains("max_output_size")); + assert!(run(&Markkit::builder().build(), &sheet()).is_ok()); + } +} diff --git a/examples/configuration.rs b/examples/configuration.rs index d9213a7..c307d9d 100644 --- a/examples/configuration.rs +++ b/examples/configuration.rs @@ -34,4 +34,17 @@ fn main() { Ok(_) => println!("Unexpected success"), Err(e) => println!("--- max_input_size(10) ---\nError: {e}"), } + + // All limits at once: output cap, timeout, nesting depth, node count, ... + let bounded = Markkit::builder() + .limits( + markkit::Limits::default() + .max_output_size(20) + .timeout(std::time::Duration::from_secs(5)), + ) + .build(); + match bounded.extract_html_sync(html) { + Ok(_) => println!("Unexpected success"), + Err(e) => println!("--- max_output_size(20) ---\nError: {e}"), + } } diff --git a/knowledge/formats/html-extraction.md b/knowledge/formats/html-extraction.md index c1bcc1a..e9c92d1 100644 --- a/knowledge/formats/html-extraction.md +++ b/knowledge/formats/html-extraction.md @@ -121,7 +121,7 @@ Trimming at render time: ## Depth Limiting -Constant: `MAX_DEPTH = 200` +Limit: `limits.max_nesting_depth` (default 256, clamped to `HARD_MAX_NESTING_DEPTH` = 1024) Behavior at depth limit: - `convert_children`: extracts all descendant text via `.text().collect()`, trims, emits as single `Text` node diff --git a/knowledge/foundations/architecture.md b/knowledge/foundations/architecture.md index 9781979..82c0c47 100644 --- a/knowledge/foundations/architecture.md +++ b/knowledge/foundations/architecture.md @@ -48,8 +48,10 @@ Input (bytes/URL/file) ## Configuration Via `MarkkitBuilder`: -- Input size limits -- Processing time limits +- Resource limits: one `Limits` struct (`limits.rs`, set whole via `.limits(..)` or per + field via builder setters): input/output size, timeout, nesting depth (hard max 1024), + node count, archive, spreadsheet and redirect caps. Violations are + `Error::ResourceLimit`; see [Threat Model](../security/threat-model.md) Resource Limits - CSS selectors for include/exclude - Link and image preservation - Office-document limits (archive entries/sizes, sheet rows/cols, grid cells, hidden sheets) diff --git a/knowledge/foundations/ir.md b/knowledge/foundations/ir.md index f861201..0c73b97 100644 --- a/knowledge/foundations/ir.md +++ b/knowledge/foundations/ir.md @@ -111,7 +111,7 @@ At render time, the `normalize_output` function: | No empty Text nodes after whitespace collapse | `collapse_whitespace` returns empty string for whitespace-only; parser skips empty | | Block lists hold no bare inline runs | `normalize_blocks` drops whitespace-only runs, wraps others in `Paragraph` | | Empty containers are elided | Parser returns `None` for headings, paragraphs, lists, etc. with no children | -| Depth bounded at 200 | `MAX_DEPTH` constant; at limit, subtree flattened to text | +| Depth bounded (`limits.max_nesting_depth`, default 256, hard max 1024) | HTML parser flattens the subtree to text at the limit | | No executable content in IR | Script, style, iframe, etc. stripped by parser | | `List.start` is `None` when `ordered=false` | Parser only reads `start` attr for `
      ` | diff --git a/knowledge/log.md b/knowledge/log.md index 1b2ff07..817b33f 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,6 +2,7 @@ ## 2026-10-26 +* **Resource limits**: all limits moved into one `Limits` struct (bashkit `ExecutionLimits` style). New: `timeout` now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB, stops table amplification); `max_nodes` (5M); `max_nesting_depth` configurable (256, was hardcoded 200 HTML / 256 XML) with a 1024 hard ceiling. Invariants S13-S15, `limits` tests. See [Threat Model](security/threat-model.md) T2. * **URL fetch hardening**: T3 SSRF is mitigated. `extract_url` blocks non-public addresses by default (URL, resolver and redirect layers), enforces scheme, allowlist, redirect, body and time limits; new invariant S12 and `url_security` tests. Proxy-resolved names are the documented residual. See [Threat Model](security/threat-model.md) and [Security Model](security/security-model.md). * **Dependency trim**: unused dependencies removed (lockfile shrinks by ~460 lines; the `instant`/`paste` unmaintained advisories and their `deny.toml` ignores are gone with the stub bench deps). quick-xml moves to 0.42, one version shared with calamine; markkit's XML walkers now read `&str` names and values. See [Threat Model](security/threat-model.md) T6d and [Maintenance](operations/maintenance.md). * **Inter-block whitespace**: indented HTML lost blank lines between blocks and leaked a leading space into every following line, because whitespace-only text between blocks stayed in the block list as `Text(" ")`. The parser now normalizes block-context lists; golden fixtures, which had recorded the broken output, are regenerated. See [HTML Extraction](formats/html-extraction.md) and [IR Specification](foundations/ir.md). diff --git a/knowledge/operations/testing.md b/knowledge/operations/testing.md index 49bdfba..0069cd2 100644 --- a/knowledge/operations/testing.md +++ b/knowledge/operations/testing.md @@ -21,6 +21,7 @@ tags: | `crates/markkit/tests/docx_integration.rs`, `spreadsheet_integration.rs` | Office mapping | | `crates/markkit/tests/document_detection.rs` | Format detection order and explicit unsupported errors | | `crates/markkit/tests/document_security.rs` | Zip bombs, grid amplification, XML entities, malformed containers | +| `crates/markkit/tests/limits.rs` | Every `Limits` field enforced for HTML, DOCX, spreadsheets; small-stack depth ceiling | | `crates/markkit/tests/common/` | In-memory DOCX/XLSX/XLS/ODS fixture builders | | `crates/markkit-cli/tests/cli_integration.rs` | CLI flags and output | | `crates/markkit/benches/extraction.rs` | Criterion throughput benchmarks | diff --git a/knowledge/security/security-model.md b/knowledge/security/security-model.md index a5086a6..70ffd08 100644 --- a/knowledge/security/security-model.md +++ b/knowledge/security/security-model.md @@ -25,7 +25,7 @@ Markkit processes untrusted input from the internet. Key threats: ### TM-SSRF: Server-side request forgery (URL fetching) - **Mitigation**: Non-public addresses refused by default, after DNS resolution and on every redirect (opt in with `allow_private_networks`) -- **Mitigation**: http/https only, optional host allowlist, redirect limit, body bounded by `max_input_size`, fetch bounded by `max_processing_secs` +- **Mitigation**: http/https only, optional host allowlist, redirect limit, body bounded by `max_input_size`, fetch bounded by `timeout` - **Residual**: an environment proxy resolves names itself; see [Threat Model](threat-model.md) T3 ### TM-DOC: Malicious office documents diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 49d78f1..913e529 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -59,19 +59,34 @@ tags: **Sub-threats**: +All limits live in one `Limits` struct (`crates/markkit/src/limits.rs`, +`MarkkitBuilder::limits`); violations return `Error::ResourceLimit` naming the limit. + #### T2a: Oversized Input - Input size checked before parsing against `max_input_size` - Default: 50 MB - Returns `Error::ResourceLimit` on violation #### T2b: Deep Nesting (Stack Overflow) -- `MAX_DEPTH=200` limits recursion in HTML parser -- At depth limit, subtree flattened to plain text (no further recursion) -- Tested with 1000 levels of nesting - -#### T2c: Many Elements (CPU Exhaustion) -- `max_processing_secs` configurable (default: 60s) -- Tested with 1000 paragraph elements +- `max_nesting_depth` (default 256) bounds recursion: HTML flattens deeper subtrees to + plain text; DOCX XML parts deeper than it are rejected +- Clamped to `HARD_MAX_NESTING_DEPTH` (1024) whatever the embedder sets: 1024 levels + of the worst recursive constructs fit a 2 MiB stack in release (debug frames are ~6x) +- Tested with 4096 levels under a small thread stack + +#### T2c: Many Elements / Slow Input (CPU Exhaustion) +- `timeout` (default 60 s) is one wall-clock budget per call covering URL fetch, + parsing and rendering. Checked cooperatively in the HTML walk, XML tree build, DOCX + conversion, spreadsheet cell loops and renderer. html5ever tokenizing and calamine + sheet decoding are not interruptible; they are linear in the (capped) input and the + deadline is checked right after them +- `max_nodes` (default 5 000 000) caps HTML DOM nodes and DOCX XML nodes (summed over parts) + +#### T2e: Output Amplification +- GFM tables pad every row to the widest row, so output can be rows x columns of input +- `max_output_size` (default 128 MiB) checked per block, list item and table row, + including nested renderers (blockquotes, list items), so no buffer grows past the + limit by more than one row #### T2d: Billion Laughs / Entity Expansion - html5ever handles entity expansion internally @@ -85,6 +100,9 @@ tags: - `html_security::resource_limit_bytes` - `html_integration::resource_limit_enforced` - `html_proptest::resource_limit_enforced` +- `limits::*` (every `Limits` field: output cap incl. table amplification, timeout, + node cap, configurable depth, hard depth ceiling on a small stack, for HTML, DOCX and + spreadsheets) ### T3: Server-Side Request Forgery (SSRF) @@ -115,7 +133,7 @@ and `http_client` is a **default** feature. - Redirects capped by `max_redirects` (default 5) - Response body bounded by `max_input_size`: declared `Content-Length` rejected before reading, undeclared bodies stopped while streaming (`ResourceLimit`) -- Whole fetch (connect, headers, body) bounded by `max_processing_secs` +- Whole fetch (connect, headers, body) bounded by `timeout`; parse and render share the remaining budget - Opt-out: embedders that fetch content themselves build with `default-features = false` and never link reqwest (CI asserts this for `docx,spreadsheet`) @@ -145,6 +163,7 @@ the proxy, or use `allow_host`. **Mitigations**: - Input size limit (pre-parse) - Depth limit prevents unbounded tree growth +- `max_nodes` caps parsed tree size; `max_output_size` caps the Markdown buffer - `String::with_capacity(4096)` for renderer output buffer (bounded initial allocation) **Test Coverage**: @@ -236,17 +255,17 @@ process, or smuggling content past a human reviewer. | Resource | Default | Configurable | Enforcement Point | |----------|---------|--------------|-------------------| | Input size | 50 MB | `max_input_size(usize)` | Before parsing | -| Processing time | 60 s | `max_processing_secs(u64)` | Whole `extract_url` fetch (parsing itself TBD) | +| Output size | 128 MiB | `max_output_size(usize)` | Renderer, per block / list item / table row | +| Processing time | 60 s | `timeout(Duration)`, `max_processing_secs(u64)` | One budget per call: fetch, parse, render (cooperative) | +| Parsed nodes | 5 000 000 | `max_nodes(usize)` | After html5ever (DOM count); while building DOCX XML trees | +| Nesting depth | 256 (hard max 1024) | `max_nesting_depth(usize)` | HTML walk (flatten); DOCX XML build (reject) | | Redirects (`extract_url`) | 5 | `max_redirects(usize)` | Every redirect hop | | Fetched body (`extract_url`) | = input size | `max_input_size(usize)` | `Content-Length` and while streaming | -| Nesting depth | 200 | Hardcoded `MAX_DEPTH` | During HTML tree walk | -| XML nesting depth (DOCX) | 256 | Hardcoded `MAX_XML_DEPTH` | While building the XML tree | | ZIP entries | 10 000 | `max_archive_entries(usize)` | On archive open | | ZIP entry size (decompressed) | 64 MiB | `max_archive_entry_size(u64)` | While decompressing | | ZIP total size (decompressed) | 256 MiB | `max_archive_total_size(u64)` | While decompressing | | Spreadsheet grid (XLS/ODS) | 4 000 000 cells | `max_spreadsheet_cells(u64)` | Pre-scan before calamine | | Rows / columns per sheet | 10 000 / 256 | `max_sheet_rows`, `max_sheet_cols` | While collecting cells (truncate + warn) | -| Output size | Unbounded | N/A | Proportional to input size | ## Malicious Input Categories @@ -257,7 +276,7 @@ process, or smuggling content past a human reviewer. | JavaScript URLs | `` | URL preserved as-is in markdown (no execution context) | | Data URI payloads | `` | Rendered as markdown image (no execution) | | Style expressions | `style="background:url(javascript:...)"` | Attribute not read | -| Deep nesting | 1000+ nested `
      ` | Flattened to text at depth 200 | +| Deep nesting | 1000+ nested `
      ` | Flattened to text at `max_nesting_depth` (256) | | Huge text nodes | 100KB+ single text node | Passed through (bounded by input limit) | | Null bytes | `\0` in content | Replaced with U+FFFD by html5ever | | Unclosed tags | `

      text

      more` | html5ever auto-closes per spec | @@ -285,6 +304,9 @@ process, or smuggling content past a human reviewer. | S10 | Spreadsheet parsing never allocates a grid beyond `max_spreadsheet_cells` | `document_security::xls_*_rejected`, `ods_repeated_rows_and_columns_rejected` | | S11 | Unsupported formats fail explicitly (no empty success) | `document_detection::pdf_is_explicitly_unsupported_not_empty` | | S12 | Default `extract_url` never connects to a non-public address | `url_security::*_blocked*`, `fetch::tests::non_public_addresses_forbidden` | +| S13 | Returned Markdown never exceeds `max_output_size` | `limits::*output_cap*` | +| S14 | No configuration of `max_nesting_depth` can overflow the stack | `limits::*hard_ceiling_survives_small_stack` | +| S15 | Extraction stops with `ResourceLimit` once `timeout` passes (checked at least after each non-interruptible third-party step) | `limits::*zero_timeout_rejects`, `limits::tests::*` | ## See also From cac90317dff668cfe3b879c77e34a96770f3437f Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Mon, 26 Oct 2026 23:49:26 +0000 Subject: [PATCH 2/3] fix(limits): drop private intra-doc link to Budget --- crates/markkit/src/limits.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/markkit/src/limits.rs b/crates/markkit/src/limits.rs index 4725e96..4ad8ac9 100644 --- a/crates/markkit/src/limits.rs +++ b/crates/markkit/src/limits.rs @@ -12,7 +12,7 @@ //! truncated output. Exceptions, kept for compatibility: per-sheet rows and //! columns truncate with a warning, and HTML nesting past //! `max_nesting_depth` is flattened to plain text. -//! - Time is checked cooperatively ([`Budget`]) in parser and renderer loops. +//! - Time is checked cooperatively (`Budget`) in parser and renderer loops. //! Third-party parse steps (html5ever tokenizing, calamine cell decoding) are //! not interruptible; they are bounded by input size and checked right after. //! - `max_nesting_depth` is clamped to [`HARD_MAX_NESTING_DEPTH`] because the From e0f8b40e0520ab6a96f52607471890e29264e876 Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Mon, 26 Oct 2026 23:52:29 +0000 Subject: [PATCH 3/3] fix(limits): lower nesting hard ceiling to 512 for release stack headroom --- CHANGELOG.md | 2 +- README.md | 2 +- crates/markkit/src/limits.rs | 9 +++++---- knowledge/formats/html-extraction.md | 2 +- knowledge/foundations/architecture.md | 2 +- knowledge/foundations/ir.md | 2 +- knowledge/log.md | 2 +- knowledge/security/threat-model.md | 7 ++++--- 8 files changed, 15 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ade409..31d1c96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/). per-field setters). New: `timeout` (default 60 s) now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB) stops output amplification (e.g. one wide table row padding thousands of rows); `max_nodes` (5 000 000) caps parsed HTML DOM / DOCX XML nodes; - `max_nesting_depth` (256) is configurable, with a hard ceiling of 1024 that keeps deep input off + `max_nesting_depth` (256) is configurable, with a hard ceiling of 512 that keeps deep input off the host stack. Violations return `Error::ResourceLimit` naming the limit. - `extract_url` (CLI, Python and Node `extract_url` too) now refuses non-public destinations by default: diff --git a/README.md b/README.md index 9faf2e0..798a0ea 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,7 @@ generous; violations return `Error::ResourceLimit` naming the limit. | `max_input_size` | 50 MiB | Input bytes, fetched URL body | | `max_output_size` | 128 MiB | Markdown output | | `timeout` | 60 s | Fetch + parse + render, per call | -| `max_nesting_depth` | 256 (hard max 1024) | HTML flattens deeper content; deeper DOCX XML is rejected | +| `max_nesting_depth` | 256 (hard max 512) | HTML flattens deeper content; deeper DOCX XML is rejected | | `max_nodes` | 5 000 000 | HTML DOM nodes, DOCX XML nodes | | `max_archive_entries` | 10 000 | ZIP entries (DOCX/XLSX/ODS) | | `max_archive_entry_size` | 64 MiB | Decompressed bytes per ZIP entry | diff --git a/crates/markkit/src/limits.rs b/crates/markkit/src/limits.rs index 4ad8ac9..49d1c3a 100644 --- a/crates/markkit/src/limits.rs +++ b/crates/markkit/src/limits.rs @@ -26,11 +26,12 @@ use std::time::{Duration, Instant}; use crate::error::Error; // THREAT[T2b]: recursion depth must stay safe on small host stacks whatever -// the embedder configures. Measured worst case (nested blockquote / inline -// tags): 1024 levels fit a 2 MiB thread in release; debug frames are ~6x -// larger (default 256 fits 2 MiB in debug). Test: limits::*hard_ceiling*. +// the embedder configures. Measured worst case (nested blockquotes, renderer +// recursion): 768 levels fit a 2 MiB thread in release, 1024 do not; 512 keeps +// ~2x headroom. Debug frames are ~6x larger (default 256 fits 2 MiB in debug). +// Test: limits::*hard_ceiling*. /// Upper bound applied to [`Limits::max_nesting_depth`]. -pub const HARD_MAX_NESTING_DEPTH: usize = 1024; +pub const HARD_MAX_NESTING_DEPTH: usize = 512; /// Resource limits applied to every extraction. #[derive(Debug, Clone, PartialEq, Eq)] diff --git a/knowledge/formats/html-extraction.md b/knowledge/formats/html-extraction.md index adf19ef..8ad911b 100644 --- a/knowledge/formats/html-extraction.md +++ b/knowledge/formats/html-extraction.md @@ -129,7 +129,7 @@ Trimming at render time: ## Depth Limiting -Limit: `limits.max_nesting_depth` (default 256, clamped to `HARD_MAX_NESTING_DEPTH` = 1024) +Limit: `limits.max_nesting_depth` (default 256, clamped to `HARD_MAX_NESTING_DEPTH` = 512) Behavior at depth limit: - `convert_children`: extracts all descendant text via `.text().collect()`, trims, emits as single `Text` node diff --git a/knowledge/foundations/architecture.md b/knowledge/foundations/architecture.md index 82c0c47..57d51aa 100644 --- a/knowledge/foundations/architecture.md +++ b/knowledge/foundations/architecture.md @@ -49,7 +49,7 @@ Input (bytes/URL/file) Via `MarkkitBuilder`: - Resource limits: one `Limits` struct (`limits.rs`, set whole via `.limits(..)` or per - field via builder setters): input/output size, timeout, nesting depth (hard max 1024), + field via builder setters): input/output size, timeout, nesting depth (hard max 512), node count, archive, spreadsheet and redirect caps. Violations are `Error::ResourceLimit`; see [Threat Model](../security/threat-model.md) Resource Limits - CSS selectors for include/exclude diff --git a/knowledge/foundations/ir.md b/knowledge/foundations/ir.md index 3c0d937..253ea35 100644 --- a/knowledge/foundations/ir.md +++ b/knowledge/foundations/ir.md @@ -138,7 +138,7 @@ At render time, the `normalize_output` function: | No empty Text nodes after whitespace collapse | `collapse_whitespace` returns empty string for whitespace-only; parser skips empty | | Block lists hold no bare inline runs | `normalize_blocks` drops whitespace-only runs, wraps others in `Paragraph` | | Empty containers are elided | Parser returns `None` for headings, paragraphs, lists, etc. with no children | -| Depth bounded (`limits.max_nesting_depth`, default 256, hard max 1024) | HTML parser flattens the subtree to text at the limit | +| Depth bounded (`limits.max_nesting_depth`, default 256, hard max 512) | HTML parser flattens the subtree to text at the limit | | No executable content in IR | Script, style, iframe, etc. stripped by parser | | `List.start` is `None` when `ordered=false` | Parser only reads `start` attr for `

        ` | | Table spans ≥ 1 | `TableCell::with_span` clamps 0 to 1; HTML parser clamps to 1000 / 65534 | diff --git a/knowledge/log.md b/knowledge/log.md index e8d5716..af349b0 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,7 +2,7 @@ ## 2026-10-26 -* **Resource limits**: all limits moved into one `Limits` struct (bashkit `ExecutionLimits` style). New: `timeout` now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB, stops table amplification); `max_nodes` (5M); `max_nesting_depth` configurable (256, was hardcoded 200 HTML / 256 XML) with a 1024 hard ceiling. Invariants S14-S16, `limits` tests. See [Threat Model](security/threat-model.md) T2. +* **Resource limits**: all limits moved into one `Limits` struct (bashkit `ExecutionLimits` style). New: `timeout` now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB, stops table amplification); `max_nodes` (5M); `max_nesting_depth` configurable (256, was hardcoded 200 HTML / 256 XML) with a 512 hard ceiling. Invariants S14-S16, `limits` tests. See [Threat Model](security/threat-model.md) T2. * **IR v2**: footnotes, definition lists, task lists, figures, table alignment and spans; `Document.ir_version` (`IR_VERSION = 2`), `#[non_exhaustive]` IR types, `Node::is_block`/`is_inline`. HTML maps GitHub/Pandoc footnotes, `
        `, checkbox items, captioned figures, `colspan`/`rowspan`/alignment and row-header ``. New threat T2e and invariant S13 (span grid budget, footnote label sanitizing). See [IR Specification](foundations/ir.md), [HTML Extraction](formats/html-extraction.md), [Threat Model](security/threat-model.md). * **URL fetch hardening**: T3 SSRF is mitigated. `extract_url` blocks non-public addresses by default (URL, resolver and redirect layers), enforces scheme, allowlist, redirect, body and time limits; new invariant S12 and `url_security` tests. Proxy-resolved names are the documented residual. See [Threat Model](security/threat-model.md) and [Security Model](security/security-model.md). diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index f48730b..ab2480f 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -70,8 +70,9 @@ All limits live in one `Limits` struct (`crates/markkit/src/limits.rs`, #### T2b: Deep Nesting (Stack Overflow) - `max_nesting_depth` (default 256) bounds recursion: HTML flattens deeper subtrees to plain text; DOCX XML parts deeper than it are rejected -- Clamped to `HARD_MAX_NESTING_DEPTH` (1024) whatever the embedder sets: 1024 levels - of the worst recursive constructs fit a 2 MiB stack in release (debug frames are ~6x) +- Clamped to `HARD_MAX_NESTING_DEPTH` (512) whatever the embedder sets: 512 levels + of the worst recursive construct (nested blockquotes) fit a 2 MiB stack in release + with ~2x headroom (1024 did not after IR v2); debug frames are ~6x larger - Tested with 4096 levels under a small thread stack #### T2c: Many Elements / Slow Input (CPU Exhaustion) @@ -266,7 +267,7 @@ process, or smuggling content past a human reviewer. | Output size | 128 MiB | `max_output_size(usize)` | Renderer, per block / list item / table row | | Processing time | 60 s | `timeout(Duration)`, `max_processing_secs(u64)` | One budget per call: fetch, parse, render (cooperative) | | Parsed nodes | 5 000 000 | `max_nodes(usize)` | After html5ever (DOM count); while building DOCX XML trees | -| Nesting depth | 256 (hard max 1024) | `max_nesting_depth(usize)` | HTML walk (flatten); DOCX XML build (reject) | +| Nesting depth | 256 (hard max 512) | `max_nesting_depth(usize)` | HTML walk (flatten); DOCX XML build (reject) | | Redirects (`extract_url`) | 5 | `max_redirects(usize)` | Every redirect hop | | Fetched body (`extract_url`) | = input size | `max_input_size(usize)` | `Content-Length` and while streaming | | Table span grid | 1 000 000 slots | Hardcoded `MAX_TABLE_GRID_CELLS` | Renderer table layout (spans dropped past it) |