diff --git a/CHANGELOG.md b/CHANGELOG.md index 0398c03..8cf3d53 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,6 +51,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Security +- Resource limits on every aspect, in one `markkit::Limits` struct (`MarkkitBuilder::limits(..)` or + per-field setters). New: `timeout` (default 60 s) now bounds parsing and rendering, not just URL + fetch; `max_output_size` (128 MiB) stops output amplification (e.g. one wide table row padding + thousands of rows); `max_nodes` (5 000 000) caps parsed HTML DOM / DOCX XML nodes; + `max_nesting_depth` (256) is configurable, with a hard ceiling of 512 that keeps deep input off + the host stack. Violations return `Error::ResourceLimit` naming the limit. + - `extract_url` (CLI, Python and Node `extract_url` too) now refuses non-public destinations by default: loopback, private, link-local / cloud metadata and other reserved addresses, checked on the URL, after DNS resolution and on every redirect. Only `http`/`https` are fetched. Redirects are capped @@ -66,6 +73,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` + (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is + now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, + `max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth + 256 (was 200). - **Breaking (features):** `http_client` is no longer a default feature; the default build pulls in no HTTP/TLS stack. Library users calling `extract_url` add `features = ["http_client"]`. The CLI and the Python/Node packages still enable it, so their `extract_url` is unchanged. @@ -114,6 +126,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` + (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is + now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, + `max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth + 256 (was 200). + - Release automation: merging `chore(release): prepare vX.Y.Z` to main tags, creates the GitHub Release and publishes to crates.io (idempotent); manual dispatch from main also works @@ -160,6 +178,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits` + (`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is + now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`, + `max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth + 256 (was 200). + - `docx` feature now uses `zip` + `quick-xml` directly; `docx-rs` dependency removed - `calamine` upgraded to 0.36 (pulls quick-xml 0.41, clearing RUSTSEC-2026-0194/0195); `zip` upgraded to 8 (deflate only), shared with calamine - PDF extraction returns `Error::UnsupportedFormat` instead of an empty string diff --git a/README.md b/README.md index 67c264e..768ec74 100644 --- a/README.md +++ b/README.md @@ -195,10 +195,36 @@ markkit = { version = "0.1", features = ["http_client"] } # markkit = { version = "0.1", default-features = false, features = ["docx", "spreadsheet"] } ``` -Office-document limits are configurable on the builder: `max_archive_entries`, -`max_archive_entry_size`, `max_archive_total_size`, `max_spreadsheet_cells`, -`max_sheet_rows`, `max_sheet_cols`, `include_hidden_sheets`. URL fetching: -`allow_private_networks`, `allow_host`, `max_redirects`. +### Resource limits + +Every resource an input can drive is bounded by `markkit::Limits`. Defaults are +generous; violations return `Error::ResourceLimit` naming the limit. + +| Limit | Default | Bounds | +|-------|---------|--------| +| `max_input_size` | 50 MiB | Input bytes, fetched URL body | +| `max_output_size` | 128 MiB | Markdown output | +| `timeout` | 60 s | Fetch + parse + render, per call | +| `max_nesting_depth` | 256 (hard max 512) | HTML flattens deeper content; deeper DOCX XML is rejected | +| `max_nodes` | 5 000 000 | HTML DOM nodes, DOCX XML nodes | +| `max_archive_entries` | 10 000 | ZIP entries (DOCX/XLSX/ODS) | +| `max_archive_entry_size` | 64 MiB | Decompressed bytes per ZIP entry | +| `max_archive_total_size` | 256 MiB | Decompressed bytes per ZIP | +| `max_spreadsheet_cells` | 4 000 000 | XLS/ODS grid before parsing | +| `max_sheet_rows` / `max_sheet_cols` | 10 000 / 256 | Rendered per sheet (truncated with a warning) | +| `max_redirects` | 5 | `extract_url` redirects | + +```rust +use markkit::{Limits, Markkit}; +use std::time::Duration; + +let markkit = Markkit::builder() + .limits(Limits::default().timeout(Duration::from_secs(10)).max_output_size(8 << 20)) + .build(); +``` + +Each limit also has a builder setter (`.max_nodes(..)`, `.timeout(..)`, ...). +Other options: `include_hidden_sheets`, `allow_private_networks`, `allow_host`. ## Performance @@ -244,7 +270,7 @@ XML entities are never expanded; macros and formulas are never executed; hidden sheets are skipped by default. `extract_url` accepts only http/https, refuses loopback, private, link-local and cloud-metadata addresses (also after DNS resolution and on every redirect), caps redirects, and bounds the body by -`max_input_size` and the fetch by `max_processing_secs`. Opt in to internal +`max_input_size` and the fetch by `timeout`. Opt in to internal hosts with `allow_private_networks(true)`; restrict hosts with `allow_host(..)`. See the [threat model](knowledge/security/threat-model.md) and the diff --git a/crates/markkit/src/archive.rs b/crates/markkit/src/archive.rs index 5e7a85d..3166a02 100644 --- a/crates/markkit/src/archive.rs +++ b/crates/markkit/src/archive.rs @@ -41,11 +41,11 @@ impl<'a> ZipGuard<'a> { /// Open the archive and check the entry count. pub fn open(bytes: &'a [u8], config: &MarkkitBuilder) -> Result { let archive = ZipArchive::new(Cursor::new(bytes))?; - if archive.len() > config.max_archive_entries { + if archive.len() > config.limits.max_archive_entries { return Err(ArchiveError::Limit(format!( "archive has {} entries, limit is {}", archive.len(), - config.max_archive_entries + config.limits.max_archive_entries ))); } let names = archive @@ -55,8 +55,8 @@ impl<'a> ZipGuard<'a> { Ok(Self { archive, names, - max_entry: config.max_archive_entry_size, - max_total: config.max_archive_total_size, + max_entry: config.limits.max_archive_entry_size, + max_total: config.limits.max_archive_total_size, total: 0, }) } diff --git a/crates/markkit/src/config.rs b/crates/markkit/src/config.rs index cf05083..ab0e7f0 100644 --- a/crates/markkit/src/config.rs +++ b/crates/markkit/src/config.rs @@ -1,12 +1,14 @@ +use std::time::Duration; + use crate::extract::Markkit; +use crate::limits::Limits; /// Builder for configuring a Markkit instance. #[derive(Debug, Clone)] pub struct MarkkitBuilder { - /// Maximum input size in bytes (default: 50MB). - pub max_input_size: usize, - /// Maximum processing time in seconds (default: 60). - pub max_processing_secs: u64, + /// Resource limits (input/output size, time, nesting, nodes, archives, + /// spreadsheets, redirects). See [`Limits`]. + pub limits: Limits, /// Whether to preserve link URLs in output. pub preserve_links: bool, /// Whether to preserve image references in output. @@ -17,19 +19,6 @@ pub struct MarkkitBuilder { pub include_selectors: Vec, /// CSS selectors for content to exclude. pub exclude_selectors: Vec, - /// Maximum number of entries in a ZIP container (DOCX/XLSX/ODS). Default: 10 000. - pub max_archive_entries: usize, - /// Maximum decompressed size of a single ZIP entry in bytes. Default: 64 MiB. - pub max_archive_entry_size: u64, - /// Maximum total decompressed size of a ZIP container in bytes. Default: 256 MiB. - pub max_archive_total_size: u64, - /// Maximum data rows rendered per sheet (header excluded). Default: 10 000. - pub max_sheet_rows: usize, - /// Maximum columns rendered per sheet. Default: 256. - pub max_sheet_cols: usize, - /// Maximum cell grid a workbook may expand to while parsing (XLS/ODS - /// pre-scan, summed over sheets). Default: 4 000 000. - pub max_spreadsheet_cells: u64, /// Render hidden / very hidden sheets. Default: false (skipped with a warning). pub include_hidden_sheets: bool, /// Let `extract_url` reach loopback, private, link-local and other @@ -38,8 +27,6 @@ pub struct MarkkitBuilder { /// Hosts `extract_url` may contact, matched exactly and case-insensitively /// on every hop (empty = any public host). pub allowed_hosts: Vec, - /// Maximum redirects `extract_url` follows. Default: 5. - pub max_redirects: usize, /// Absolute URL that relative HTML links, images and metadata URLs resolve /// against (a document `` applies on top). `extract_url` uses the /// final response URL instead. Default: none (relative URLs kept as is). @@ -59,8 +46,7 @@ pub struct MarkkitBuilder { impl Default for MarkkitBuilder { fn default() -> Self { Self { - max_input_size: 50 * 1024 * 1024, - max_processing_secs: 60, + limits: Limits::default(), preserve_links: true, preserve_images: true, extract_code_blocks: true, @@ -73,16 +59,9 @@ impl Default for MarkkitBuilder { "style".into(), "noscript".into(), ], - max_archive_entries: 10_000, - max_archive_entry_size: 64 * 1024 * 1024, - max_archive_total_size: 256 * 1024 * 1024, - max_sheet_rows: 10_000, - max_sheet_cols: 256, - max_spreadsheet_cells: 4_000_000, include_hidden_sheets: false, allow_private_networks: false, allowed_hosts: Vec::new(), - max_redirects: 5, base_url: None, track_sources: false, transforms: Default::default(), @@ -93,12 +72,41 @@ impl Default for MarkkitBuilder { impl MarkkitBuilder { pub fn max_input_size(mut self, size: usize) -> Self { - self.max_input_size = size; + self.limits.max_input_size = size; + self + } + + /// Replace all limits at once. + pub fn limits(mut self, limits: Limits) -> Self { + self.limits = limits; self } + /// Wall-clock budget per call (fetch + parse + render). + pub fn timeout(mut self, timeout: Duration) -> Self { + self.limits.timeout = timeout; + self + } + + /// [`MarkkitBuilder::timeout`] in whole seconds. pub fn max_processing_secs(mut self, secs: u64) -> Self { - self.max_processing_secs = secs; + self.limits.timeout = Duration::from_secs(secs); + self + } + + pub fn max_output_size(mut self, bytes: usize) -> Self { + self.limits.max_output_size = bytes; + self + } + + /// Clamped to [`crate::limits::HARD_MAX_NESTING_DEPTH`]. + pub fn max_nesting_depth(mut self, depth: usize) -> Self { + self.limits.max_nesting_depth = depth; + self + } + + pub fn max_nodes(mut self, nodes: usize) -> Self { + self.limits.max_nodes = nodes; self } @@ -135,32 +143,32 @@ impl MarkkitBuilder { } pub fn max_archive_entries(mut self, n: usize) -> Self { - self.max_archive_entries = n; + self.limits.max_archive_entries = n; self } pub fn max_archive_entry_size(mut self, bytes: u64) -> Self { - self.max_archive_entry_size = bytes; + self.limits.max_archive_entry_size = bytes; self } pub fn max_archive_total_size(mut self, bytes: u64) -> Self { - self.max_archive_total_size = bytes; + self.limits.max_archive_total_size = bytes; self } pub fn max_sheet_rows(mut self, rows: usize) -> Self { - self.max_sheet_rows = rows; + self.limits.max_sheet_rows = rows; self } pub fn max_sheet_cols(mut self, cols: usize) -> Self { - self.max_sheet_cols = cols; + self.limits.max_sheet_cols = cols; self } pub fn max_spreadsheet_cells(mut self, cells: u64) -> Self { - self.max_spreadsheet_cells = cells; + self.limits.max_spreadsheet_cells = cells; self } @@ -183,7 +191,7 @@ impl MarkkitBuilder { } pub fn max_redirects(mut self, n: usize) -> Self { - self.max_redirects = n; + self.limits.max_redirects = n; self } diff --git a/crates/markkit/src/docx.rs b/crates/markkit/src/docx.rs index 4f030d6..62f24dc 100644 --- a/crates/markkit/src/docx.rs +++ b/crates/markkit/src/docx.rs @@ -24,6 +24,7 @@ use crate::archive::{ArchiveError, ZipGuard}; use crate::config::MarkkitBuilder; use crate::error::Error; use crate::ir::{Document, ListItem, Node, Source, TableCell}; +use crate::limits::Budget; use crate::render::escape_table_cell; use crate::xml::{self, Element, XmlError}; @@ -51,10 +52,21 @@ fn xml_err(part: &str, e: XmlError) -> Error { } } -fn read_xml(zip: &mut ZipGuard<'_>, path: &str) -> Result, Error> { - match zip.read(path).map_err(archive_err)? { - Some(bytes) => xml::parse(&bytes).map(Some).map_err(|e| xml_err(path, e)), - None => Ok(None), +/// Bounded XML reader: archive limits, nesting depth, node count and deadline. +struct Parts<'z, 'b> { + zip: ZipGuard<'z>, + max_depth: usize, + budget: &'b Budget, +} + +impl Parts<'_, '_> { + fn read_xml(&mut self, path: &str) -> Result, Error> { + match self.zip.read(path).map_err(archive_err)? { + Some(bytes) => xml::parse(&bytes, self.max_depth, self.budget) + .map(Some) + .map_err(|e| xml_err(path, e)), + None => Ok(None), + } } } @@ -365,6 +377,7 @@ struct Ctx<'a> { rels: &'a HashMap, styles: &'a Styles, numbering: &'a Numbering, + budget: &'a Budget, } impl Ctx<'_> { @@ -682,6 +695,9 @@ fn build_list(entries: &[ListEntry], pos: &mut usize, level: u32) -> Node { impl Ctx<'_> { fn blocks(&self, container: &Element, body: &mut Body) { for child in container.elements() { + if self.budget.expired() { + return; + } match child.name.as_str() { "p" => { let index = body.next_index; @@ -899,10 +915,18 @@ fn table_cell(nodes: Vec) -> Vec { // ─── Entry point ──────────────────────────────────────────────────── -pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result { - let mut zip = ZipGuard::open(bytes, config).map_err(archive_err)?; +pub(crate) fn parse( + bytes: &[u8], + config: &MarkkitBuilder, + budget: &Budget, +) -> Result { + let mut zip = Parts { + zip: ZipGuard::open(bytes, config).map_err(archive_err)?, + max_depth: config.limits.nesting_depth(), + budget, + }; - let root_rels = parse_rels(read_xml(&mut zip, "_rels/.rels")?); + let root_rels = parse_rels(zip.read_xml("_rels/.rels")?); let main_path = rel_target_by_type(&root_rels, "/officeDocument") .map(|t| resolve_target("", t)) .unwrap_or_else(|| "word/document.xml".to_string()); @@ -910,10 +934,11 @@ pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result Result Result Extraction { - let mut renderer = crate::render::MarkdownRenderer::new(); + let markdown = crate::render::MarkdownRenderer::new().render(&self.document); + self.into_extraction(markdown) + } + + /// [`ParsedDocument::render`] with `max_output_size` and the deadline enforced. + #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + fn render_bounded(self, max_output: usize, budget: &Budget) -> crate::Result { + let markdown = crate::render::MarkdownRenderer::new().render_bounded( + &self.document, + max_output, + budget, + )?; + Ok(self.into_extraction(markdown)) + } + + fn into_extraction(self, markdown: String) -> Extraction { Extraction { - markdown: renderer.render(&self.document), + markdown, title: self.title, format: self.format, warnings: self.warnings, @@ -160,19 +176,17 @@ impl Markkit { /// Synchronous HTML extraction (no async needed for parsing). pub fn extract_html_sync(&self, html: &str) -> crate::Result { - if html.len() > self.config.max_input_size { - return Err(Error::ResourceLimit(format!( - "Input size {} exceeds limit {}", - html.len(), - self.config.max_input_size - ))); - } + self.config.limits.check_input(html.len())?; #[cfg(feature = "html")] { - let doc = self.parse_html(html)?; - let mut renderer = crate::render::MarkdownRenderer::new(); - Ok(renderer.render(&doc)) + let budget = Budget::new(&self.config.limits); + let doc = self.parse_html_with(html, &budget)?; + crate::render::MarkdownRenderer::new().render_bounded( + &doc, + self.config.limits.max_output_size, + &budget, + ) } #[cfg(not(feature = "html"))] Err(Error::UnsupportedFormat( @@ -183,15 +197,13 @@ impl Markkit { /// Parse HTML into the IR without rendering. #[cfg(feature = "html")] pub fn parse_html(&self, html: &str) -> crate::Result { - if html.len() > self.config.max_input_size { - return Err(Error::ResourceLimit(format!( - "Input size {} exceeds limit {}", - html.len(), - self.config.max_input_size - ))); - } - let parser = crate::html::HtmlParser::new(&self.config); - let mut doc = parser.parse(html)?; + self.config.limits.check_input(html.len())?; + self.parse_html_with(html, &Budget::new(&self.config.limits)) + } + + #[cfg(feature = "html")] + fn parse_html_with(&self, html: &str, budget: &Budget) -> crate::Result { + let mut doc = crate::html::HtmlParser::with_budget(&self.config, budget).parse(html)?; self.config.transforms.apply(&mut doc)?; Ok(doc) } @@ -229,7 +241,7 @@ impl Markkit { /// Synchronous [`Markkit::extract_document`]. /// - /// Errors: [`Error::ResourceLimit`] when input/archive/grid limits are + /// Errors: [`Error::ResourceLimit`] when any [`Limits`](crate::Limits) value is /// exceeded, [`Error::UnsupportedFormat`] for unknown formats or formats whose /// feature is disabled, and a format-specific variant for malformed input. pub fn extract_document_sync( @@ -237,8 +249,22 @@ impl Markkit { bytes: &[u8], hint: DocumentHint<'_>, ) -> crate::Result { - self.parse_document_sync(bytes, hint) - .map(ParsedDocument::render) + self.extract_document_with(bytes, hint, &Budget::new(&self.config.limits)) + } + + fn extract_document_with( + &self, + bytes: &[u8], + hint: DocumentHint<'_>, + budget: &Budget, + ) -> crate::Result { + let parsed = self.parse_document_with(bytes, hint, budget)?; + #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + { + parsed.render_bounded(self.config.limits.max_output_size, budget) + } + #[cfg(not(any(feature = "html", feature = "docx", feature = "spreadsheet")))] + Ok(parsed.render()) } /// Parse a document (HTML, DOCX, XLSX/XLS/ODS) into the IR without @@ -257,20 +283,23 @@ impl Markkit { bytes: &[u8], hint: DocumentHint<'_>, ) -> crate::Result { - if bytes.len() > self.config.max_input_size { - return Err(Error::ResourceLimit(format!( - "Input size {} exceeds limit {}", - bytes.len(), - self.config.max_input_size - ))); - } + self.parse_document_with(bytes, hint, &Budget::new(&self.config.limits)) + } + + fn parse_document_with( + &self, + bytes: &[u8], + hint: DocumentHint<'_>, + budget: &Budget, + ) -> crate::Result { + self.config.limits.check_input(bytes.len())?; match Format::detect(bytes, hint) { - Format::Html => self.parse_html_doc(bytes), + Format::Html => self.parse_html_doc(bytes, budget), Format::Pdf => self.parse_pdf(bytes), - Format::Docx => self.parse_docx(bytes), + Format::Docx => self.parse_docx(bytes, budget), Format::Xlsx | Format::Xlsb | Format::Xls | Format::Ods => { - self.parse_spreadsheet(bytes) + self.parse_spreadsheet(bytes, budget) } Format::Doc => Err(Error::UnsupportedFormat( "legacy Word 97-2003 (.doc) documents are not supported".into(), @@ -286,9 +315,9 @@ impl Markkit { } #[cfg(feature = "html")] - fn parse_html_doc(&self, bytes: &[u8]) -> crate::Result { + fn parse_html_doc(&self, bytes: &[u8], budget: &Budget) -> crate::Result { let text = String::from_utf8_lossy(bytes); - let parser = crate::html::HtmlParser::new(&self.config); + let parser = crate::html::HtmlParser::with_budget(&self.config, budget); let (mut doc, page) = parser.parse_page(&text)?; self.config.transforms.apply(&mut doc)?; let mut p = ParsedDocument::new(doc, Format::Html); @@ -298,16 +327,16 @@ impl Markkit { } #[cfg(not(feature = "html"))] - fn parse_html_doc(&self, _bytes: &[u8]) -> crate::Result { + fn parse_html_doc(&self, _bytes: &[u8], _budget: &Budget) -> crate::Result { Err(Error::UnsupportedFormat( "HTML support requires the 'html' feature".to_string(), )) } #[cfg(feature = "docx")] - fn parse_docx(&self, bytes: &[u8]) -> crate::Result { + fn parse_docx(&self, bytes: &[u8], budget: &Budget) -> crate::Result { let out = guarded("DOCX", Error::DocxExtract, || { - crate::docx::parse(bytes, &self.config) + crate::docx::parse(bytes, &self.config, budget) })?; let mut doc = out.doc; self.config.transforms.apply(&mut doc)?; @@ -319,16 +348,16 @@ impl Markkit { } #[cfg(not(feature = "docx"))] - fn parse_docx(&self, _bytes: &[u8]) -> crate::Result { + fn parse_docx(&self, _bytes: &[u8], _budget: &Budget) -> crate::Result { Err(Error::UnsupportedFormat( "DOCX support requires the 'docx' feature".to_string(), )) } #[cfg(feature = "spreadsheet")] - fn parse_spreadsheet(&self, bytes: &[u8]) -> crate::Result { + fn parse_spreadsheet(&self, bytes: &[u8], budget: &Budget) -> crate::Result { let out = guarded("spreadsheet", Error::SpreadsheetExtract, || { - crate::spreadsheet::parse(bytes, &self.config) + crate::spreadsheet::parse(bytes, &self.config, budget) })?; let mut doc = out.doc; self.config.transforms.apply(&mut doc)?; @@ -341,7 +370,7 @@ impl Markkit { } #[cfg(not(feature = "spreadsheet"))] - fn parse_spreadsheet(&self, _bytes: &[u8]) -> crate::Result { + fn parse_spreadsheet(&self, _bytes: &[u8], _budget: &Budget) -> crate::Result { Err(Error::UnsupportedFormat( "spreadsheet support requires the 'spreadsheet' feature".to_string(), )) @@ -354,8 +383,8 @@ impl Markkit { /// refused with [`Error::UrlBlocked`] on every hop (including after DNS /// resolution) unless [`MarkkitBuilder::allow_private_networks`] is set; /// [`MarkkitBuilder::allow_host`] restricts hosts; redirects capped by - /// `max_redirects`; body capped by `max_input_size`; whole fetch capped - /// by `max_processing_secs` ([`Error::ResourceLimit`]). + /// `max_redirects`; body capped by `max_input_size`; fetch, parse and render + /// together capped by `limits.timeout` ([`Error::ResourceLimit`]). /// /// Relative links and images resolve against the final URL after redirects. #[cfg(feature = "http_client")] @@ -369,15 +398,19 @@ impl Markkit { /// Relative HTML URLs resolve against the final URL after redirects. #[cfg(feature = "http_client")] pub async fn extract_url_document(&self, url: &str) -> crate::Result { + let start = std::time::Instant::now(); let fetched = crate::fetch::fetch(url, &self.config).await?; + // Fetch, parse and render share one `timeout` budget. + let budget = Budget::starting_at(&self.config.limits, start); let mut config = self.config.clone(); config.base_url = Some(fetched.url); - Markkit::new(config).extract_document_sync( + Markkit::new(config).extract_document_with( &fetched.bytes, DocumentHint { content_type: fetched.content_type.as_deref(), file_name: None, }, + &budget, ) } @@ -404,7 +437,7 @@ mod tests { #[tokio::test] async fn test_builder_defaults() { let m = Markkit::builder().build(); - assert_eq!(m.config().max_input_size, 50 * 1024 * 1024); + assert_eq!(m.config().limits.max_input_size, 50 * 1024 * 1024); assert!(m.config().preserve_links); } diff --git a/crates/markkit/src/fetch.rs b/crates/markkit/src/fetch.rs index b7b1276..57ad41a 100644 --- a/crates/markkit/src/fetch.rs +++ b/crates/markkit/src/fetch.rs @@ -12,7 +12,7 @@ //! layers 1-2 still apply, layer 3 cannot. Documented in the threat model. //! - Body bounded by `max_input_size` while streaming (declared //! `Content-Length` rejected up front); whole fetch bounded by -//! `max_processing_secs`. +//! `limits.timeout`. //! - Policy and resolver run inside reqwest and can only return opaque //! errors, so they record the block reason in a per-call slot that is //! turned into [`Error::UrlBlocked`] afterwards. @@ -49,7 +49,7 @@ pub(crate) async fn fetch(url: &str, cfg: &MarkkitBuilder) -> crate::Result crate::Result crate::Result crate::Result { - let max_redirects = cfg.max_redirects; + let max_redirects = cfg.limits.max_redirects; let policy_cfg = cfg.clone(); let policy_slot = blocked.clone(); let policy = reqwest::redirect::Policy::custom(move |attempt| { @@ -118,11 +118,11 @@ fn client(cfg: &MarkkitBuilder, blocked: &Blocked) -> crate::Result { + Owned(Budget), + Borrowed(&'a Budget), +} + /// Parses HTML into the canonical IR. +/// +/// Limits: nesting past `limits.max_nesting_depth` is flattened to text; DOM +/// node count and the deadline are enforced ([`Error::ResourceLimit`]). pub struct HtmlParser<'a> { config: &'a MarkkitBuilder, + budget: BudgetRef<'a>, + max_depth: usize, /// Footnote target id (without `#`) -> label, filled by references. footnote_labels: RefCell>, /// Base for relative URLs (caller base, then the document ``). @@ -54,9 +63,24 @@ pub struct HtmlParser<'a> { } impl<'a> HtmlParser<'a> { + /// Parser with a fresh per-call budget (deadline starts now). pub fn new(config: &'a MarkkitBuilder) -> Self { Self { config, + budget: BudgetRef::Owned(Budget::new(&config.limits)), + max_depth: config.limits.nesting_depth(), + footnote_labels: RefCell::new(HashMap::new()), + base: urls::parse_base(config.base_url.as_deref()), + sibling_index: RefCell::new(HashMap::new()), + drop_unlikely: Cell::new(false), + } + } + + pub(crate) fn with_budget(config: &'a MarkkitBuilder, budget: &'a Budget) -> Self { + Self { + config, + budget: BudgetRef::Borrowed(budget), + max_depth: config.limits.nesting_depth(), footnote_labels: RefCell::new(HashMap::new()), base: urls::parse_base(config.base_url.as_deref()), sibling_index: RefCell::new(HashMap::new()), @@ -64,6 +88,13 @@ impl<'a> HtmlParser<'a> { } } + fn budget(&self) -> &Budget { + match &self.budget { + BudgetRef::Owned(b) => b, + BudgetRef::Borrowed(b) => b, + } + } + /// Parse an HTML string into a [`Document`]. pub fn parse(&self, html: &str) -> Result { self.parse_dom(&Html::parse_document(html)) @@ -83,10 +114,13 @@ impl<'a> HtmlParser<'a> { self.for_document(dom).convert_dom(dom) } - /// Parser whose base includes the document's ``. - fn for_document(&self, dom: &Html) -> Self { - Self { + /// Parser whose base includes the document's ``; shares this + /// parser's budget. + fn for_document(&self, dom: &Html) -> HtmlParser<'_> { + HtmlParser { config: self.config, + budget: BudgetRef::Borrowed(self.budget()), + max_depth: self.max_depth, footnote_labels: RefCell::new(HashMap::new()), sibling_index: RefCell::new(HashMap::new()), base: urls::document_base(dom, self.base.clone()), @@ -95,6 +129,9 @@ impl<'a> HtmlParser<'a> { } fn convert_dom(&self, dom: &Html) -> Result { + // html5ever is not interruptible: check right after it finishes. + self.budget().check_time()?; + self.budget().add_nodes(dom.tree.nodes().count())?; let root = dom.root_element(); let exclude_selectors: Vec = self @@ -117,6 +154,8 @@ impl<'a> HtmlParser<'a> { } return Ok(Document::new(normalize_blocks(nodes))); } + // The walk stops early once the deadline passes; report it here. + self.budget().check_time()?; let body_sel = Selector::parse("body").unwrap(); let body = dom.select(&body_sel).next().unwrap_or(root); @@ -199,7 +238,10 @@ impl<'a> HtmlParser<'a> { out: &mut Vec, depth: usize, ) { - if depth >= MAX_DEPTH { + if self.budget().expired() { + return; + } + if depth >= self.max_depth { // At max depth, just extract text let text: String = parent.text().collect(); let trimmed = text.trim(); @@ -267,7 +309,10 @@ impl<'a> HtmlParser<'a> { exclude: &[Selector], depth: usize, ) -> Option { - if depth >= MAX_DEPTH { + if self.budget().expired() { + return None; + } + if depth >= self.max_depth { let text: String = el.text().collect(); let trimmed = text.trim(); if trimmed.is_empty() { @@ -603,7 +648,7 @@ impl<'a> HtmlParser<'a> { depth: usize, items: &mut Vec, ) { - if depth >= MAX_DEPTH { + if depth >= self.max_depth || self.budget().expired() { return; } for child in el.children().filter_map(scraper::ElementRef::wrap) { @@ -721,7 +766,10 @@ impl<'a> HtmlParser<'a> { depth: usize, ) -> Vec { let mut nodes = Vec::new(); - if depth >= MAX_DEPTH { + if self.budget().expired() { + return nodes; + } + if depth >= self.max_depth { let text: String = el.text().collect(); let trimmed = text.trim(); if !trimmed.is_empty() { diff --git a/crates/markkit/src/lib.rs b/crates/markkit/src/lib.rs index ba64824..3a1f2bc 100644 --- a/crates/markkit/src/lib.rs +++ b/crates/markkit/src/lib.rs @@ -57,6 +57,7 @@ pub mod format; #[cfg(feature = "html")] pub mod html; pub mod ir; +pub mod limits; pub mod render; #[cfg(feature = "spreadsheet")] mod spreadsheet; @@ -72,6 +73,7 @@ pub use error::Error; pub use extract::{Extraction, ExtractionStats, Markkit, Metadata, ParsedDocument}; pub use format::{DocumentHint, Format}; pub use ir::Document; +pub use limits::Limits; pub use render::MarkdownRenderer; /// Result type alias for markkit operations. diff --git a/crates/markkit/src/limits.rs b/crates/markkit/src/limits.rs new file mode 100644 index 0000000..9477ee9 --- /dev/null +++ b/crates/markkit/src/limits.rs @@ -0,0 +1,335 @@ +//! Resource limits for extraction. +//! +//! Every input to markkit is untrusted, so every resource an input can drive +//! (bytes in, bytes out, time, nesting, node count, decompression, grid size, +//! redirects) has a limit here. Defaults are generous: real documents should +//! never hit them; hostile ones always do. +//! +//! Decisions: +//! - One [`Limits`] struct (bashkit `ExecutionLimits` style) instead of loose +//! builder fields, so the CLI and the Python/Node bindings map one shape. +//! - Violations are hard errors ([`Error::ResourceLimit`]), never silently +//! truncated output. Exceptions, kept for compatibility: per-sheet rows and +//! columns truncate with a warning, and HTML nesting past +//! `max_nesting_depth` is flattened to plain text. +//! - Time is checked cooperatively (`Budget`) in parser and renderer loops. +//! Third-party parse steps (html5ever tokenizing, calamine cell decoding) are +//! not interruptible; they are bounded by input size and checked right after. +//! - `max_nesting_depth` is clamped to [`HARD_MAX_NESTING_DEPTH`] because the +//! HTML walker, DOCX conversion and IR drop recurse on the host stack. +//! +//! See `knowledge/security/threat-model.md` (T2, T6) for the threat mapping. + +use std::cell::Cell; +use std::time::{Duration, Instant}; + +use crate::error::Error; + +// THREAT[T2b]: recursion depth must stay safe on small host stacks whatever +// the embedder configures. Measured worst case (nested blockquotes, renderer +// recursion): 768 levels fit a 2 MiB thread in release, 1024 do not; 512 keeps +// ~2x headroom. Debug frames are ~6x larger (default 256 fits 2 MiB in debug). +// Test: limits::*hard_ceiling*. +/// Upper bound applied to [`Limits::max_nesting_depth`]. +pub const HARD_MAX_NESTING_DEPTH: usize = 512; + +/// Resource limits applied to every extraction. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct Limits { + /// Maximum input size in bytes, also the cap on a fetched URL body. + /// Default: 50 MiB. + pub max_input_size: usize, + /// Maximum Markdown output size in bytes. Default: 128 MiB. + pub max_output_size: usize, + /// Wall-clock budget for one call: URL fetch, parsing and rendering. + /// Default: 60 s. + pub timeout: Duration, + /// Maximum element nesting. HTML deeper than this is flattened to text; + /// XML parts (DOCX) deeper than this are rejected. Clamped to + /// [`HARD_MAX_NESTING_DEPTH`]. Default: 256. + pub max_nesting_depth: usize, + /// Maximum parsed nodes (HTML DOM nodes; XML elements and text nodes summed + /// over DOCX parts). Default: 5 000 000. + pub max_nodes: usize, + /// Maximum number of entries in a ZIP container (DOCX/XLSX/ODS). Default: 10 000. + pub max_archive_entries: usize, + /// Maximum decompressed size of a single ZIP entry in bytes. Default: 64 MiB. + pub max_archive_entry_size: u64, + /// Maximum total decompressed size of a ZIP container in bytes. Default: 256 MiB. + pub max_archive_total_size: u64, + /// Maximum data rows rendered per sheet (header excluded); extra rows are + /// dropped with a warning. Default: 10 000. + pub max_sheet_rows: usize, + /// Maximum columns rendered per sheet; extra columns are dropped with a + /// warning. Default: 256. + pub max_sheet_cols: usize, + /// Maximum cell grid a workbook may expand to while parsing (XLS/ODS + /// pre-scan, summed over sheets). Default: 4 000 000. + pub max_spreadsheet_cells: u64, + /// Maximum redirects `extract_url` follows. Default: 5. + pub max_redirects: usize, +} + +impl Default for Limits { + fn default() -> Self { + Self { + max_input_size: 50 * 1024 * 1024, + max_output_size: 128 * 1024 * 1024, + timeout: Duration::from_secs(60), + max_nesting_depth: 256, + max_nodes: 5_000_000, + max_archive_entries: 10_000, + max_archive_entry_size: 64 * 1024 * 1024, + max_archive_total_size: 256 * 1024 * 1024, + max_sheet_rows: 10_000, + max_sheet_cols: 256, + max_spreadsheet_cells: 4_000_000, + max_redirects: 5, + } + } +} + +impl Limits { + /// Default limits. + pub fn new() -> Self { + Self::default() + } + + /// Effective nesting depth after the hard ceiling. + pub fn nesting_depth(&self) -> usize { + self.max_nesting_depth.min(HARD_MAX_NESTING_DEPTH) + } + + pub fn max_input_size(mut self, bytes: usize) -> Self { + self.max_input_size = bytes; + self + } + + pub fn max_output_size(mut self, bytes: usize) -> Self { + self.max_output_size = bytes; + self + } + + pub fn timeout(mut self, timeout: Duration) -> Self { + self.timeout = timeout; + self + } + + pub fn max_nesting_depth(mut self, depth: usize) -> Self { + self.max_nesting_depth = depth; + self + } + + pub fn max_nodes(mut self, nodes: usize) -> Self { + self.max_nodes = nodes; + self + } + + pub fn max_archive_entries(mut self, n: usize) -> Self { + self.max_archive_entries = n; + self + } + + pub fn max_archive_entry_size(mut self, bytes: u64) -> Self { + self.max_archive_entry_size = bytes; + self + } + + pub fn max_archive_total_size(mut self, bytes: u64) -> Self { + self.max_archive_total_size = bytes; + self + } + + pub fn max_sheet_rows(mut self, rows: usize) -> Self { + self.max_sheet_rows = rows; + self + } + + pub fn max_sheet_cols(mut self, cols: usize) -> Self { + self.max_sheet_cols = cols; + self + } + + pub fn max_spreadsheet_cells(mut self, cells: u64) -> Self { + self.max_spreadsheet_cells = cells; + self + } + + pub fn max_redirects(mut self, n: usize) -> Self { + self.max_redirects = n; + self + } + + /// Reject inputs larger than `max_input_size`. + pub(crate) fn check_input(&self, len: usize) -> crate::Result<()> { + if len > self.max_input_size { + return Err(Error::ResourceLimit(format!( + "input size {len} exceeds max_input_size {}", + self.max_input_size + ))); + } + Ok(()) + } +} + +// Budget is only driven by the feature-gated parsers. +/// How many [`Budget::expired`] calls share one clock read. +#[cfg_attr( + not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + allow(dead_code) +)] +const CLOCK_EVERY: u32 = 256; + +/// Per-call deadline and node counter, shared by reference through parsers and +/// the renderer (interior mutability so `&self` walkers can charge it). +#[derive(Debug)] +#[cfg_attr( + not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + allow(dead_code) +)] +pub(crate) struct Budget { + deadline: Option, + timeout: Duration, + ticks: Cell, + expired: Cell, + #[cfg_attr(not(any(feature = "html", feature = "docx")), allow(dead_code))] + nodes: Cell, + #[cfg_attr(not(any(feature = "html", feature = "docx")), allow(dead_code))] + max_nodes: usize, +} + +#[cfg_attr( + not(any(feature = "html", feature = "docx", feature = "spreadsheet")), + allow(dead_code) +)] +impl Budget { + /// Budget starting now. + pub fn new(limits: &Limits) -> Self { + #[cfg(not(all(target_arch = "wasm32", target_os = "unknown")))] + { + Self::starting_at(limits, Instant::now()) + } + // wasm32-unknown-unknown has no clock (`Instant::now` panics): no + // deadline there; hosts bound time themselves (e.g. a Worker timeout). + #[cfg(all(target_arch = "wasm32", target_os = "unknown"))] + { + Self { + deadline: None, + timeout: limits.timeout, + ticks: Cell::new(0), + expired: Cell::new(false), + nodes: Cell::new(0), + max_nodes: limits.max_nodes, + } + } + } + + /// Budget whose clock started at `start` (e.g. before a URL fetch). + #[cfg_attr(all(target_arch = "wasm32", target_os = "unknown"), allow(dead_code))] + pub fn starting_at(limits: &Limits, start: Instant) -> Self { + Self { + // Overflow (e.g. Duration::MAX) means "no deadline". + deadline: start.checked_add(limits.timeout), + timeout: limits.timeout, + ticks: Cell::new(0), + expired: Cell::new(false), + nodes: Cell::new(0), + max_nodes: limits.max_nodes, + } + } + + pub fn deadline(&self) -> Option { + self.deadline + } + + /// Cheap periodic deadline check for hot loops; reads the clock every + /// [`CLOCK_EVERY`] calls. Sticky once expired. + pub fn expired(&self) -> bool { + if self.expired.get() { + return true; + } + let t = self.ticks.get().wrapping_add(1); + self.ticks.set(t); + if t.is_multiple_of(CLOCK_EVERY) { + self.read_clock(); + } + self.expired.get() + } + + fn read_clock(&self) { + if let Some(d) = self.deadline + && Instant::now() >= d + { + self.expired.set(true); + } + } + + /// Error if the deadline has passed (reads the clock). + pub fn check_time(&self) -> crate::Result<()> { + self.read_clock(); + if self.expired.get() { + return Err(self.timeout_error()); + } + Ok(()) + } + + pub fn timeout_error(&self) -> Error { + Error::ResourceLimit(format!("processing exceeded timeout of {:?}", self.timeout)) + } + + /// Charge `n` parsed nodes against `max_nodes`. + #[cfg_attr(not(any(feature = "html", feature = "docx")), allow(dead_code))] + pub fn add_nodes(&self, n: usize) -> crate::Result<()> { + let total = self.nodes.get().saturating_add(n); + self.nodes.set(total); + if total > self.max_nodes { + return Err(Error::ResourceLimit(format!( + "document has more than max_nodes {} nodes", + self.max_nodes + ))); + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn nesting_depth_clamped_to_hard_ceiling() { + let l = Limits::default().max_nesting_depth(usize::MAX); + assert_eq!(l.nesting_depth(), HARD_MAX_NESTING_DEPTH); + assert_eq!(Limits::default().nesting_depth(), 256); + } + + #[test] + fn zero_timeout_expires() { + let b = Budget::new(&Limits::default().timeout(Duration::ZERO)); + assert!(matches!(b.check_time(), Err(Error::ResourceLimit(_)))); + assert!((0..CLOCK_EVERY).any(|_| b.expired())); + } + + #[test] + fn huge_timeout_never_expires() { + let b = Budget::new(&Limits::default().timeout(Duration::MAX)); + assert!(b.check_time().is_ok()); + assert!(!(0..CLOCK_EVERY * 2).any(|_| b.expired())); + } + + #[test] + fn node_budget_accumulates() { + let b = Budget::new(&Limits::default().max_nodes(10)); + assert!(b.add_nodes(6).is_ok()); + assert!(b.add_nodes(4).is_ok()); + assert!(matches!(b.add_nodes(1), Err(Error::ResourceLimit(_)))); + } + + #[test] + fn input_check() { + let l = Limits::default().max_input_size(3); + assert!(l.check_input(3).is_ok()); + assert!(matches!(l.check_input(4), Err(Error::ResourceLimit(_)))); + } +} diff --git a/crates/markkit/src/render/markdown.rs b/crates/markkit/src/render/markdown.rs index dca3459..c985c55 100644 --- a/crates/markkit/src/render/markdown.rs +++ b/crates/markkit/src/render/markdown.rs @@ -9,6 +9,13 @@ //! content goes in its top-left slot, covered slots stay empty. Layout is //! bounded by `MAX_TABLE_GRID_CELLS`; a table whose spans would exceed it is //! laid out ignoring spans (threat model T2, invariant S13). +//! - Output size and the deadline are checked per block, list item and table +//! row (not per byte). Tables pad every row to the widest row, so output can +//! be quadratic in input; checking per row keeps any buffer within +//! `max_output_size` plus one row. Nested renderers inherit the limits and +//! report a trip back to their parent. + +use std::time::Instant; use crate::ir::{Alignment, DefinitionItem, Document, ListItem, Node, TableCell}; @@ -18,29 +25,101 @@ const MAX_TABLE_GRID_CELLS: usize = 1_000_000; const MAX_COLSPAN: usize = 1000; const MAX_ROWSPAN: usize = 65534; +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Trip { + Output, + Time, +} + /// Renders a [`Document`] IR into Markdown text. pub struct MarkdownRenderer { /// Output buffer. buf: String, + max_output: usize, + deadline: Option, + tripped: Option, } impl MarkdownRenderer { pub fn new() -> Self { Self { buf: String::with_capacity(4096), + max_output: usize::MAX, + deadline: None, + tripped: None, } } - /// Render a document to a Markdown string. + /// Render a document to a Markdown string (no limits; the IR is trusted). pub fn render(&mut self, doc: &Document) -> String { self.buf.clear(); self.render_nodes(&doc.nodes, 0); normalize_output(&self.buf) } + /// Render with `max_output` bytes and the budget's deadline enforced. + #[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + pub(crate) fn render_bounded( + &mut self, + doc: &Document, + max_output: usize, + budget: &crate::limits::Budget, + ) -> crate::Result { + self.max_output = max_output; + self.deadline = budget.deadline(); + self.tripped = None; + let out = self.render(doc); + match self.tripped { + Some(Trip::Time) => Err(budget.timeout_error()), + Some(Trip::Output) => Err(output_error(max_output)), + None if out.len() > max_output => Err(output_error(max_output)), + None => Ok(out), + } + } + + /// Nested renderer sharing this renderer's limits. + fn sub(&self) -> Self { + Self { + max_output: self.max_output, + deadline: self.deadline, + ..Self::new() + } + } + + /// Render block children to normalized Markdown in a nested renderer. + fn render_blocks(&mut self, nodes: &[Node], depth: usize) -> String { + let mut sub = self.sub(); + sub.render_nodes(nodes, depth); + self.absorb(&sub); + normalize_output(&sub.buf) + } + + /// Fold a nested renderer's trip into this one. + fn absorb(&mut self, sub: &Self) { + if self.tripped.is_none() { + self.tripped = sub.tripped; + } + } + + /// True once a limit is hit; rendering then stops at the next check. + fn over_limit(&mut self) -> bool { + if self.tripped.is_some() { + return true; + } + if self.buf.len() > self.max_output { + self.tripped = Some(Trip::Output); + } else if self.deadline.is_some_and(|d| Instant::now() >= d) { + self.tripped = Some(Trip::Time); + } + self.tripped.is_some() + } + /// Render a slice of nodes. fn render_nodes(&mut self, nodes: &[Node], depth: usize) { for (i, node) in nodes.iter().enumerate() { + if self.over_limit() { + return; + } if i > 0 && node.is_block() && nodes[i - 1].is_block() { self.buf.push('\n'); } @@ -80,8 +159,9 @@ impl MarkdownRenderer { Node::BlockQuote { children, .. } => { let inner = { - let mut sub = MarkdownRenderer::new(); + let mut sub = self.sub(); sub.render_nodes(children, depth); + self.absorb(&sub); normalize_output(&sub.buf) }; for line in inner.lines() { @@ -153,7 +233,7 @@ impl MarkdownRenderer { Node::FootnoteDefinition { label, children, .. } => { - let inner = render_blocks(children, depth); + let inner = self.render_blocks(children, depth); self.buf.push_str("[^"); self.buf.push_str(&footnote_label(label)); self.buf.push_str("]:"); @@ -179,6 +259,9 @@ impl MarkdownRenderer { /// Render a definition list: `Term` line, then `: definition` per definition. fn render_definition_list(&mut self, items: &[DefinitionItem], depth: usize) { for (i, item) in items.iter().enumerate() { + if self.over_limit() { + return; + } if i > 0 { self.buf.push('\n'); } @@ -187,7 +270,7 @@ impl MarkdownRenderer { trim_trailing_inline(&mut self.buf, start); self.buf.push('\n'); for def in &item.definitions { - let inner = render_blocks(def, depth); + let inner = self.render_blocks(def, depth); self.buf.push(':'); push_hanging(&mut self.buf, &inner, " ", " "); } @@ -200,6 +283,9 @@ impl MarkdownRenderer { let start_num = start.unwrap_or(1); for (i, item) in items.iter().enumerate() { + if self.over_limit() { + return; + } let marker = if ordered { format!("{}{}. ", indent, start_num + i as u64) } else { @@ -235,8 +321,9 @@ impl MarkdownRenderer { self.render_list(*sub_ord, *sub_start, sub_items, depth + 1); } _ => { - let mut sub = MarkdownRenderer::new(); + let mut sub = self.sub(); sub.render_node(child, depth + 1); + self.absorb(&sub); let text = sub.buf.trim_end().to_string(); self.buf.push_str(&text); self.buf.push('\n'); @@ -253,8 +340,9 @@ impl MarkdownRenderer { self.render_list(*sub_ord, *sub_start, sub_items, depth + 1); } _ => { - let mut sub = MarkdownRenderer::new(); + let mut sub = self.sub(); sub.render_node(child, depth + 1); + self.absorb(&sub); let inner = sub.buf.trim_end().to_string(); let sub_indent = " ".repeat(depth + 1); for line in inner.lines() { @@ -368,6 +456,9 @@ impl MarkdownRenderer { } for (r, row) in grid.iter().enumerate() { + if self.over_limit() { + return; + } self.buf.push('|'); for i in 0..col_count { self.buf.push(' '); @@ -454,13 +545,6 @@ fn layout_table<'a>(rows: &[&'a [TableCell]], spans: bool) -> Option String { - let mut sub = MarkdownRenderer::new(); - sub.render_nodes(nodes, depth); - normalize_output(&sub.buf) -} - /// Append `inner` with `first` before its first line and `rest` before the /// other non-blank lines (hanging indent for footnotes / definitions). fn push_hanging(buf: &mut String, inner: &str, first: &str, rest: &str) { @@ -517,6 +601,11 @@ impl Default for MarkdownRenderer { } } +#[cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] +fn output_error(max: usize) -> crate::Error { + crate::Error::ResourceLimit(format!("output exceeds max_output_size {max}")) +} + /// Trim leading and trailing whitespace from inline content rendered since `start`. fn trim_trailing_inline(buf: &mut String, start: usize) { // Trim trailing spaces diff --git a/crates/markkit/src/spreadsheet.rs b/crates/markkit/src/spreadsheet.rs index 1df6fbe..5e4c574 100644 --- a/crates/markkit/src/spreadsheet.rs +++ b/crates/markkit/src/spreadsheet.rs @@ -29,6 +29,7 @@ use crate::config::MarkkitBuilder; use crate::error::Error; use crate::format::Format; use crate::ir::{Document, Node, Source, TableCell}; +use crate::limits::Budget; use crate::render::escape_table_cell; pub(crate) struct SheetOutput { @@ -63,7 +64,11 @@ enum Container { Xls, } -pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result { +pub(crate) fn parse( + bytes: &[u8], + config: &MarkkitBuilder, + budget: &Budget, +) -> Result { let mut patched = None; let (container, title) = if bytes.starts_with(b"PK") { let mut zip = ZipGuard::open(bytes, config).map_err(archive_err)?; @@ -86,13 +91,13 @@ pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result Result Result Result<(), calamine::XlsxError> { let mut reader = wb.worksheet_cells_reader(&name)?; while let Some(cell) = reader.next_cell()? { + if budget.expired() { + break; + } let (r, c) = cell.get_position(); grid.add(r, c, fmt_ref(cell.get_value())); } Ok(()) })(); + budget.check_time()?; match res { Ok(()) => out.push_sheet(&name, grid), Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), @@ -132,13 +142,13 @@ pub(crate) fn parse(bytes: &[u8], config: &MarkkitBuilder) -> Result { let mut wb: Ods<_> = Ods::new(Cursor::new(bytes)).map_err(err)?; let sel = out.select(wb.sheets_metadata()); - read_ranges(&mut wb, sel, &mut out, config); + read_ranges(&mut wb, sel, &mut out, config, budget)?; out.finish(Format::Ods, title) } Container::Xls => { let mut wb: Xls<_> = Xls::new(Cursor::new(bytes)).map_err(err)?; let sel = out.select(wb.sheets_metadata()); - read_ranges(&mut wb, sel, &mut out, config); + read_ranges(&mut wb, sel, &mut out, config, budget)?; out.finish(Format::Xls, title) } } @@ -149,7 +159,9 @@ fn read_ranges( sel: Vec<(String, bool)>, out: &mut Collector, config: &MarkkitBuilder, -) where + budget: &Budget, +) -> Result<(), Error> +where RS: Read + Seek, R: Reader, R::Error: std::fmt::Display, @@ -158,17 +170,25 @@ fn read_ranges( if !include { continue; } - match wb.worksheet_range(&name) { + let range = wb.worksheet_range(&name); + // calamine decodes the whole sheet before returning: check after it. + budget.check_time()?; + match range { Ok(range) => { let mut grid = Grid::new(config); for (r, c, v) in range.used_cells() { + if budget.expired() { + break; + } grid.add(r as u32, c as u32, fmt_data(v)); } + budget.check_time()?; out.push_sheet(&name, grid); } Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), } } + Ok(()) } // ─── Cell formatting ──────────────────────────────────────────────── @@ -265,8 +285,8 @@ impl Grid { fn new(config: &MarkkitBuilder) -> Self { Self { rows: BTreeMap::new(), - max_rows: config.max_sheet_rows, - max_cols: config.max_sheet_cols.max(1), + max_rows: config.limits.max_sheet_rows, + max_cols: config.limits.max_sheet_cols.max(1), truncated_rows: false, truncated_cols: false, } @@ -496,15 +516,20 @@ fn clean_title(raw: &str) -> Option { /// `dc:title` from OOXML `docProps/core.xml` or ODF `meta.xml`. Best effort: /// a missing or malformed part yields `None`; only archive limits are errors. -fn read_dc_title(zip: &mut ZipGuard<'_>, part: &str) -> Result, Error> { +fn read_dc_title( + zip: &mut ZipGuard<'_>, + part: &str, + config: &MarkkitBuilder, + budget: &Budget, +) -> Result, Error> { let Some(xml) = zip.read(part).map_err(archive_err)? else { return Ok(None); }; - Ok(dc_title(&xml)) + Ok(dc_title(&xml, config.limits.nesting_depth(), budget)) } -fn dc_title(xml: &[u8]) -> Option { - let root = crate::xml::parse(xml).ok()?; +fn dc_title(xml: &[u8], max_depth: usize, budget: &Budget) -> Option { + let root = crate::xml::parse(xml, max_depth, budget).ok()?; clean_title(&root.find("title")?.text()) } @@ -607,7 +632,7 @@ fn prescan_xls( .find(|n| comp.is_stream(n)) .ok_or_else(|| err("XLS has no Workbook stream"))?; let mut stream = Vec::new(); - let cap = config.max_archive_entry_size; + let cap = config.limits.max_archive_entry_size; comp.open_stream(name) .map_err(|e| err(format!("cannot open Workbook stream: {e}")))? .take(cap.saturating_add(1)) @@ -618,7 +643,7 @@ fn prescan_xls( "XLS Workbook stream exceeds {cap} bytes" ))); } - scan_biff(&stream, config.max_spreadsheet_cells) + scan_biff(&stream, config.limits.max_spreadsheet_cells) } /// Estimate the cells calamine will allocate and reject inputs that would @@ -901,12 +926,16 @@ mod tests { fn dc_title_reads_and_normalizes() { let core = br#" Q3 Budget & Plan "#; - assert_eq!(dc_title(core).as_deref(), Some("Q3 Budget & Plan")); - assert_eq!(dc_title(b" "), None); - assert_eq!(dc_title(b""), None); - assert_eq!(dc_title(b"\xff\xfe<<"), None); + let b = Budget::new(&crate::Limits::default()); + assert_eq!(dc_title(core, 256, &b).as_deref(), Some("Q3 Budget & Plan")); + assert_eq!(dc_title(b" ", 256, &b), None); + assert_eq!(dc_title(b"", 256, &b), None); + assert_eq!(dc_title(b"\xff\xfe<<", 256, &b), None); let long = format!("{}", "x".repeat(5000)); - assert_eq!(dc_title(long.as_bytes()).unwrap().len(), MAX_TITLE_CHARS); + assert_eq!( + dc_title(long.as_bytes(), 256, &b).unwrap().len(), + MAX_TITLE_CHARS + ); } fn summary_info(codepage: u16, typ: u32, value: &[u8], count: u32) -> Vec { diff --git a/crates/markkit/src/tool.rs b/crates/markkit/src/tool.rs index b14ccee..e5208b3 100644 --- a/crates/markkit/src/tool.rs +++ b/crates/markkit/src/tool.rs @@ -595,6 +595,7 @@ fn parse_request(inner: &ToolInner, args: serde_json::Value) -> Result XmlError { + match e { + Error::ResourceLimit(m) => XmlError::Limit(m), + other => XmlError::Invalid(other.to_string()), + } +} + /// Parse an XML document into a tree. Returns a synthetic root whose children /// are the document's top-level nodes. -pub(crate) fn parse(bytes: &[u8]) -> Result { +pub(crate) fn parse(bytes: &[u8], max_depth: usize, budget: &Budget) -> Result { let mut reader = Reader::from_reader(bytes); reader.config_mut().check_end_names = false; let mut stack: Vec = vec![Element::default()]; @@ -137,11 +146,20 @@ pub(crate) fn parse(bytes: &[u8]) -> Result { let event = reader .read_event_into(&mut buf) .map_err(|e| XmlError::Invalid(format!("malformed XML: {e}")))?; + if matches!( + event, + Event::Start(_) | Event::Empty(_) | Event::Text(_) | Event::CData(_) + ) { + budget.add_nodes(1).map_err(limit)?; + if budget.expired() { + return Err(limit(budget.timeout_error())); + } + } match event { Event::Start(s) => { - if stack.len() > MAX_XML_DEPTH { + if stack.len() > max_depth { return Err(XmlError::Limit(format!( - "XML nesting deeper than {MAX_XML_DEPTH}" + "XML nesting deeper than max_nesting_depth {max_depth}" ))); } stack.push(element_from(&s)); @@ -202,6 +220,11 @@ pub(crate) fn parse(bytes: &[u8]) -> Result { #[cfg(test)] mod tests { use super::*; + use crate::limits::Limits; + + fn parse(bytes: &[u8]) -> Result { + super::parse(bytes, 256, &Budget::new(&Limits::default())) + } #[test] fn parses_local_names_text_and_refs() { @@ -216,8 +239,35 @@ mod tests { #[test] fn depth_limited() { - let deep = "".repeat(MAX_XML_DEPTH + 10); + let deep = "".repeat(256 + 10); assert!(matches!(parse(deep.as_bytes()), Err(XmlError::Limit(_)))); + let ok = format!("{}{}", "".repeat(100), "".repeat(100)); + assert!(parse(ok.as_bytes()).is_ok()); + let budget = Budget::new(&Limits::default()); + assert!(matches!( + super::parse(ok.as_bytes(), 50, &budget), + Err(XmlError::Limit(_)) + )); + } + + #[test] + fn node_count_limited() { + let many = format!("{}", "".repeat(100)); + let budget = Budget::new(&Limits::default().max_nodes(50)); + assert!(matches!( + super::parse(many.as_bytes(), 256, &budget), + Err(XmlError::Limit(m)) if m.contains("max_nodes") + )); + } + + #[test] + fn deadline_enforced() { + let many = format!("{}", "".repeat(1000)); + let budget = Budget::new(&Limits::default().timeout(std::time::Duration::ZERO)); + assert!(matches!( + super::parse(many.as_bytes(), 256, &budget), + Err(XmlError::Limit(m)) if m.contains("timeout") + )); } #[test] diff --git a/crates/markkit/tests/html_integration.rs b/crates/markkit/tests/html_integration.rs index 9f446af..a10e233 100644 --- a/crates/markkit/tests/html_integration.rs +++ b/crates/markkit/tests/html_integration.rs @@ -593,7 +593,10 @@ fn code_block_with_backticks() { #[test] fn config_max_processing_secs() { let m = Markkit::builder().max_processing_secs(30).build(); - assert_eq!(m.config().max_processing_secs, 30); + assert_eq!( + m.config().limits.timeout, + std::time::Duration::from_secs(30) + ); } #[test] diff --git a/crates/markkit/tests/html_security.rs b/crates/markkit/tests/html_security.rs index c229a6e..af22b21 100644 --- a/crates/markkit/tests/html_security.rs +++ b/crates/markkit/tests/html_security.rs @@ -165,36 +165,36 @@ fn null_bytes() { // ─── Depth Limits ─────────────────────────────────────────────────── #[test] -fn depth_limit_200() { - // MAX_DEPTH is 200. With 200 nested divs, the inner

is at depth 200 - // (body=0, div1=1, ..., div200=200). At depth >= 200 the parser - // falls back to text extraction. The content should still appear. +fn depth_limit_default() { + // Default max_nesting_depth is 256. With 256 nested divs, the inner

is + // at depth 256 (body=0, div1=1, ...). At depth >= 256 the parser falls + // back to text extraction. The content should still appear. let mut html = String::new(); - for _ in 0..200 { + for _ in 0..256 { html.push_str("

"); } html.push_str("

Deep content

"); - for _ in 0..200 { + for _ in 0..256 { html.push_str("
"); } let md = convert(&html); assert!( md.contains("Deep content"), - "Content at depth 200 should still be extracted as text: {:?}", + "Content at depth 256 should still be extracted as text: {:?}", md ); } #[test] fn depth_limit_exceeded() { - // 250 nested divs — well past MAX_DEPTH (200). + // 300 nested divs — past the default max_nesting_depth (256). // Content should still be extracted (flattened to text), not lost. let mut html = String::new(); - for _ in 0..250 { + for _ in 0..300 { html.push_str("
"); } html.push_str("

Very deep content

"); - for _ in 0..250 { + for _ in 0..300 { html.push_str("
"); } let md = convert(&html); diff --git a/crates/markkit/tests/limits.rs b/crates/markkit/tests/limits.rs new file mode 100644 index 0000000..046240f --- /dev/null +++ b/crates/markkit/tests/limits.rs @@ -0,0 +1,269 @@ +//! Resource limits (threat model T2/T6): every limit in `markkit::Limits` is +//! enforced, defaults are generous, and the nesting hard ceiling keeps deep +//! input off the host stack. + +#![cfg(any(feature = "html", feature = "docx", feature = "spreadsheet"))] + +mod common; + +use std::time::Duration; + +use markkit::{DocumentHint, Error, Limits, Markkit}; + +#[allow(dead_code)] +const DOCX_CT: &str = "application/vnd.openxmlformats-officedocument.wordprocessingml.document"; +#[allow(dead_code)] +const XLSX_CT: &str = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; + +fn limit_msg(r: markkit::Result) -> String { + match r { + Err(Error::ResourceLimit(m)) => m, + other => panic!("expected ResourceLimit, got {other:?}"), + } +} + +#[test] +fn defaults_are_generous() { + let l = Limits::default(); + assert_eq!(l.max_input_size, 50 * 1024 * 1024); + assert_eq!(l.max_output_size, 128 * 1024 * 1024); + assert_eq!(l.timeout, Duration::from_secs(60)); + assert_eq!(l.max_nesting_depth, 256); + assert_eq!(l.max_nodes, 5_000_000); + assert_eq!(l.max_redirects, 5); + assert_eq!(Markkit::builder().build().config().limits, l); +} + +#[test] +fn builder_limits_replaces_all() { + let l = Limits::default().max_output_size(7).max_nodes(9); + let m = Markkit::builder().limits(l.clone()).build(); + assert_eq!(m.config().limits, l); + let m = Markkit::builder() + .max_processing_secs(3) + .max_nesting_depth(12) + .build(); + assert_eq!(m.config().limits.timeout, Duration::from_secs(3)); + assert_eq!(m.config().limits.max_nesting_depth, 12); +} + +/// Stack for hard-ceiling tests: 2 MiB (Rust's default thread stack) in +/// release; debug frames are ~6x larger, so debug runs get 8 MiB. +#[allow(dead_code)] +fn small_stack() -> usize { + if cfg!(debug_assertions) { + 8 << 20 + } else { + 2 << 20 + } +} + +// ─── HTML ─────────────────────────────────────────────────────────── + +#[cfg(feature = "html")] +mod html { + use super::*; + use markkit::limits::HARD_MAX_NESTING_DEPTH; + + fn deep(tag: &str, n: usize, inner: &str) -> String { + format!( + "{}{inner}{}", + format!("<{tag}>").repeat(n), + format!("").repeat(n) + ) + } + + #[test] + fn output_cap_rejects_large_output() { + let html = "

hello world

".repeat(1000); + let m = Markkit::builder().max_output_size(1000).build(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + let m = Markkit::builder().max_output_size(100_000).build(); + assert!(m.extract_html_sync(&html).is_ok()); + } + + #[test] + fn output_cap_stops_table_amplification() { + // One wide row pads every other row: output ~ rows x cols. + let wide = format!("{}", "x".repeat(2000)); + let html = format!( + "{wide}{}
", + "y".repeat(2000) + ); + let m = Markkit::builder().max_output_size(1 << 20).build(); + let start = std::time::Instant::now(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + assert!(start.elapsed() < Duration::from_secs(10)); + } + + #[test] + fn output_cap_applies_inside_nested_renderers() { + let wide = format!("{}", "x".repeat(500)); + let table = format!("{wide}{}
", "y".repeat(500)); + for html in [ + format!("
{table}
"), + format!("

  • a

    {table}
"), + ] { + let m = Markkit::builder().max_output_size(64 * 1024).build(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + } + } + + #[test] + fn zero_timeout_rejects() { + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(m.extract_html_sync("

x

")).contains("timeout")); + let r = m.extract_document_sync(b"

x

", DocumentHint::content_type("text/html")); + assert!(limit_msg(r).contains("timeout")); + } + + #[test] + fn node_cap_rejects() { + let html = "

a

".repeat(1000); + let m = Markkit::builder().max_nodes(500).build(); + assert!(limit_msg(m.extract_html_sync(&html)).contains("max_nodes")); + assert!(limit_msg(m.parse_html(&html)).contains("max_nodes")); + let m = Markkit::builder().max_nodes(10_000).build(); + assert!(m.extract_html_sync(&html).is_ok()); + } + + #[test] + fn nesting_depth_configurable_flattens() { + let html = deep("blockquote", 20, "

Deep

"); + let full = Markkit::builder().build().extract_html_sync(&html).unwrap(); + assert!(full.contains(&"> ".repeat(20))); + let m = Markkit::builder().max_nesting_depth(5).build(); + let md = m.extract_html_sync(&html).unwrap(); + assert!(md.contains("Deep")); + assert!(!md.contains(&"> ".repeat(6))); + } + + #[test] + fn hard_ceiling_survives_small_stack() { + // Configured depth is clamped; deep nesting of recursive constructs + // must fit a small thread stack. + let inputs = [ + deep("blockquote", HARD_MAX_NESTING_DEPTH * 4, "

q

"), + deep("ul", HARD_MAX_NESTING_DEPTH * 4, "
  • l
    • "), + deep("div", HARD_MAX_NESTING_DEPTH * 4, "

    d

    "), + deep("b", HARD_MAX_NESTING_DEPTH * 4, "t"), + ]; + std::thread::Builder::new() + .stack_size(small_stack()) + .spawn(move || { + let m = Markkit::builder().max_nesting_depth(usize::MAX).build(); + for html in inputs { + m.extract_html_sync(&html).unwrap(); + } + }) + .unwrap() + .join() + .expect("deep input overflowed a small stack"); + } +} + +// ─── DOCX ─────────────────────────────────────────────────────────── + +#[cfg(feature = "docx")] +mod docx { + use super::*; + use common::*; + use markkit::limits::HARD_MAX_NESTING_DEPTH; + + fn run(m: &Markkit, bytes: &[u8]) -> markkit::Result { + m.extract_document_sync(bytes, DocumentHint::content_type(DOCX_CT)) + } + + #[test] + fn node_cap_rejects() { + let body: String = (0..200).map(|i| para(&format!("p{i}"))).collect(); + let bytes = docx(&body); + let m = Markkit::builder().max_nodes(100).build(); + assert!(limit_msg(run(&m, &bytes)).contains("max_nodes")); + assert!(run(&Markkit::builder().build(), &bytes).is_ok()); + } + + #[test] + fn nesting_depth_configurable() { + let body = format!( + "{}{}{}", + "".repeat(10), + para("x"), + "".repeat(10) + ); + let bytes = docx(&body); + assert!(run(&Markkit::builder().build(), &bytes).is_ok()); + let m = Markkit::builder().max_nesting_depth(8).build(); + assert!(limit_msg(run(&m, &bytes)).contains("max_nesting_depth")); + } + + #[test] + fn hard_ceiling_survives_small_stack() { + let n = HARD_MAX_NESTING_DEPTH / 2 - 8; + let body = format!( + "{}{}{}", + "".repeat(n), + para("x"), + "".repeat(n) + ); + let bytes = docx(&body); + let deeper = docx(&"".repeat(HARD_MAX_NESTING_DEPTH * 2)); + std::thread::Builder::new() + .stack_size(small_stack()) + .spawn(move || { + let m = Markkit::builder().max_nesting_depth(usize::MAX).build(); + assert!(run(&m, &bytes).unwrap().markdown.contains('x')); + assert!(limit_msg(run(&m, &deeper)).contains("max_nesting_depth")); + }) + .unwrap() + .join() + .expect("deep DOCX overflowed a small stack"); + } + + #[test] + fn zero_timeout_rejects() { + let bytes = docx(¶("x")); + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(run(&m, &bytes)).contains("timeout")); + } + + #[test] + fn output_cap_rejects() { + let body: String = (0..200).map(|i| para(&format!("paragraph {i}"))).collect(); + let bytes = docx(&body); + let m = Markkit::builder().max_output_size(500).build(); + assert!(limit_msg(run(&m, &bytes)).contains("max_output_size")); + } +} + +// ─── Spreadsheets ─────────────────────────────────────────────────── + +#[cfg(feature = "spreadsheet")] +mod spreadsheet { + use super::*; + use common::*; + + fn sheet() -> Vec { + let rows = (0..200) + .map(|i| vec![Cell::Str(format!("row {i}")), Cell::N(i as f64)]) + .collect(); + xlsx(&[Sheet::new("S", rows)]) + } + + fn run(m: &Markkit, bytes: &[u8]) -> markkit::Result { + m.extract_document_sync(bytes, DocumentHint::content_type(XLSX_CT)) + } + + #[test] + fn zero_timeout_rejects() { + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(run(&m, &sheet())).contains("timeout")); + } + + #[test] + fn output_cap_rejects() { + let m = Markkit::builder().max_output_size(500).build(); + assert!(limit_msg(run(&m, &sheet())).contains("max_output_size")); + assert!(run(&Markkit::builder().build(), &sheet()).is_ok()); + } +} diff --git a/examples/configuration.rs b/examples/configuration.rs index d9213a7..c307d9d 100644 --- a/examples/configuration.rs +++ b/examples/configuration.rs @@ -34,4 +34,17 @@ fn main() { Ok(_) => println!("Unexpected success"), Err(e) => println!("--- max_input_size(10) ---\nError: {e}"), } + + // All limits at once: output cap, timeout, nesting depth, node count, ... + let bounded = Markkit::builder() + .limits( + markkit::Limits::default() + .max_output_size(20) + .timeout(std::time::Duration::from_secs(5)), + ) + .build(); + match bounded.extract_html_sync(html) { + Ok(_) => println!("Unexpected success"), + Err(e) => println!("--- max_output_size(20) ---\nError: {e}"), + } } diff --git a/knowledge/formats/html-extraction.md b/knowledge/formats/html-extraction.md index c024cce..3788639 100644 --- a/knowledge/formats/html-extraction.md +++ b/knowledge/formats/html-extraction.md @@ -131,7 +131,7 @@ Trimming at render time: ## Depth Limiting -Constant: `MAX_DEPTH = 200` +Limit: `limits.max_nesting_depth` (default 256, clamped to `HARD_MAX_NESTING_DEPTH` = 512) Behavior at depth limit: - `convert_children`: extracts all descendant text via `.text().collect()`, trims, emits as single `Text` node diff --git a/knowledge/foundations/architecture.md b/knowledge/foundations/architecture.md index d81cd77..d4c6c09 100644 --- a/knowledge/foundations/architecture.md +++ b/knowledge/foundations/architecture.md @@ -61,8 +61,10 @@ are trusted embedder code; the first error aborts and is returned as is ## Configuration Via `MarkkitBuilder`: -- Input size limits -- Processing time limits +- Resource limits: one `Limits` struct (`limits.rs`, set whole via `.limits(..)` or per + field via builder setters): input/output size, timeout, nesting depth (hard max 512), + node count, archive, spreadsheet and redirect caps. Violations are + `Error::ResourceLimit`; see [Threat Model](../security/threat-model.md) Resource Limits - CSS selectors for include/exclude - Link and image preservation - Office-document limits (archive entries/sizes, sheet rows/cols, grid cells, hidden sheets) diff --git a/knowledge/foundations/ir.md b/knowledge/foundations/ir.md index 58b71ab..28b00bb 100644 --- a/knowledge/foundations/ir.md +++ b/knowledge/foundations/ir.md @@ -162,7 +162,7 @@ At render time, the `normalize_output` function: | No empty Text nodes after whitespace collapse | `collapse_whitespace` returns empty string for whitespace-only; parser skips empty | | Block lists hold no bare inline runs | `normalize_blocks` drops whitespace-only runs, wraps others in `Paragraph` | | Empty containers are elided | Parser returns `None` for headings, paragraphs, lists, etc. with no children | -| Depth bounded at 200 | `MAX_DEPTH` constant; at limit, subtree flattened to text | +| Depth bounded (`limits.max_nesting_depth`, default 256, hard max 512) | HTML parser flattens the subtree to text at the limit | | No executable content in IR | Script, style, iframe, etc. stripped by parser | | `List.start` is `None` when `ordered=false` | Parser only reads `start` attr for `
      ` | | Table spans ≥ 1 | `TableCell::with_span` clamps 0 to 1; HTML parser clamps to 1000 / 65534 | diff --git a/knowledge/log.md b/knowledge/log.md index eb9324f..293615e 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -27,3 +27,4 @@ * **Accuracy pass**: [Python Package](runtimes/python-package.md) now names the shipped `markkit` module; [Examples](operations/examples.md) lists the four real examples; [Evaluation Harness](operations/eval.md) is marked as a stub; [Threat Model](security/threat-model.md) T3 records that `http_client` is a default feature and SSRF mitigations are still open. * **Maintenance contract**: [Maintenance](operations/maintenance.md) adopts the analyze, fix, ship invocation contract and a workflow credential audit. * **Benchmarks**: `markkit-bench` is no longer a stub. It measures latency, ops/s, throughput and per-op heap allocations (count, bytes, peak) plus process peak RSS over a generated corpus, and `just bench` saves JSON + Markdown runs to `crates/markkit-bench/results/`, like bashkit. Criterion moves to `just bench-criterion`. See [Performance Results](operations/performance-results.md). +* **Resource limits**: all limits moved into one `Limits` struct (bashkit `ExecutionLimits` style). New: `timeout` now bounds parsing and rendering, not just URL fetch; `max_output_size` (128 MiB, stops table amplification); `max_nodes` (5M); `max_nesting_depth` configurable (256, was hardcoded 200 HTML / 256 XML) with a 512 hard ceiling. Threat T2h, invariants S14-S16, `limits` tests. See [Threat Model](security/threat-model.md) T2. diff --git a/knowledge/operations/testing.md b/knowledge/operations/testing.md index c59bcb1..f83a318 100644 --- a/knowledge/operations/testing.md +++ b/knowledge/operations/testing.md @@ -21,6 +21,7 @@ tags: | `crates/markkit/tests/docx_integration.rs`, `spreadsheet_integration.rs` | Office mapping | | `crates/markkit/tests/document_detection.rs` | Format detection order and explicit unsupported errors | | `crates/markkit/tests/document_security.rs` | Zip bombs, grid amplification, XML entities, malformed containers | +| `crates/markkit/tests/limits.rs` | Every `Limits` field enforced for HTML, DOCX, spreadsheets; small-stack depth ceiling | | `crates/markkit/tests/common/` | In-memory DOCX/XLSX/XLS/ODS fixture builders | | `crates/markkit-cli/tests/cli_integration.rs` | CLI flags and output | | `crates/markkit/benches/extraction.rs` | Criterion micro-benchmarks | diff --git a/knowledge/runtimes/wasm-package.md b/knowledge/runtimes/wasm-package.md index b8415a5..33bbb45 100644 --- a/knowledge/runtimes/wasm-package.md +++ b/knowledge/runtimes/wasm-package.md @@ -44,6 +44,10 @@ Unknown keys and wrong types throw `invalid options`: values go through `serde_json::Value` because serde-wasm-bindgen only reads named fields, so `deny_unknown_fields` alone would never see a typo. +wasm32-unknown-unknown has no clock (`Instant::now` panics), so the core +`timeout` limit is not enforced here (`Budget` runs without a deadline); every +other limit applies. Hosts bound time themselves, e.g. by terminating a Worker. + Errors throw `Error` with the Rust `Display` text. `MarkkitTool.execute` rejects with an `Error` carrying `userFacing` (see [Tool Contract](../integrations/tool-contract.md)). diff --git a/knowledge/security/security-model.md b/knowledge/security/security-model.md index 47789df..fecb3b1 100644 --- a/knowledge/security/security-model.md +++ b/knowledge/security/security-model.md @@ -28,7 +28,7 @@ Markkit processes untrusted input from the internet. Key threats: ### TM-SSRF: Server-side request forgery (URL fetching) - **Mitigation**: Non-public addresses refused by default, after DNS resolution and on every redirect (opt in with `allow_private_networks`) -- **Mitigation**: http/https only, optional host allowlist, redirect limit, body bounded by `max_input_size`, fetch bounded by `max_processing_secs` +- **Mitigation**: http/https only, optional host allowlist, redirect limit, body bounded by `max_input_size`, fetch bounded by `timeout` - **Residual**: an environment proxy resolves names itself; see [Threat Model](threat-model.md) T3 ### TM-DOC: Malicious office documents diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index e16a499..6113b8f 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -66,19 +66,31 @@ tags: **Sub-threats**: +All limits live in one `Limits` struct (`crates/markkit/src/limits.rs`, +`MarkkitBuilder::limits`); violations return `Error::ResourceLimit` naming the limit. + #### T2a: Oversized Input - Input size checked before parsing against `max_input_size` - Default: 50 MB - Returns `Error::ResourceLimit` on violation #### T2b: Deep Nesting (Stack Overflow) -- `MAX_DEPTH=200` limits recursion in HTML parser -- At depth limit, subtree flattened to plain text (no further recursion) -- Tested with 1000 levels of nesting - -#### T2c: Many Elements (CPU Exhaustion) -- `max_processing_secs` configurable (default: 60s) -- Tested with 1000 paragraph elements +- `max_nesting_depth` (default 256) bounds recursion: HTML flattens deeper subtrees to + plain text; DOCX XML parts deeper than it are rejected +- Clamped to `HARD_MAX_NESTING_DEPTH` (512) whatever the embedder sets: 512 levels + of the worst recursive construct (nested blockquotes) fit a 2 MiB stack in release + with ~2x headroom (1024 did not after IR v2); debug frames are ~6x larger +- Tested with 4096 levels under a small thread stack + +#### T2c: Many Elements / Slow Input (CPU Exhaustion) +- `timeout` (default 60 s) is one wall-clock budget per call covering URL fetch, + parsing and rendering. Checked cooperatively in the HTML walk, XML tree build, DOCX + conversion, spreadsheet cell loops and renderer. html5ever tokenizing and calamine + sheet decoding are not interruptible; they are linear in the (capped) input and the + deadline is checked right after them +- `max_nodes` (default 5 000 000) caps HTML DOM nodes and DOCX XML nodes (summed over parts) +- wasm32-unknown-unknown has no clock, so `timeout` is not enforced in `@everruns/markkit-wasm`; + hosts bound time themselves (e.g. terminate the Worker). All other limits apply #### T2e: Table Span Amplification - `colspan`/`rowspan` clamped by the HTML parser (1000 / 65534) and again at layout @@ -103,6 +115,12 @@ tags: - Tests: `html_main_content::deep_nesting_does_not_overflow`, `many_paragraphs_stay_linear`, `many_sibling_containers_stay_bounded`, `non_content_elements_still_stripped` +#### T2h: Output Amplification +- GFM tables pad every row to the widest row, so output can be rows x columns of input +- `max_output_size` (default 128 MiB) checked per block, list item and table row, + including nested renderers (blockquotes, list items), so no buffer grows past the + limit by more than one row + **Test Coverage**: - `html_security::deeply_nested_elements` - `html_security::very_long_text_node` @@ -112,6 +130,9 @@ tags: - `html_security::hostile_colspan_rowspan_bounded` - `html_security::deep_definition_lists_bounded` - `html_proptest::resource_limit_enforced` +- `limits::*` (every `Limits` field: output cap incl. table amplification, timeout, + node cap, configurable depth, hard depth ceiling on a small stack, for HTML, DOCX and + spreadsheets) ### T3: Server-Side Request Forgery (SSRF) @@ -142,7 +163,7 @@ feature is enabled (off by default; the CLI and Python/Node bindings enable it). - Redirects capped by `max_redirects` (default 5) - Response body bounded by `max_input_size`: declared `Content-Length` rejected before reading, undeclared bodies stopped while streaming (`ResourceLimit`) -- Whole fetch (connect, headers, body) bounded by `max_processing_secs` +- Whole fetch (connect, headers, body) bounded by `timeout`; parse and render share the remaining budget - Opt-out: embedders that fetch content themselves build with `default-features = false` and never link reqwest (CI asserts this for `docx,spreadsheet`) @@ -172,6 +193,7 @@ the proxy, or use `allow_host`. **Mitigations**: - Input size limit (pre-parse) - Depth limit prevents unbounded tree growth +- `max_nodes` caps parsed tree size; `max_output_size` caps the Markdown buffer - `String::with_capacity(4096)` for renderer output buffer (bounded initial allocation) **Test Coverage**: @@ -269,18 +291,18 @@ process, or smuggling content past a human reviewer. | Resource | Default | Configurable | Enforcement Point | |----------|---------|--------------|-------------------| | Input size | 50 MB | `max_input_size(usize)` | Before parsing | -| Processing time | 60 s | `max_processing_secs(u64)` | Whole `extract_url` fetch (parsing itself TBD) | +| Output size | 128 MiB | `max_output_size(usize)` | Renderer, per block / list item / table row | +| Processing time | 60 s | `timeout(Duration)`, `max_processing_secs(u64)` | One budget per call: fetch, parse, render (cooperative) | +| Parsed nodes | 5 000 000 | `max_nodes(usize)` | After html5ever (DOM count); while building DOCX XML trees | +| Nesting depth | 256 (hard max 512) | `max_nesting_depth(usize)` | HTML walk (flatten); DOCX XML build (reject) | | Redirects (`extract_url`) | 5 | `max_redirects(usize)` | Every redirect hop | | Fetched body (`extract_url`) | = input size | `max_input_size(usize)` | `Content-Length` and while streaming | -| Nesting depth | 200 | Hardcoded `MAX_DEPTH` | During HTML tree walk | | Table span grid | 1 000 000 slots | Hardcoded `MAX_TABLE_GRID_CELLS` | Renderer table layout (spans dropped past it) | -| XML nesting depth (DOCX) | 256 | Hardcoded `MAX_XML_DEPTH` | While building the XML tree | | ZIP entries | 10 000 | `max_archive_entries(usize)` | On archive open | | ZIP entry size (decompressed) | 64 MiB | `max_archive_entry_size(u64)` | While decompressing | | ZIP total size (decompressed) | 256 MiB | `max_archive_total_size(u64)` | While decompressing | | Spreadsheet grid (XLS/ODS) | 4 000 000 cells | `max_spreadsheet_cells(u64)` | Pre-scan before calamine | | Rows / columns per sheet | 10 000 / 256 | `max_sheet_rows`, `max_sheet_cols` | While collecting cells (truncate + warn) | -| Output size | Unbounded | N/A | Proportional to input size | ## Malicious Input Categories @@ -291,7 +313,7 @@ process, or smuggling content past a human reviewer. | JavaScript URLs | `` | URL preserved as-is in markdown (no execution context); never rewritten by base resolution | | Data URI payloads | `` | Rendered as markdown image (no execution) | | Style expressions | `style="background:url(javascript:...)"` | Attribute not read | -| Deep nesting | 1000+ nested `
      ` | Flattened to text at depth 200 | +| Deep nesting | 1000+ nested `
      ` | Flattened to text at `max_nesting_depth` (256) | | Huge text nodes | 100KB+ single text node | Passed through (bounded by input limit) | | Null bytes | `\0` in content | Replaced with U+FFFD by html5ever | | Unclosed tags | `

      text

      more` | html5ever auto-closes per spec | @@ -321,6 +343,9 @@ process, or smuggling content past a human reviewer. | S11 | Unsupported formats fail explicitly (no empty success) | `document_detection::pdf_is_explicitly_unsupported_not_empty` | | S12 | Default `extract_url` never connects to a non-public address | `url_security::*_blocked*`, `fetch::tests::non_public_addresses_forbidden` | | S13 | Table spans never grow the render grid past `MAX_TABLE_GRID_CELLS`; footnote labels cannot leave `[^...]` | `html_security::hostile_colspan_rowspan_bounded`, `html_integration::footnote_label_cannot_break_out` | +| S14 | Returned Markdown never exceeds `max_output_size` | `limits::*output_cap*` | +| S15 | No configuration of `max_nesting_depth` can overflow the stack | `limits::*hard_ceiling_survives_small_stack` | +| S16 | Extraction stops with `ResourceLimit` once `timeout` passes (checked at least after each non-interruptible third-party step) | `limits::*zero_timeout_rejects`, `limits::tests::*` | ## See also