diff --git a/CHANGELOG.md b/CHANGELOG.md index f4a3194..90229c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -142,6 +142,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking:** `Error::ResourceLimit` carries a `LimitExceeded { kind: LimitKind, detail }` instead of + a `String`. `LimitKind` names the `Limits` field (`MaxInputSize`, `Timeout`, `MaxArchiveEntrySize`, + ...); `Error::limit_kind()` returns it. `ResourceLimit(_)` patterns and the error message are unchanged. - DOCX uses far less memory: body blocks are converted while `document.xml` is parsed and freed right after, and the XML tree is compact (interned names and values, exact-size lists). A 1.5 MB `document.xml` peaks at 3.9 MiB instead of 18.5 MiB, with half the allocations. Markdown @@ -160,6 +163,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). no HTTP/TLS stack. Library users calling `extract_url` add `features = ["http_client"]`. The CLI and the Python/Node packages still enable it, so their `extract_url` is unchanged. - **Breaking:** `Error` is `#[non_exhaustive]` (new `Error::Transform`); matches need a `_` arm. +- **Breaking:** `Extraction::warnings` and `ParsedDocument::warnings` are `Vec` + (`code: WarningCode` + `detail: String`) instead of `"code: detail"` strings, so callers can match + on the code. `Warning`'s `Display` keeps the old string form; JSON serializes + `{"code": "truncated_sheet", "detail": "..."}`. The agent tool output is unchanged. + Node and WASM `extractDocument` return `warnings` as `{ code, detail }` objects too. +- `Markkit` implements `Clone` and `Debug`. - **Breaking (IR API):** `Node`, `Document`, `ListItem` and the new IR structs are `#[non_exhaustive]`; matches need a `_` arm and structs are built with constructors. `Node::Table` cells are now `TableCell` (`children`, `colspan`, `rowspan`) with an `alignments` field. Block variants gain a diff --git a/README.md b/README.md index f4ec409..180df6c 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,8 @@ println!("{:?} {:?} {:?}", doc.format, doc.title, doc.warnings); println!("{}", doc.markdown); ``` -`Extraction` carries `markdown`, `title`, `format`, `warnings` (e.g. `truncated_sheet: …`), +`Extraction` carries `markdown`, `title`, `format`, `warnings` (structured: `code` such as +`WarningCode::TruncatedSheet` plus a human-readable `detail`), `stats` (sheets/rows/paragraphs/tables) and, for HTML, `metadata` (description, author, published date, language, canonical URL, site name, lead image from ``, Open Graph and JSON-LD). `extract_bytes` remains as a Markdown-only shortcut. @@ -218,7 +219,8 @@ markkit = { version = "0.1", features = ["http_client"] } ### Resource limits Every resource an input can drive is bounded by `markkit::Limits`. Defaults are -generous; violations return `Error::ResourceLimit` naming the limit. +generous; violations return `Error::ResourceLimit`, whose `LimitKind` (`err.limit_kind()`) +names the field that was exceeded. | Limit | Default | Bounds | |-------|---------|--------| diff --git a/crates/markkit-js/README.md b/crates/markkit-js/README.md index 3789f8b..3014518 100644 --- a/crates/markkit-js/README.md +++ b/crates/markkit-js/README.md @@ -28,7 +28,7 @@ await extractHtml(html, { includeSelectors: ['article'], excludeSelectors: ['.ad // URLs: non-public addresses are blocked by default. await extractUrl('https://docs.example.com/page', { allowedHosts: ['docs.example.com'], maxRedirects: 2 }); -// Documents with metadata (HTML, DOCX, XLSX/XLS/ODS). +// Documents with metadata (HTML, DOCX, PPTX, XLSX/XLSB/XLS/ODS). `warnings` are `{ code, detail }`. const doc = extractDocumentSync(fs.readFileSync('report.docx'), { fileName: 'report.docx' }, { maxSheetRows: 500 }); console.log(doc.title, doc.format, doc.warnings, doc.markdown); ``` diff --git a/crates/markkit-js/src/lib.rs b/crates/markkit-js/src/lib.rs index 355ff29..dfa3692 100644 --- a/crates/markkit-js/src/lib.rs +++ b/crates/markkit-js/src/lib.rs @@ -88,14 +88,23 @@ pub struct Metadata { pub image: Option, } +/// Structured extraction warning. +#[napi(object)] +pub struct Warning { + /// Stable machine code (`truncated_sheet`, `hidden_slide_skipped`, ...). + pub code: String, + /// Human-readable detail. + pub detail: String, +} + /// Markdown plus metadata. #[napi(object)] pub struct Extraction { pub markdown: String, pub title: Option, - /// Detected format (`html`, `docx`, `xlsx`, `xls`, `ods`, `text`, `markdown`, `csv`, `tsv`, `ipynb`). + /// Detected format (`html`, `docx`, `pptx`, `xlsx`, `xlsb`, `xls`, `ods`, `text`, `markdown`, `csv`, `tsv`, `ipynb`). pub format: String, - pub warnings: Vec, + pub warnings: Vec, pub stats: ExtractionStats, pub metadata: Metadata, } @@ -222,7 +231,14 @@ fn to_js(e: markkit::Extraction) -> Extraction { markdown: e.markdown, title: e.title, format: e.format.as_str().to_string(), - warnings: e.warnings, + warnings: e + .warnings + .into_iter() + .map(|w| Warning { + code: w.code.as_str().to_string(), + detail: w.detail, + }) + .collect(), stats: ExtractionStats { pages: count(e.stats.pages), sheets: count(e.stats.sheets), diff --git a/crates/markkit-wasm/js/index.d.ts b/crates/markkit-wasm/js/index.d.ts index b465432..8e69612 100644 --- a/crates/markkit-wasm/js/index.d.ts +++ b/crates/markkit-wasm/js/index.d.ts @@ -61,11 +61,17 @@ export interface Metadata { image: string | null; } +export interface Warning { + /** Stable machine code (`truncated_sheet`, `hidden_slide_skipped`, ...). */ + code: string; + detail: string; +} + export interface Extraction { markdown: string; title: string | null; format: string; - warnings: string[]; + warnings: Warning[]; stats: ExtractionStats; metadata: Metadata; } diff --git a/crates/markkit/src/archive.rs b/crates/markkit/src/archive.rs index 10616fb..58aac8f 100644 --- a/crates/markkit/src/archive.rs +++ b/crates/markkit/src/archive.rs @@ -12,12 +12,13 @@ use zip::ZipArchive; use zip::result::ZipError; use crate::config::MarkkitBuilder; +use crate::limits::{LimitExceeded, LimitKind}; /// Archive failure, split so callers can map limit violations to /// [`crate::Error::ResourceLimit`] and everything else to a format error. #[derive(Debug)] pub(crate) enum ArchiveError { - Limit(String), + Limit(LimitExceeded), Invalid(String), } @@ -42,10 +43,13 @@ impl<'a> ZipGuard<'a> { pub fn open(bytes: &'a [u8], config: &MarkkitBuilder) -> Result { let archive = ZipArchive::new(Cursor::new(bytes))?; if archive.len() > config.limits.max_archive_entries { - return Err(ArchiveError::Limit(format!( - "archive has {} entries, limit is {}", - archive.len(), - config.limits.max_archive_entries + return Err(ArchiveError::Limit(LimitExceeded::new( + LimitKind::MaxArchiveEntries, + format!( + "archive has {} entries, limit is {}", + archive.len(), + config.limits.max_archive_entries + ), ))); } let names = archive @@ -115,9 +119,9 @@ impl<'a> ZipGuard<'a> { fn add_total(&mut self, n: u64) -> Result<(), ArchiveError> { self.total = self.total.saturating_add(n); if self.total > self.max_total { - return Err(ArchiveError::Limit(format!( - "archive decompresses to more than {} bytes", - self.max_total + return Err(ArchiveError::Limit(LimitExceeded::new( + LimitKind::MaxArchiveTotalSize, + format!("archive decompresses to more than {} bytes", self.max_total), ))); } Ok(()) @@ -136,8 +140,9 @@ fn copy_capped( let n = std::io::copy(&mut reader.take(cap.saturating_add(1)), out) .map_err(|e| ArchiveError::Invalid(format!("cannot read entry '{name}': {e}")))?; if n > cap { - return Err(ArchiveError::Limit(format!( - "entry '{name}' decompresses to more than {cap} bytes" + return Err(ArchiveError::Limit(LimitExceeded::new( + LimitKind::MaxArchiveEntrySize, + format!("entry '{name}' decompresses to more than {cap} bytes"), ))); } Ok(n) diff --git a/crates/markkit/src/csv.rs b/crates/markkit/src/csv.rs index 4c648bc..53dc11d 100644 --- a/crates/markkit/src/csv.rs +++ b/crates/markkit/src/csv.rs @@ -22,7 +22,7 @@ use crate::limits::Budget; /// Parsed CSV/TSV document. pub(crate) struct CsvOutput { pub doc: Document, - pub warnings: Vec, + pub warnings: Vec, /// Data rows (header excluded). pub rows: usize, /// 1 when a table was produced, else 0. @@ -59,16 +59,21 @@ pub(crate) fn parse( records.push(rec); } if reader.unterminated { - warnings.push("malformed_csv: unterminated quoted field".into()); + warnings.push(crate::extract::Warning::new( + crate::extract::WarningCode::MalformedCsv, + "unterminated quoted field", + )); } if truncated_rows { - warnings.push(format!( - "truncated_table: table truncated to {max_rows} rows" + warnings.push(crate::extract::Warning::new( + crate::extract::WarningCode::TruncatedTable, + format!("table truncated to {max_rows} rows"), )); } if reader.truncated_cols { - warnings.push(format!( - "truncated_table: table truncated to {max_cols} columns" + warnings.push(crate::extract::Warning::new( + crate::extract::WarningCode::TruncatedTable, + format!("table truncated to {max_cols} columns"), )); } diff --git a/crates/markkit/src/docx.rs b/crates/markkit/src/docx.rs index 9a56220..f95ae48 100644 --- a/crates/markkit/src/docx.rs +++ b/crates/markkit/src/docx.rs @@ -26,7 +26,7 @@ use crate::archive::{ArchiveError, ZipGuard}; use crate::config::MarkkitBuilder; use crate::error::Error; use crate::ir::{Document, Node, Source, TableCell}; -use crate::limits::Budget; +use crate::limits::{Budget, LimitExceeded}; use crate::ooxml::{ Fmt, ListEntry, Relationship, Seg, build_inline, build_list, is_blank, parse_rels, plain_text, rel_target_by_type, resolve_target, split_dir, table_cell, @@ -36,7 +36,7 @@ use crate::xml::{self, Element, XmlError}; /// Result of parsing a DOCX package. pub(crate) struct DocxOutput { pub doc: Document, - pub warnings: Vec, + pub warnings: Vec, pub title: Option, pub paragraphs: usize, pub tables: usize, @@ -53,7 +53,9 @@ fn archive_err(e: ArchiveError) -> Error { fn xml_err(part: &str, e: XmlError) -> Error { match e { - XmlError::Limit(m) => Error::ResourceLimit(format!("{part}: {m}")), + XmlError::Limit(m) => { + Error::ResourceLimit(LimitExceeded::new(m.kind, format!("{part}: {}", m.detail))) + } XmlError::Invalid(m) => Error::DocxExtract(format!("{part}: {m}")), } } @@ -693,8 +695,9 @@ pub(crate) fn parse( doc: Document::new(body.blocks), warnings: match ctx.hidden_runs.get() { 0 => Vec::new(), - n => vec![format!( - "hidden_content_skipped: {n} hidden text run(s) skipped" + n => vec![crate::extract::Warning::new( + crate::extract::WarningCode::HiddenContentSkipped, + format!("{n} hidden text run(s) skipped"), )], }, }) diff --git a/crates/markkit/src/error.rs b/crates/markkit/src/error.rs index 681e3b2..417fe73 100644 --- a/crates/markkit/src/error.rs +++ b/crates/markkit/src/error.rs @@ -34,8 +34,10 @@ pub enum Error { #[error("Unsupported format: {0}")] UnsupportedFormat(String), + /// A [`Limits`](crate::Limits) value was exceeded; match on + /// [`LimitExceeded::kind`](crate::LimitExceeded) (or use [`Error::limit_kind`]). #[error("Resource limit exceeded: {0}")] - ResourceLimit(String), + ResourceLimit(crate::limits::LimitExceeded), /// A custom [`crate::registry::FormatParser`] failed or panicked. #[error("Parser error: {0}")] @@ -48,3 +50,13 @@ pub enum Error { #[error("I/O error: {0}")] Io(#[from] std::io::Error), } + +impl Error { + /// The exceeded limit, for [`Error::ResourceLimit`]. + pub fn limit_kind(&self) -> Option { + match self { + Error::ResourceLimit(e) => Some(e.kind), + _ => None, + } + } +} diff --git a/crates/markkit/src/extract.rs b/crates/markkit/src/extract.rs index 55e158b..20423d5 100644 --- a/crates/markkit/src/extract.rs +++ b/crates/markkit/src/extract.rs @@ -20,17 +20,103 @@ pub struct Extraction { pub title: Option, /// Format the document was parsed as. pub format: Format, - /// Non-fatal issues, each prefixed with a stable code - /// (`truncated_sheet:`, `hidden_sheet_skipped:`, `sheet_error:`, - /// `truncated_pdf:`, `pdf_unmapped_font:`, `pdf_no_text:`, `invalid_utf8:`, - /// `malformed_csv:`, `truncated_table:`). - pub warnings: Vec, + /// Non-fatal issues, each with a stable [`WarningCode`]. + pub warnings: Vec, /// Format-specific counters. pub stats: ExtractionStats, /// Document metadata (HTML ``, Open Graph, JSON-LD); empty when unknown. pub metadata: Metadata, } +/// Stable code of a [`Warning`]. Serialized (and displayed) in `snake_case`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize)] +#[serde(rename_all = "snake_case")] +#[non_exhaustive] +pub enum WarningCode { + /// A sheet exceeded `max_sheet_rows` / `max_sheet_cols` and was cut. + TruncatedSheet, + /// A hidden sheet was skipped (`include_hidden_sheets` is off). + HiddenSheetSkipped, + /// A sheet could not be read; the other sheets still render. + SheetError, + /// A hidden slide was skipped (`include_hidden_content` is off). + HiddenSlideSkipped, + /// A slide could not be read; the other slides still render. + SlideError, + /// Hidden HTML elements or DOCX runs were skipped (`include_hidden_content` is off). + HiddenContentSkipped, + /// PDF pages past the page cap were not extracted. + TruncatedPdf, + /// A PDF font has no Unicode mapping; its text may be garbled or missing. + PdfUnmappedFont, + /// A PDF has no extractable text layer (scanned images, no OCR). + PdfNoText, + /// Input was not valid UTF-8 and was decoded lossily. + InvalidUtf8, + /// A CSV/TSV row was malformed (ragged or unterminated quote). + MalformedCsv, + /// A CSV/TSV table exceeded the row/column caps and was cut. + TruncatedTable, + /// Output was cut to a caller's size limit (agent tool `max_chars`). + Truncated, + /// Any other issue, e.g. from a custom [`crate::registry::FormatParser`]. + Other, +} + +impl WarningCode { + /// The `snake_case` code, e.g. `"truncated_sheet"`. + pub fn as_str(&self) -> &'static str { + match self { + WarningCode::TruncatedSheet => "truncated_sheet", + WarningCode::HiddenSheetSkipped => "hidden_sheet_skipped", + WarningCode::SheetError => "sheet_error", + WarningCode::HiddenSlideSkipped => "hidden_slide_skipped", + WarningCode::SlideError => "slide_error", + WarningCode::HiddenContentSkipped => "hidden_content_skipped", + WarningCode::TruncatedPdf => "truncated_pdf", + WarningCode::PdfUnmappedFont => "pdf_unmapped_font", + WarningCode::PdfNoText => "pdf_no_text", + WarningCode::InvalidUtf8 => "invalid_utf8", + WarningCode::MalformedCsv => "malformed_csv", + WarningCode::TruncatedTable => "truncated_table", + WarningCode::Truncated => "truncated", + WarningCode::Other => "other", + } + } +} + +impl std::fmt::Display for WarningCode { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.as_str()) + } +} + +/// A non-fatal issue found while converting a document. +/// +/// Match on [`Warning::code`]; `detail` is human-readable and may change. +/// `Display` renders `": "`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[non_exhaustive] +pub struct Warning { + pub code: WarningCode, + pub detail: String, +} + +impl Warning { + pub fn new(code: WarningCode, detail: impl Into) -> Self { + Self { + code, + detail: detail.into(), + } + } +} + +impl std::fmt::Display for Warning { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}: {}", self.code, self.detail) + } +} + /// Document-level metadata. Every field is `None` when absent. #[derive(Debug, Clone, Default, PartialEq, Eq, Serialize)] #[non_exhaustive] @@ -88,7 +174,7 @@ pub struct ParsedDocument { /// Format the document was parsed as. pub format: Format, /// Non-fatal issues (see [`Extraction::warnings`]). - pub warnings: Vec, + pub warnings: Vec, /// Format-specific counters. pub stats: ExtractionStats, /// Document metadata (see [`Extraction::metadata`]). @@ -154,7 +240,10 @@ fn guarded( /// Core extraction engine. /// -/// Use [`MarkkitBuilder`] to create a configured instance. +/// Use [`MarkkitBuilder`] to create a configured instance. Holds only its +/// configuration: cloning is cheap enough to give each task its own copy, and +/// one instance can be shared across threads (`Send + Sync`). +#[derive(Debug, Clone)] pub struct Markkit { config: MarkkitBuilder, } @@ -558,6 +647,45 @@ mod tests { assert!(m.config().preserve_links); } + #[test] + fn markkit_is_clone_debug_send_sync() { + fn assert_traits() {} + assert_traits::(); + let m = Markkit::builder().max_sheet_rows(7).build(); + let c = m.clone(); + assert_eq!(c.config().limits.max_sheet_rows, 7); + assert!(format!("{m:?}").contains("max_sheet_rows: 7")); + } + + #[test] + fn warnings_display_and_serialize_structured() { + let w = Warning::new( + WarningCode::TruncatedSheet, + "sheet 'S' truncated to 10 rows", + ); + assert_eq!( + w.to_string(), + "truncated_sheet: sheet 'S' truncated to 10 rows" + ); + assert_eq!( + serde_json::to_value(&w).unwrap(), + serde_json::json!({"code": "truncated_sheet", "detail": "sheet 'S' truncated to 10 rows"}) + ); + for code in [ + WarningCode::TruncatedSheet, + WarningCode::HiddenSheetSkipped, + WarningCode::SheetError, + WarningCode::HiddenSlideSkipped, + WarningCode::SlideError, + WarningCode::Truncated, + ] { + assert_eq!( + serde_json::to_value(code).unwrap(), + serde_json::Value::String(code.as_str().into()) + ); + } + } + #[tokio::test] async fn test_resource_limit() { let m = Markkit::builder().max_input_size(10).build(); diff --git a/crates/markkit/src/fetch.rs b/crates/markkit/src/fetch.rs index 57ad41a..05292d4 100644 --- a/crates/markkit/src/fetch.rs +++ b/crates/markkit/src/fetch.rs @@ -26,6 +26,7 @@ use reqwest::dns::{Addrs, Name, Resolve, Resolving}; use crate::config::MarkkitBuilder; use crate::error::Error; +use crate::limits::{LimitExceeded, LimitKind}; /// Fetched body plus its declared content type. pub(crate) struct Fetched { @@ -56,9 +57,11 @@ pub(crate) async fn fetch(url: &str, cfg: &MarkkitBuilder) -> crate::Result limit as u64 { - return Ok(Err(Error::ResourceLimit(format!( - "response Content-Length {len} exceeds max_input_size {limit}" - )))); + return Ok(Err(LimitExceeded::new( + LimitKind::MaxInputSize, + format!("response Content-Length {len} exceeds max_input_size {limit}"), + ) + .into())); } let content_type = response .headers() @@ -69,9 +72,11 @@ pub(crate) async fn fetch(url: &str, cfg: &MarkkitBuilder) -> crate::Result limit { - return Ok(Err(Error::ResourceLimit(format!( - "response body exceeds max_input_size {limit}" - )))); + return Ok(Err(LimitExceeded::new( + LimitKind::MaxInputSize, + format!("response body exceeds max_input_size {limit}"), + ) + .into())); } bytes.extend_from_slice(&chunk); } @@ -90,10 +95,11 @@ pub(crate) async fn fetch(url: &str, cfg: &MarkkitBuilder) -> crate::Result &'static str { + match self { + LimitKind::MaxInputSize => "max_input_size", + LimitKind::MaxOutputSize => "max_output_size", + LimitKind::Timeout => "timeout", + LimitKind::MaxNestingDepth => "max_nesting_depth", + LimitKind::MaxNodes => "max_nodes", + LimitKind::MaxArchiveEntries => "max_archive_entries", + LimitKind::MaxArchiveEntrySize => "max_archive_entry_size", + LimitKind::MaxArchiveTotalSize => "max_archive_total_size", + LimitKind::MaxSpreadsheetCells => "max_spreadsheet_cells", + LimitKind::MaxPdfStreamSize => "max_pdf_stream_size", + LimitKind::MaxPdfDecodedSize => "max_pdf_decoded_size", + } + } +} + +impl std::fmt::Display for LimitKind { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.as_str()) + } +} + +/// Payload of [`Error::ResourceLimit`]: the limit hit plus a human-readable +/// detail. Match on `kind`; `detail` may change. `Display` renders `detail`. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct LimitExceeded { + pub kind: LimitKind, + pub detail: String, +} + +impl LimitExceeded { + pub fn new(kind: LimitKind, detail: impl Into) -> Self { + Self { + kind, + detail: detail.into(), + } + } +} + +impl std::fmt::Display for LimitExceeded { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(&self.detail) + } +} + +impl From for Error { + fn from(e: LimitExceeded) -> Self { + Error::ResourceLimit(e) + } +} + // THREAT[T2b]: recursion depth must stay safe on small host stacks whatever // the embedder configures. Measured worst case (nested blockquotes, renderer // recursion): 768 levels fit a 2 MiB thread in release, 1024 do not; 512 keeps @@ -189,10 +265,14 @@ impl Limits { /// Reject inputs larger than `max_input_size`. pub(crate) fn check_input(&self, len: usize) -> crate::Result<()> { if len > self.max_input_size { - return Err(Error::ResourceLimit(format!( - "input size {len} exceeds max_input_size {}", - self.max_input_size - ))); + return Err(LimitExceeded::new( + LimitKind::MaxInputSize, + format!( + "input size {len} exceeds max_input_size {}", + self.max_input_size + ), + ) + .into()); } Ok(()) } @@ -285,7 +365,11 @@ impl Budget { } pub fn timeout_error(&self) -> Error { - Error::ResourceLimit(format!("processing exceeded timeout of {:?}", self.timeout)) + LimitExceeded::new( + LimitKind::Timeout, + format!("processing exceeded timeout of {:?}", self.timeout), + ) + .into() } /// Charge `n` parsed nodes against `max_nodes`. @@ -293,10 +377,11 @@ impl Budget { let total = self.nodes.get().saturating_add(n); self.nodes.set(total); if total > self.max_nodes { - return Err(Error::ResourceLimit(format!( - "document has more than max_nodes {} nodes", - self.max_nodes - ))); + return Err(LimitExceeded::new( + LimitKind::MaxNodes, + format!("document has more than max_nodes {} nodes", self.max_nodes), + ) + .into()); } Ok(()) } diff --git a/crates/markkit/src/pdf/guard.rs b/crates/markkit/src/pdf/guard.rs index 012d83a..09dc22f 100644 --- a/crates/markkit/src/pdf/guard.rs +++ b/crates/markkit/src/pdf/guard.rs @@ -19,6 +19,8 @@ use hayro_syntax::Filter; +use crate::limits::{LimitExceeded, LimitKind}; + /// Raised when a stream would decode past a limit. #[derive(Debug)] pub(crate) struct Limit(pub String); @@ -118,11 +120,15 @@ pub(crate) fn prescan( let budget = stream_cap.min(total_cap.saturating_sub(total)); let out = decode(&chain, &bytes[data_start..end], budget).map_err(|_| { if budget < stream_cap { - PrescanError::Limit(format!( - "PDF streams decode to more than {total_cap} bytes in total" + PrescanError::Limit(LimitExceeded::new( + LimitKind::MaxPdfDecodedSize, + format!("PDF streams decode to more than {total_cap} bytes in total (max_pdf_decoded_size)"), )) } else { - PrescanError::Limit(over(stream_cap).0) + PrescanError::Limit(LimitExceeded::new( + LimitKind::MaxPdfStreamSize, + format!("{} (max_pdf_stream_size)", over(stream_cap).0), + )) } })?; total += out.len(); @@ -134,7 +140,7 @@ pub(crate) fn prescan( pub(crate) enum PrescanError { Encrypted, Unverifiable, - Limit(String), + Limit(LimitExceeded), } fn find(hay: &[u8], needle: &[u8]) -> Option { diff --git a/crates/markkit/src/pdf/mod.rs b/crates/markkit/src/pdf/mod.rs index 26c94e3..1263f82 100644 --- a/crates/markkit/src/pdf/mod.rs +++ b/crates/markkit/src/pdf/mod.rs @@ -51,7 +51,7 @@ pub(crate) struct PdfOutput { pub doc: Document, pub title: Option, pub pages: usize, - pub warnings: Vec, + pub warnings: Vec, } pub(crate) fn parse( @@ -120,24 +120,30 @@ pub(crate) fn parse( page_lines.push(build_lines(std::mem::take(&mut interp.spans))); } if pages.len() > cfg.limits.max_pdf_pages { - warnings.push(format!( - "truncated_pdf: rendered {} of {} pages (max_pdf_pages)", - cfg.limits.max_pdf_pages, - pages.len() + warnings.push(crate::extract::Warning::new( + crate::extract::WarningCode::TruncatedPdf, + format!( + "rendered {} of {} pages (max_pdf_pages)", + cfg.limits.max_pdf_pages, + pages.len() + ), )); } let mut unmapped: Vec<&String> = interp.unmapped.iter().collect(); unmapped.sort(); for name in unmapped { - warnings.push(format!( - "pdf_unmapped_font: text in font {name} has no Unicode mapping and was skipped" + warnings.push(crate::extract::Warning::new( + crate::extract::WarningCode::PdfUnmappedFont, + format!("text in font {name} has no Unicode mapping and was skipped"), )); } let nodes = build_nodes(page_lines, cfg.track_sources); if nodes.is_empty() && !pages.is_empty() { - warnings - .push("pdf_no_text: no text layer found (scanned PDF? OCR is not supported)".into()); + warnings.push(crate::extract::Warning::new( + crate::extract::WarningCode::PdfNoText, + "no text layer found (scanned PDF? OCR is not supported)", + )); } Ok(PdfOutput { doc: Document::new(nodes), @@ -252,7 +258,11 @@ struct Interpreter<'c> { } fn limit() -> Error { - Error::ResourceLimit("PDF content exceeds max_pdf_decoded_size".into()) + crate::limits::LimitExceeded::new( + crate::limits::LimitKind::MaxPdfDecodedSize, + "PDF content exceeds max_pdf_decoded_size", + ) + .into() } fn num(o: Option<&Object<'_>>) -> f32 { diff --git a/crates/markkit/src/pptx.rs b/crates/markkit/src/pptx.rs index 8a4d7cb..f3c5977 100644 --- a/crates/markkit/src/pptx.rs +++ b/crates/markkit/src/pptx.rs @@ -25,6 +25,7 @@ use std::rc::Rc; use crate::archive::{ArchiveError, ZipGuard}; use crate::config::MarkkitBuilder; use crate::error::Error; +use crate::extract::{Warning, WarningCode}; use crate::ir::{Document, Node, Source, TableCell}; use crate::limits::Budget; use crate::ooxml::{ @@ -37,7 +38,7 @@ use crate::xml::{self, Element, XmlError}; pub(crate) struct PptxOutput { pub doc: Document, pub title: Option, - pub warnings: Vec, + pub warnings: Vec, pub slides: usize, pub tables: usize, } @@ -56,7 +57,10 @@ fn archive_err(e: ArchiveError) -> Error { fn xml_err(part: &str, e: XmlError) -> Error { match e { - XmlError::Limit(m) => Error::ResourceLimit(format!("{part}: {m}")), + XmlError::Limit(m) => Error::ResourceLimit(crate::limits::LimitExceeded::new( + m.kind, + format!("{part}: {}", m.detail), + )), XmlError::Invalid(m) => Error::PptxExtract(format!("{part}: {m}")), } } @@ -440,19 +444,31 @@ pub(crate) fn parse( let n = i + 1; budget.check_time()?; let Some(path) = internal_target(&rels, rid, "/slide", dir) else { - warnings.push(format!("slide_error: slide {n}: dangling relationship")); + warnings.push(Warning::new( + WarningCode::SlideError, + format!("slide {n}: dangling relationship"), + )); continue; }; let Some(xml) = zip.read_xml(&path)? else { - warnings.push(format!("slide_error: slide {n}: missing part {path}")); + warnings.push(Warning::new( + WarningCode::SlideError, + format!("slide {n}: missing part {path}"), + )); continue; }; let Some(sld) = xml.child("sld") else { - warnings.push(format!("slide_error: slide {n}: not a p:sld")); + warnings.push(Warning::new( + WarningCode::SlideError, + format!("slide {n}: not a p:sld"), + )); continue; }; if matches!(sld.attr("show"), Some("0" | "false")) && !config.include_hidden_content { - warnings.push(format!("hidden_slide_skipped: slide {n} is hidden")); + warnings.push(Warning::new( + WarningCode::HiddenSlideSkipped, + format!("slide {n} is hidden"), + )); continue; } let slide_rels = zip.part_rels(&path)?; diff --git a/crates/markkit/src/render/markdown.rs b/crates/markkit/src/render/markdown.rs index 44632e4..f28ef8e 100644 --- a/crates/markkit/src/render/markdown.rs +++ b/crates/markkit/src/render/markdown.rs @@ -719,7 +719,11 @@ impl Default for MarkdownRenderer { } fn output_error(max: usize) -> crate::Error { - crate::Error::ResourceLimit(format!("output exceeds max_output_size {max}")) + crate::limits::LimitExceeded::new( + crate::limits::LimitKind::MaxOutputSize, + format!("output exceeds max_output_size {max}"), + ) + .into() } /// Trim leading and trailing whitespace from inline content rendered since `start`. diff --git a/crates/markkit/src/spreadsheet.rs b/crates/markkit/src/spreadsheet.rs index 21fc03b..5bf8c2e 100644 --- a/crates/markkit/src/spreadsheet.rs +++ b/crates/markkit/src/spreadsheet.rs @@ -37,9 +37,10 @@ use quick_xml::events::{BytesStart, Event}; use crate::archive::{ArchiveError, ZipGuard}; use crate::config::MarkkitBuilder; use crate::error::Error; +use crate::extract::{Warning, WarningCode}; use crate::format::Format; use crate::ir::{Document, Node, Source, TableCell}; -use crate::limits::Budget; +use crate::limits::{Budget, LimitExceeded, LimitKind}; mod xlsb; @@ -47,7 +48,7 @@ pub(crate) struct SheetOutput { pub doc: Document, pub format: Format, pub title: Option, - pub warnings: Vec, + pub warnings: Vec, pub sheets: usize, pub rows: usize, } @@ -148,7 +149,7 @@ pub(crate) fn parse( budget.check_time()?; match res { Ok(()) => out.push_sheet(&name, grid), - Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), + Err(e) => out.warn(WarningCode::SheetError, format!("sheet '{name}': {e}")), } } out.finish(Format::Xlsx, title) @@ -202,7 +203,10 @@ fn parse_xlsb( continue; }; let Some(data) = zip.read(part).map_err(archive_err)? else { - out.warn(format!("sheet_error: sheet '{name}': missing part {part}")); + out.warn( + WarningCode::SheetError, + format!("sheet '{name}': missing part {part}"), + ); continue; }; let mut grid = Grid::new(config); @@ -219,7 +223,7 @@ fn parse_xlsb( budget.check_time()?; match res { Ok(()) => out.push_sheet(&name, grid), - Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), + Err(e) => out.warn(WarningCode::SheetError, format!("sheet '{name}': {e}")), } } out.finish(Format::Xlsb, title) @@ -256,7 +260,7 @@ where budget.check_time()?; out.push_sheet(&name, grid); } - Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), + Err(e) => out.warn(WarningCode::SheetError, format!("sheet '{name}': {e}")), } } Ok(()) @@ -485,7 +489,7 @@ impl Grid { struct Collector<'c> { config: &'c MarkkitBuilder, nodes: Vec, - warnings: Vec, + warnings: Vec, sheets: usize, rows: usize, } @@ -501,8 +505,8 @@ impl<'c> Collector<'c> { } } - fn warn(&mut self, w: String) { - self.warnings.push(w); + fn warn(&mut self, code: WarningCode, detail: String) { + self.warnings.push(Warning::new(code, detail)); } /// Sheet names in workbook order, flagged with whether to render them. @@ -514,10 +518,10 @@ impl<'c> Collector<'c> { } let hidden = s.visible != SheetVisible::Visible; if hidden && !self.config.include_hidden_sheets { - self.warn(format!( - "hidden_sheet_skipped: sheet '{}' is hidden", - s.name - )); + self.warn( + WarningCode::HiddenSheetSkipped, + format!("sheet '{}' is hidden", s.name), + ); out.push((s.name.clone(), false)); } else { out.push((s.name.clone(), true)); @@ -539,14 +543,16 @@ impl<'c> Collector<'c> { return; }; if trunc_rows { - self.warn(format!( - "truncated_sheet: sheet '{name}' truncated to {max_rows} rows" - )); + self.warn( + WarningCode::TruncatedSheet, + format!("sheet '{name}' truncated to {max_rows} rows"), + ); } if trunc_cols { - self.warn(format!( - "truncated_sheet: sheet '{name}' truncated to {max_cols} columns" - )); + self.warn( + WarningCode::TruncatedSheet, + format!("sheet '{name}' truncated to {max_cols} columns"), + ); } self.sheets += 1; self.rows += rows.len(); @@ -870,6 +876,10 @@ fn bof_is_biff8(data: &[u8]) -> bool { /// Scan the workbook stream and return a fresh OLE2 container holding only the /// scanned (and formula-neutralised) stream, which is what calamine reads. +fn cells_error(detail: String) -> Error { + LimitExceeded::new(LimitKind::MaxSpreadsheetCells, detail).into() +} + fn prescan_xls( comp: &mut cfb::CompoundFile, config: &MarkkitBuilder, @@ -886,9 +896,11 @@ fn prescan_xls( .read_to_end(&mut stream) .map_err(|e| err(format!("cannot read Workbook stream: {e}")))?; if stream.len() as u64 > cap { - return Err(Error::ResourceLimit(format!( - "XLS Workbook stream exceeds {cap} bytes" - ))); + return Err(LimitExceeded::new( + LimitKind::MaxArchiveEntrySize, + format!("XLS Workbook stream exceeds {cap} bytes"), + ) + .into()); } // Neutralising rewrites bytes, which can reframe records of an overlapping // sheet; rescan until stable. Honest files settle on the second pass. @@ -932,8 +944,7 @@ fn scan_biff(stream: &[u8], max_cells: u64) -> Result, Error> { _ => {} } } - let limit = - |n: u64| Error::ResourceLimit(format!("XLS grid of {n} cells exceeds limit {max_cells}")); + let limit = |n: u64| cells_error(format!("XLS grid of {n} cells exceeds limit {max_cells}")); let mut total: u64 = 0; let mut edits = Vec::new(); for pos in offsets { @@ -1127,7 +1138,7 @@ fn prescan_ods(content: &[u8], max_cells: u64) -> Result<(), Error> { let mut total: u64 = 0; let check = |total: u64| { if total > max_cells { - Err(Error::ResourceLimit(format!( + Err(cells_error(format!( "ODS grid of at least {total} cells exceeds limit {max_cells}" ))) } else { diff --git a/crates/markkit/src/text.rs b/crates/markkit/src/text.rs index 1cd6f1d..419b148 100644 --- a/crates/markkit/src/text.rs +++ b/crates/markkit/src/text.rs @@ -19,7 +19,7 @@ use crate::limits::Budget; /// Decoded text plus non-fatal warnings. pub(crate) struct Decoded { pub text: String, - pub warnings: Vec, + pub warnings: Vec, } /// Decode bytes to normalized text (see module docs). @@ -40,17 +40,24 @@ pub(crate) fn decode(bytes: &[u8]) -> Decoded { } } -fn utf8(bytes: &[u8], warnings: &mut Vec) -> String { +fn utf8(bytes: &[u8], warnings: &mut Vec) -> String { match std::str::from_utf8(bytes) { Ok(s) => s.to_string(), Err(_) => { - warnings.push("invalid_utf8: invalid UTF-8 sequences replaced with U+FFFD".into()); + warnings.push(crate::extract::Warning::new( + crate::extract::WarningCode::InvalidUtf8, + "invalid UTF-8 sequences replaced with U+FFFD", + )); String::from_utf8_lossy(bytes).into_owned() } } } -fn utf16(bytes: &[u8], read: fn([u8; 2]) -> u16, warnings: &mut Vec) -> String { +fn utf16( + bytes: &[u8], + read: fn([u8; 2]) -> u16, + warnings: &mut Vec, +) -> String { let units = bytes.as_chunks::<2>().0.iter().map(|c| read(*c)); let mut lossy = !bytes.len().is_multiple_of(2); let s: String = char::decode_utf16(units) @@ -62,7 +69,10 @@ fn utf16(bytes: &[u8], read: fn([u8; 2]) -> u16, warnings: &mut Vec) -> }) .collect(); if lossy { - warnings.push("invalid_utf8: invalid UTF-16 sequences replaced with U+FFFD".into()); + warnings.push(crate::extract::Warning::new( + crate::extract::WarningCode::InvalidUtf8, + "invalid UTF-16 sequences replaced with U+FFFD", + )); } s } @@ -157,7 +167,7 @@ mod tests { fn decode_invalid_utf8_warns() { let d = decode(b"ok \xFF\xFE!"); assert_eq!(d.text, "ok \u{FFFD}\u{FFFD}!"); - assert!(d.warnings[0].starts_with("invalid_utf8:")); + assert_eq!(d.warnings[0].code, crate::extract::WarningCode::InvalidUtf8); } #[test] diff --git a/crates/markkit/src/tool.rs b/crates/markkit/src/tool.rs index 2b15e8f..0708259 100644 --- a/crates/markkit/src/tool.rs +++ b/crates/markkit/src/tool.rs @@ -62,7 +62,7 @@ use serde::{Deserialize, Serialize}; use crate::config::MarkkitBuilder; use crate::error::Error; -use crate::extract::{Extraction, Markkit}; +use crate::extract::{Extraction, Markkit, Warning, WarningCode}; use crate::format::DocumentHint; type ToolExecutionFuture = Pin> + Send>>; @@ -655,9 +655,9 @@ async fn run(inner: &ToolInner, req: Request) -> Result { .map_or(extraction.markdown.len(), |(i, _)| i); let full_chars = extraction.markdown.chars().count(); extraction.markdown.truncate(cut); - extraction.warnings.push(format!( - "truncated: output cut at {} of {full_chars} chars", - req.max_chars + extraction.warnings.push(Warning::new( + WarningCode::Truncated, + format!("output cut at {} of {full_chars} chars", req.max_chars), )); } Ok(extraction) @@ -695,7 +695,7 @@ fn output_from_extraction( let truncated = extraction .warnings .iter() - .any(|w| w.starts_with("truncated:")); + .any(|w| w.code == WarningCode::Truncated); let stats = serde_json::to_value(&extraction.stats) .map_err(|e| ToolError::Internal(format!("failed to serialize stats: {e}")))?; let chars = extraction.markdown.chars().count(); @@ -704,7 +704,8 @@ fn output_from_extraction( "markdown": extraction.markdown, "title": extraction.title, "format": extraction.format, - "warnings": extraction.warnings, + // Tool contract: warnings are `": "` strings. + "warnings": extraction.warnings.iter().map(ToString::to_string).collect::>(), "truncated": truncated, }), images: Vec::new(), @@ -763,7 +764,7 @@ fn output_schema() -> serde_json::Value { "properties": { "markdown": { "type": "string", "description": "Extracted Markdown" }, "title": { "type": ["string", "null"], "description": "Document title when known" }, - "format": { "type": "string", "description": "Detected format (html, docx, xlsx, xls, ods)" }, + "format": { "type": "string", "description": "Detected format (html, docx, pptx, xlsx, xlsb, xls, ods)" }, "warnings": { "type": "array", "items": { "type": "string" }, diff --git a/crates/markkit/src/xml.rs b/crates/markkit/src/xml.rs index 489389c..1965f22 100644 --- a/crates/markkit/src/xml.rs +++ b/crates/markkit/src/xml.rs @@ -23,11 +23,11 @@ use quick_xml::events::{BytesStart, Event}; use quick_xml::{Reader, XmlVersion}; use crate::error::Error; -use crate::limits::Budget; +use crate::limits::{Budget, LimitExceeded, LimitKind}; #[derive(Debug)] pub(crate) enum XmlError { - Limit(String), + Limit(LimitExceeded), Invalid(String), } @@ -310,8 +310,9 @@ fn build( match event { Event::Start(s) => { if b.open.len() >= max_depth { - return Err(XmlError::Limit(format!( - "XML nesting deeper than max_nesting_depth {max_depth}" + return Err(XmlError::Limit(LimitExceeded::new( + LimitKind::MaxNestingDepth, + format!("XML nesting deeper than max_nesting_depth {max_depth}"), ))); } let (name, attrs) = b.start(&s); @@ -442,7 +443,7 @@ mod tests { let budget = Budget::new(&Limits::default().max_nodes(50)); assert!(matches!( super::parse(many.as_bytes(), 256, &budget), - Err(XmlError::Limit(m)) if m.contains("max_nodes") + Err(XmlError::Limit(m)) if m.kind == LimitKind::MaxNodes )); } @@ -452,7 +453,7 @@ mod tests { let budget = Budget::new(&Limits::default().timeout(std::time::Duration::ZERO)); assert!(matches!( super::parse(many.as_bytes(), 256, &budget), - Err(XmlError::Limit(m)) if m.contains("timeout") + Err(XmlError::Limit(m)) if m.kind == LimitKind::Timeout )); } diff --git a/crates/markkit/tests/csv_integration.rs b/crates/markkit/tests/csv_integration.rs index 846da61..58b8739 100644 --- a/crates/markkit/tests/csv_integration.rs +++ b/crates/markkit/tests/csv_integration.rs @@ -97,7 +97,7 @@ fn security_row_cap_stops_reading() { assert_eq!(e.stats.rows, Some(3)); assert_eq!(e.markdown.lines().count(), 5); assert_eq!( - e.warnings, + warns(&e), vec!["truncated_table: table truncated to 3 rows"] ); } @@ -113,7 +113,7 @@ fn security_column_cap_never_stores_extra_fields() { ); assert_eq!(e.markdown, "| a | |\n| --- | --- |\n"); assert_eq!( - e.warnings, + warns(&e), vec!["truncated_table: table truncated to 2 columns"] ); } @@ -135,7 +135,7 @@ fn security_unterminated_quote_warns() { let input = format!("a,\"{}", "x\n".repeat(100_000)); let e = extract(input.as_bytes(), DocumentHint::file_name("a.csv")); assert_eq!(e.markdown.lines().count(), 2); - assert_eq!(e.warnings, vec!["malformed_csv: unterminated quoted field"]); + assert_eq!(warns(&e), vec!["malformed_csv: unterminated quoted field"]); } #[test] @@ -154,7 +154,7 @@ fn security_control_chars_and_invalid_utf8() { DocumentHint::file_name("a.tsv"), ); assert_eq!(e.markdown, "| h\\[2J |\n| --- |\n| \u{FFFD}v |\n"); - assert!(e.warnings[0].starts_with("invalid_utf8:")); + assert_eq!(e.warnings[0].code, markkit::WarningCode::InvalidUtf8); } #[test] @@ -202,7 +202,11 @@ fn security_timeout_applies() { .build(); let r = m.extract_document_sync(b"a,b\n1,2\n", DocumentHint::file_name("a.csv")); assert!( - matches!(&r, Err(markkit::Error::ResourceLimit(msg)) if msg.contains("timeout")), + matches!(&r, Err(e) if e.limit_kind() == Some(markkit::LimitKind::Timeout)), "{r:?}" ); } + +fn warns(e: &markkit::Extraction) -> Vec { + e.warnings.iter().map(ToString::to_string).collect() +} diff --git a/crates/markkit/tests/document_security.rs b/crates/markkit/tests/document_security.rs index 5f5c279..6d42f0c 100644 --- a/crates/markkit/tests/document_security.rs +++ b/crates/markkit/tests/document_security.rs @@ -6,7 +6,7 @@ mod common; use common::*; -use markkit::{DocumentHint, Error, Markkit}; +use markkit::{DocumentHint, Error, LimitKind, Markkit, WarningCode}; use proptest::prelude::*; const DOCX_CT: &str = "application/vnd.openxmlformats-officedocument.wordprocessingml.document"; @@ -47,7 +47,11 @@ fn docx_zip_bomb_entry_rejected() { .max_archive_entry_size(1024 * 1024) .build(); let err = run(&m, &bytes, DOCX_CT).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert_eq!( + err.limit_kind(), + Some(LimitKind::MaxArchiveEntrySize), + "{err:?}" + ); } #[cfg(feature = "docx")] @@ -66,7 +70,11 @@ fn docx_total_decompressed_size_capped() { .max_archive_total_size(1024 * 1024) .build(); let err = run(&m, &bytes, DOCX_CT).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert_eq!( + err.limit_kind(), + Some(LimitKind::MaxArchiveTotalSize), + "{err:?}" + ); } #[test] @@ -86,7 +94,11 @@ fn archive_entry_count_capped() { } for ct in types { let err = run(&m, &bytes, ct).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{ct}: {err:?}"); + assert_eq!( + err.limit_kind(), + Some(LimitKind::MaxArchiveEntries), + "{ct}: {err:?}" + ); } } @@ -105,7 +117,11 @@ fn xlsx_zip_bomb_sheet_rejected() { .max_archive_entry_size(1024 * 1024) .build(); let err = run(&m, &bytes, XLSX_CT).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert_eq!( + err.limit_kind(), + Some(LimitKind::MaxArchiveEntrySize), + "{err:?}" + ); } #[cfg(feature = "spreadsheet")] @@ -156,7 +172,11 @@ fn xlsb_huge_declared_dimensions_do_not_allocate() { .unwrap(); assert!(e.markdown.contains("far")); assert!(e.markdown.len() < 64 * 1024); - assert!(e.warnings.iter().any(|w| w.starts_with("truncated_sheet:"))); + assert!( + e.warnings + .iter() + .any(|w| w.code == WarningCode::TruncatedSheet) + ); } #[cfg(feature = "spreadsheet")] @@ -173,7 +193,11 @@ fn xlsx_bomb_in_unreferenced_entry_rejected() { .max_archive_entry_size(1024 * 1024) .build(); let err = run(&m, &bytes, XLSX_CT).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert_eq!( + err.limit_kind(), + Some(LimitKind::MaxArchiveEntrySize), + "{err:?}" + ); } #[test] @@ -182,7 +206,11 @@ fn max_input_size_honoured_for_office_documents() { let m = Markkit::builder().max_input_size(bytes.len() - 1).build(); for hint in hints() { let err = m.extract_document_sync(&bytes, hint).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{hint:?}: {err:?}"); + assert_eq!( + err.limit_kind(), + Some(LimitKind::MaxInputSize), + "{hint:?}: {err:?}" + ); } } @@ -200,7 +228,11 @@ fn xls_huge_dimensions_rejected() { let err = m .extract_document_sync(&bytes, DocumentHint::default()) .unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert_eq!( + err.limit_kind(), + Some(LimitKind::MaxSpreadsheetCells), + "{err:?}" + ); } #[cfg(feature = "spreadsheet")] @@ -215,7 +247,11 @@ fn xls_far_apart_cells_rejected() { let err = m .extract_document_sync(&bytes, DocumentHint::default()) .unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert_eq!( + err.limit_kind(), + Some(LimitKind::MaxSpreadsheetCells), + "{err:?}" + ); } #[cfg(feature = "spreadsheet")] @@ -446,7 +482,11 @@ fn ods_repeated_rows_and_columns_rejected() { let err = m .extract_document_sync(&bytes, DocumentHint::default()) .unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert_eq!( + err.limit_kind(), + Some(LimitKind::MaxSpreadsheetCells), + "{err:?}" + ); } // ─── XML tricks ───────────────────────────────────────────────────── diff --git a/crates/markkit/tests/hidden_content.rs b/crates/markkit/tests/hidden_content.rs index 4518468..529548e 100644 --- a/crates/markkit/tests/hidden_content.rs +++ b/crates/markkit/tests/hidden_content.rs @@ -102,7 +102,7 @@ mod docx { assert!( e.warnings .iter() - .any(|w| w.starts_with("hidden_content_skipped")), + .any(|w| w.code == markkit::WarningCode::HiddenContentSkipped), "{:?}", e.warnings ); diff --git a/crates/markkit/tests/limits.rs b/crates/markkit/tests/limits.rs index eb1239b..742e396 100644 --- a/crates/markkit/tests/limits.rs +++ b/crates/markkit/tests/limits.rs @@ -8,16 +8,24 @@ mod common; use std::time::Duration; -use markkit::{DocumentHint, Error, Limits, Markkit}; +use markkit::{DocumentHint, Error, LimitKind, Limits, Markkit}; #[allow(dead_code)] const DOCX_CT: &str = "application/vnd.openxmlformats-officedocument.wordprocessingml.document"; #[allow(dead_code)] const XLSX_CT: &str = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; -fn limit_msg(r: markkit::Result) -> String { +/// The exceeded limit; also checks the message names it and `limit_kind` agrees. +fn limit(r: markkit::Result) -> LimitKind { match r { - Err(Error::ResourceLimit(m)) => m, + Err(e @ Error::ResourceLimit(_)) => { + let Error::ResourceLimit(m) = &e else { + unreachable!() + }; + assert_eq!(e.limit_kind(), Some(m.kind)); + assert!(m.detail.contains(m.kind.as_str()), "{m}"); + m.kind + } other => panic!("expected ResourceLimit, got {other:?}"), } } @@ -77,7 +85,7 @@ mod html { fn output_cap_rejects_large_output() { let html = "

hello world

".repeat(1000); let m = Markkit::builder().max_output_size(1000).build(); - assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + assert!(limit(m.extract_html_sync(&html)) == LimitKind::MaxOutputSize); let m = Markkit::builder().max_output_size(100_000).build(); assert!(m.extract_html_sync(&html).is_ok()); } @@ -92,7 +100,7 @@ mod html { ); let m = Markkit::builder().max_output_size(1 << 20).build(); let start = std::time::Instant::now(); - assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + assert!(limit(m.extract_html_sync(&html)) == LimitKind::MaxOutputSize); assert!(start.elapsed() < Duration::from_secs(10)); } @@ -105,24 +113,24 @@ mod html { format!("
  • a

    {table}
"), ] { let m = Markkit::builder().max_output_size(64 * 1024).build(); - assert!(limit_msg(m.extract_html_sync(&html)).contains("max_output_size")); + assert!(limit(m.extract_html_sync(&html)) == LimitKind::MaxOutputSize); } } #[test] fn zero_timeout_rejects() { let m = Markkit::builder().timeout(Duration::ZERO).build(); - assert!(limit_msg(m.extract_html_sync("

x

")).contains("timeout")); + assert!(limit(m.extract_html_sync("

x

")) == LimitKind::Timeout); let r = m.extract_document_sync(b"

x

", DocumentHint::content_type("text/html")); - assert!(limit_msg(r).contains("timeout")); + assert!(limit(r) == LimitKind::Timeout); } #[test] fn node_cap_rejects() { let html = "

a

".repeat(1000); let m = Markkit::builder().max_nodes(500).build(); - assert!(limit_msg(m.extract_html_sync(&html)).contains("max_nodes")); - assert!(limit_msg(m.parse_html(&html)).contains("max_nodes")); + assert!(limit(m.extract_html_sync(&html)) == LimitKind::MaxNodes); + assert!(limit(m.parse_html(&html)) == LimitKind::MaxNodes); let m = Markkit::builder().max_nodes(10_000).build(); assert!(m.extract_html_sync(&html).is_ok()); } @@ -179,7 +187,7 @@ mod docx { let body: String = (0..200).map(|i| para(&format!("p{i}"))).collect(); let bytes = docx(&body); let m = Markkit::builder().max_nodes(100).build(); - assert!(limit_msg(run(&m, &bytes)).contains("max_nodes")); + assert!(limit(run(&m, &bytes)) == LimitKind::MaxNodes); assert!(run(&Markkit::builder().build(), &bytes).is_ok()); } @@ -194,7 +202,7 @@ mod docx { let bytes = docx(&body); assert!(run(&Markkit::builder().build(), &bytes).is_ok()); let m = Markkit::builder().max_nesting_depth(8).build(); - assert!(limit_msg(run(&m, &bytes)).contains("max_nesting_depth")); + assert!(limit(run(&m, &bytes)) == LimitKind::MaxNestingDepth); } #[test] @@ -213,7 +221,7 @@ mod docx { .spawn(move || { let m = Markkit::builder().max_nesting_depth(usize::MAX).build(); assert!(run(&m, &bytes).unwrap().markdown.contains('x')); - assert!(limit_msg(run(&m, &deeper)).contains("max_nesting_depth")); + assert!(limit(run(&m, &deeper)) == LimitKind::MaxNestingDepth); }) .unwrap() .join() @@ -224,7 +232,7 @@ mod docx { fn zero_timeout_rejects() { let bytes = docx(¶("x")); let m = Markkit::builder().timeout(Duration::ZERO).build(); - assert!(limit_msg(run(&m, &bytes)).contains("timeout")); + assert!(limit(run(&m, &bytes)) == LimitKind::Timeout); } #[test] @@ -232,7 +240,7 @@ mod docx { let body: String = (0..200).map(|i| para(&format!("paragraph {i}"))).collect(); let bytes = docx(&body); let m = Markkit::builder().max_output_size(500).build(); - assert!(limit_msg(run(&m, &bytes)).contains("max_output_size")); + assert!(limit(run(&m, &bytes)) == LimitKind::MaxOutputSize); } } @@ -257,13 +265,13 @@ mod spreadsheet { #[test] fn zero_timeout_rejects() { let m = Markkit::builder().timeout(Duration::ZERO).build(); - assert!(limit_msg(run(&m, &sheet())).contains("timeout")); + assert!(limit(run(&m, &sheet())) == LimitKind::Timeout); } #[test] fn output_cap_rejects() { let m = Markkit::builder().max_output_size(500).build(); - assert!(limit_msg(run(&m, &sheet())).contains("max_output_size")); + assert!(limit(run(&m, &sheet())) == LimitKind::MaxOutputSize); assert!(run(&Markkit::builder().build(), &sheet()).is_ok()); } @@ -275,7 +283,10 @@ mod spreadsheet { let bytes = xlsb(&[Sheet::new("S", rows)]); let hint = DocumentHint::file_name("t.xlsb"); let m = Markkit::builder().timeout(Duration::ZERO).build(); - assert!(limit_msg(m.extract_document_sync(&bytes, hint)).contains("timeout")); + assert_eq!( + limit(m.extract_document_sync(&bytes, hint)), + LimitKind::Timeout + ); assert!( Markkit::builder() .build() diff --git a/crates/markkit/tests/notebook_integration.rs b/crates/markkit/tests/notebook_integration.rs index e07f233..662c146 100644 --- a/crates/markkit/tests/notebook_integration.rs +++ b/crates/markkit/tests/notebook_integration.rs @@ -196,7 +196,7 @@ fn security_limits_apply() { .build(); let r = m.extract_document_sync(nb.as_bytes(), DocumentHint::file_name("nb.ipynb")); assert!( - matches!(&r, Err(Error::ResourceLimit(msg)) if msg.contains("timeout")), + matches!(&r, Err(e) if e.limit_kind() == Some(markkit::LimitKind::Timeout)), "{r:?}" ); let cells: Vec<_> = (0..50) diff --git a/crates/markkit/tests/pdf_integration.rs b/crates/markkit/tests/pdf_integration.rs index 47236c3..71483dc 100644 --- a/crates/markkit/tests/pdf_integration.rs +++ b/crates/markkit/tests/pdf_integration.rs @@ -191,7 +191,7 @@ fn multiple_pages_and_page_cap() { assert_eq!(e.markdown.trim(), "Page one"); assert_eq!(e.stats.pages, Some(1)); assert!( - e.warnings[0].starts_with("truncated_pdf:"), + e.warnings[0].code == markkit::WarningCode::TruncatedPdf, "{:?}", e.warnings ); @@ -238,8 +238,8 @@ fn unmapped_composite_font_is_skipped_with_warning() { let e = extract(&pdf); assert_eq!(e.markdown.trim(), "Visible"); assert_eq!(e.warnings.len(), 1, "{:?}", e.warnings); - assert!(e.warnings[0].starts_with("pdf_unmapped_font: ")); - assert!(e.warnings[0].contains("NoMap")); + assert_eq!(e.warnings[0].code, markkit::WarningCode::PdfUnmappedFont); + assert!(e.warnings[0].detail.contains("NoMap")); } #[test] @@ -247,7 +247,7 @@ fn page_without_text_warns() { let e = extract(&pdf_pages(&["0 0 m 100 100 l S"])); assert_eq!(e.markdown.trim(), ""); assert!( - e.warnings[0].starts_with("pdf_no_text:"), + e.warnings[0].code == markkit::WarningCode::PdfNoText, "{:?}", e.warnings ); diff --git a/crates/markkit/tests/pdf_security.rs b/crates/markkit/tests/pdf_security.rs index ce2fdcd..7f50745 100644 --- a/crates/markkit/tests/pdf_security.rs +++ b/crates/markkit/tests/pdf_security.rs @@ -34,7 +34,10 @@ fn flate_bomb_content_stream_rejected() { let mut objs = catalog_page("5 0 R"); objs.push(pdf_stream("/Filter /FlateDecode", &bomb)); let err = run(&Markkit::builder().build(), &pdf_bytes(&objs)).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert!( + err.limit_kind() == Some(markkit::LimitKind::MaxPdfStreamSize), + "{err:?}" + ); } #[test] @@ -50,7 +53,10 @@ fn bomb_in_unreferenced_object_stream_rejected_before_parsing() { )); let m = Markkit::builder().max_pdf_stream_size(1024 * 1024).build(); let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert!( + err.limit_kind() == Some(markkit::LimitKind::MaxPdfStreamSize), + "{err:?}" + ); } #[test] @@ -65,7 +71,10 @@ fn chained_filters_are_measured_after_every_stage() { )); let m = Markkit::builder().max_pdf_stream_size(1024 * 1024).build(); let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert!( + err.limit_kind() == Some(markkit::LimitKind::MaxPdfStreamSize), + "{err:?}" + ); } #[test] @@ -75,7 +84,10 @@ fn escaped_filter_name_cannot_evade_scan() { objs.push(pdf_stream("/Filter /Fl#61teDecode", &bomb)); let m = Markkit::builder().max_pdf_stream_size(1024 * 1024).build(); let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert!( + err.limit_kind() == Some(markkit::LimitKind::MaxPdfStreamSize), + "{err:?}" + ); } #[test] @@ -89,7 +101,10 @@ fn total_decoded_size_capped_across_streams() { .max_pdf_decoded_size(1024 * 1024) .build(); let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert!( + err.limit_kind() == Some(markkit::LimitKind::MaxPdfDecodedSize), + "{err:?}" + ); // Each stream alone is fine. assert!(run(&Markkit::builder().build(), &pdf_bytes(&objs)).is_ok()); } @@ -124,7 +139,10 @@ fn form_xobject_reuse_hits_interpretation_budget() { .max_pdf_decoded_size(4 * 1024 * 1024) .build(); let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert!( + err.limit_kind() == Some(markkit::LimitKind::MaxPdfDecodedSize), + "{err:?}" + ); } #[test] @@ -149,7 +167,10 @@ fn repeated_contents_reference_is_charged_each_time() { .max_pdf_decoded_size(8 * 1024 * 1024) .build(); let err = run(&m, &pdf_bytes(&objs)).unwrap_err(); - assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); + assert!( + err.limit_kind() == Some(markkit::LimitKind::MaxPdfDecodedSize), + "{err:?}" + ); } #[test] @@ -314,7 +335,7 @@ fn zero_timeout_rejects() { .timeout(std::time::Duration::ZERO) .build(); match run(&m, &bytes) { - Err(Error::ResourceLimit(msg)) => assert!(msg.contains("timeout"), "{msg}"), + Err(e) => assert_eq!(e.limit_kind(), Some(markkit::LimitKind::Timeout), "{e}"), other => panic!("expected timeout, got {other:?}"), } } @@ -345,7 +366,11 @@ fn output_cap_rejects() { let bytes = pdf_pages(&[content.as_str()]); let m = Markkit::builder().max_output_size(500).build(); match run(&m, &bytes) { - Err(Error::ResourceLimit(msg)) => assert!(msg.contains("max_output_size"), "{msg}"), + Err(e) => assert_eq!( + e.limit_kind(), + Some(markkit::LimitKind::MaxOutputSize), + "{e}" + ), other => panic!("expected output cap, got {other:?}"), } } diff --git a/crates/markkit/tests/pptx_integration.rs b/crates/markkit/tests/pptx_integration.rs index beba996..3784a01 100644 --- a/crates/markkit/tests/pptx_integration.rs +++ b/crates/markkit/tests/pptx_integration.rs @@ -5,7 +5,7 @@ mod common; use common::*; -use markkit::{DocumentHint, Error, Markkit, format::Format}; +use markkit::{DocumentHint, Error, Markkit, Warning, WarningCode, format::Format}; fn extract_with(m: &Markkit, bytes: &[u8]) -> markkit::Extraction { m.extract_document_sync(bytes, DocumentHint::default()) @@ -133,7 +133,7 @@ fn hidden_slides_skipped_by_default() { assert!( e.warnings .iter() - .any(|w| w == "hidden_slide_skipped: slide 2 is hidden"), + .any(|w| w.to_string() == "hidden_slide_skipped: slide 2 is hidden"), "{:?}", e.warnings ); @@ -189,7 +189,7 @@ fn broken_slide_references_warn_and_other_slides_render() { assert!( e.warnings .iter() - .any(|w| w.starts_with("slide_error: slide 1")), + .any(|w| w.code == WarningCode::SlideError && w.detail.starts_with("slide 1")), "{:?}", e.warnings ); @@ -220,7 +220,10 @@ fn real_deck_from_python_pptx() { ); assert_eq!( e.warnings, - vec!["hidden_slide_skipped: slide 4 is hidden".to_string()] + vec![Warning::new( + WarningCode::HiddenSlideSkipped, + "slide 4 is hidden" + )] ); } diff --git a/crates/markkit/tests/spreadsheet_integration.rs b/crates/markkit/tests/spreadsheet_integration.rs index 457e593..cf9dabf 100644 --- a/crates/markkit/tests/spreadsheet_integration.rs +++ b/crates/markkit/tests/spreadsheet_integration.rs @@ -6,7 +6,7 @@ mod common; use common::Cell::*; use common::*; -use markkit::{DocumentHint, Markkit, format::Format}; +use markkit::{DocumentHint, Markkit, WarningCode, format::Format}; const XLSX_CT: &str = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; @@ -101,11 +101,11 @@ fn xlsx_row_cap_truncates_with_warning() { ); assert_eq!(e.warnings.len(), 1, "{:?}", e.warnings); assert!( - e.warnings[0].starts_with("truncated_sheet:"), + e.warnings[0].code == WarningCode::TruncatedSheet, "{:?}", e.warnings ); - assert!(e.warnings[0].contains("Big")); + assert!(e.warnings[0].detail.contains("Big")); } #[test] @@ -123,7 +123,11 @@ fn xlsx_col_cap_truncates_with_warning() { e.markdown, "## Wide\n\n| a | b |\n| --- | --- |\n| 1 | 2 |\n" ); - assert!(e.warnings.iter().any(|w| w.starts_with("truncated_sheet:"))); + assert!( + e.warnings + .iter() + .any(|w| w.code == WarningCode::TruncatedSheet) + ); } #[test] @@ -137,7 +141,7 @@ fn xlsx_hidden_sheet_skipped_by_default() { assert!( e.warnings .iter() - .any(|w| w.starts_with("hidden_sheet_skipped:")), + .any(|w| w.code == WarningCode::HiddenSheetSkipped), "{:?}", e.warnings ); @@ -173,7 +177,11 @@ fn xlsx_sparse_far_cell_does_not_allocate_grid() { let e = extract(&bytes); assert!(e.markdown.contains("top"), "{}", e.markdown); assert!(e.markdown.len() < 64 * 1024); - assert!(e.warnings.iter().any(|w| w.starts_with("truncated_sheet:"))); + assert!( + e.warnings + .iter() + .any(|w| w.code == WarningCode::TruncatedSheet) + ); } #[test] @@ -474,7 +482,7 @@ fn xlsb_hidden_sheet_skipped_by_default() { assert!( e.warnings .iter() - .any(|w| w.starts_with("hidden_sheet_skipped:")) + .any(|w| w.code == WarningCode::HiddenSheetSkipped) ); let m = Markkit::builder().include_hidden_sheets(true).build(); assert!(extract_with(&m, &bytes).markdown.contains("## Secret")); @@ -495,7 +503,7 @@ fn xlsb_caps_and_title() { assert_eq!( e.warnings .iter() - .filter(|w| w.starts_with("truncated_sheet:")) + .filter(|w| w.code == WarningCode::TruncatedSheet) .count(), 2, "{:?}", @@ -510,7 +518,7 @@ fn xlsb_missing_and_truncated_sheet_parts_warn() { let gone = rezip(&bytes, &["xl/worksheets/sheet1.bin"], &[]); let e = extract(&gone); assert!( - e.warnings.iter().any(|w| w.starts_with("sheet_error:")), + e.warnings.iter().any(|w| w.code == WarningCode::SheetError), "{:?}", e.warnings ); @@ -523,7 +531,7 @@ fn xlsb_missing_and_truncated_sheet_parts_warn() { ); let e = extract(&cut); assert!( - e.warnings.iter().any(|w| w.starts_with("sheet_error:")), + e.warnings.iter().any(|w| w.code == WarningCode::SheetError), "{:?}", e.warnings ); diff --git a/crates/markkit/tests/text_integration.rs b/crates/markkit/tests/text_integration.rs index 56b9f7b..35d53e3 100644 --- a/crates/markkit/tests/text_integration.rs +++ b/crates/markkit/tests/text_integration.rs @@ -127,7 +127,7 @@ fn security_invalid_utf8_lossy_with_warning() { let e = extract(b"caf\xE9 \xC3", DocumentHint::file_name("a.txt")); assert_eq!(e.markdown, "caf\u{FFFD} \u{FFFD}\n"); assert_eq!(e.warnings.len(), 1); - assert!(e.warnings[0].starts_with("invalid_utf8:")); + assert_eq!(e.warnings[0].code, markkit::WarningCode::InvalidUtf8); } #[test] @@ -237,7 +237,7 @@ fn security_text_and_markdown_share_limits() { continue; } assert!( - matches!(&r, Err(Error::ResourceLimit(msg)) if msg.contains("timeout")), + matches!(&r, Err(e) if e.limit_kind() == Some(markkit::LimitKind::Timeout)), "{name}: {r:?}" ); } diff --git a/knowledge/formats/format-support.md b/knowledge/formats/format-support.md index 9bab4e1..824ef9b 100644 --- a/knowledge/formats/format-support.md +++ b/knowledge/formats/format-support.md @@ -54,7 +54,7 @@ pub struct Extraction { // #[non_exhaustive] pub markdown: String, pub title: Option, // see "Document titles" below pub format: Format, - pub warnings: Vec, // ": " + pub warnings: Vec, // { code: WarningCode, detail: String }; Display ": " pub stats: ExtractionStats, // pages / sheets / rows / paragraphs / tables (Option each) pub metadata: Metadata, // HTML page metadata, see HTML Extraction; empty otherwise } @@ -82,10 +82,14 @@ malformed property part yields `None` without failing the extraction. only the Markdown. Parsing is CPU-bound; async callers should use the sync variant in `spawn_blocking`. -Warning codes: `truncated_sheet`, `hidden_sheet_skipped`, `sheet_error`, `truncated_pdf`, -`pdf_unmapped_font`, `pdf_no_text`, `invalid_utf8`, `hidden_content_skipped`, -`hidden_slide_skipped`, `slide_error`, -`malformed_csv`, `truncated_table`. +Warning codes (`WarningCode`, serialized `snake_case`): `truncated_sheet`, +`hidden_sheet_skipped`, `sheet_error`, `hidden_slide_skipped`, `slide_error`, `truncated_pdf`, +`pdf_unmapped_font`, `pdf_no_text`, `invalid_utf8`, `hidden_content_skipped`, `malformed_csv`, +`truncated_table`, `truncated` (agent tool only) and `other` (custom parsers). Callers match on +`code`; `detail` is for humans and may change. + +`Markkit` is `Clone + Debug + Send + Sync` and holds only its configuration, so embedders can +clone it per task instead of wrapping it in an `Arc`. ## Format Detection diff --git a/knowledge/integrations/tool-contract.md b/knowledge/integrations/tool-contract.md index d9843f5..9ccf654 100644 --- a/knowledge/integrations/tool-contract.md +++ b/knowledge/integrations/tool-contract.md @@ -80,11 +80,13 @@ Exactly one source per call. `ToolOutput::result`: ```json -{ "markdown": "string", "title": "string|null", "format": "html|docx|xlsx|xls|ods", +{ "markdown": "string", "title": "string|null", "format": "html|docx|pptx|xlsx|xlsb|xls|ods", "warnings": ["string"], "truncated": false } ``` -Truncation cuts on a character boundary and adds a `truncated:` warning. +Truncation cuts on a character boundary and adds a `WarningCode::Truncated` warning. Tool +warnings stay `": "` strings (the `Warning` `Display` form), so the tool schema +is unchanged by the library's structured warnings. `metadata.extra` is `{ "stats": ExtractionStats, "chars": }`. `metadata.duration` is zero on `wasm32-unknown-unknown` (no `Instant` there). diff --git a/knowledge/log.md b/knowledge/log.md index d508609..24764a4 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,6 +2,8 @@ ## 2026-10-27 +* **Embedder API (#11)**: `Markkit` derives `Clone` + `Debug`; warnings are structured (`Warning { code: WarningCode, detail }`) instead of `"code: detail"` strings. The agent tool keeps string warnings. See [Format Support](formats/format-support.md) and [Tool Contract](integrations/tool-contract.md). +* **Structured limit errors (#11)**: `Error::ResourceLimit` carries `LimitExceeded { kind: LimitKind, detail }`; every breach site (input, output, timeout, nodes, nesting, archive entries/entry size/total size, spreadsheet cells, PDF stream/decoded size) sets its kind. Tests assert the kind, not message text. See [Threat Model](security/threat-model.md) Resource Limits. * **PPTX support (#11)**: new `pptx` feature (in `all_formats`) converts presentations slide by slide: titles, bulleted body placeholders, text boxes, tables, links, speaker notes; hidden slides skipped. Shared OOXML helpers moved from `docx.rs` to `ooxml.rs`. Legacy `.ppt` is now its own `Format::Ppt` (rejected). See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md) T6. * **Fuzzing**: new [Fuzzing](security/fuzzing.md) concept. Eleven cargo-fuzz targets (`crates/markkit/fuzz`, every parser incl. XLSB, PDF, Markdown, CSV, plus the renderer) assert the S17 oracle and run nightly (`fuzz.yml`). The `xls_biff` target found calamine panicking on a dozen truncated BIFF record shapes and in its formula decompiler; the XLS pre-scan now rejects them, neutralises formula tokens and hands calamine a container rebuilt from the scanned stream. See [Threat Model](security/threat-model.md) T6c. * **XLSB support (#11)**: Excel binary workbooks are read by a new bounds-checked BIFF12 reader (`spreadsheet/xlsb.rs`) instead of being rejected; calamine's XLSB reader was not used because it panics on malformed records. Same caps, hidden-sheet and macro rules as XLSX. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md) T6c. diff --git a/knowledge/runtimes/node-package.md b/knowledge/runtimes/node-package.md index 7343ced..552ecb0 100644 --- a/knowledge/runtimes/node-package.md +++ b/knowledge/runtimes/node-package.md @@ -25,7 +25,7 @@ const sync = m.extractHtmlSync('

Hello

'); const fromUrl = await m.extractUrl('https://example.com', { allowedHosts: ['example.com'] }); const fromBuffer = await m.extractBytes(Buffer.from('

Hello

'), 'text/html'); const doc = m.extractDocumentSync(buf, { fileName: 'report.docx' }); -// { markdown, title, format, warnings, stats, metadata } +// { markdown, title, format, warnings: [{ code, detail }], stats, metadata } ``` Every function takes an optional trailing `options` object (`MarkkitOptions`): diff --git a/knowledge/runtimes/wasm-package.md b/knowledge/runtimes/wasm-package.md index e1b65e9..4ba6967 100644 --- a/knowledge/runtimes/wasm-package.md +++ b/knowledge/runtimes/wasm-package.md @@ -31,7 +31,7 @@ import { initMarkkit, Markkit, MarkkitTool, version } from "@everruns/markkit-wa await initMarkkit(); // once; idempotent const mk = new Markkit({ includeSelectors: ["article"] }); mk.extractHtml(html); // string -mk.extractDocument(bytes, { fileName }); // { markdown, title, format, warnings, stats, metadata } +mk.extractDocument(bytes, { fileName }); // { markdown, title, format, warnings: [{ code, detail }], stats, metadata } mk.extractBytes(bytes, contentType); // string const tool = new MarkkitTool({ locale, maxOutputChars }); tool.toolDefinition(); await tool.execute(args); diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 3d8e665..7b16a26 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -92,7 +92,9 @@ that renders as headings, links, code fences or tables). **Sub-threats**: All limits live in one `Limits` struct (`crates/markkit/src/limits.rs`, -`MarkkitBuilder::limits`); violations return `Error::ResourceLimit` naming the limit. +`MarkkitBuilder::limits`); violations return `Error::ResourceLimit(LimitExceeded { kind, detail })`; `kind` is a +`LimitKind` named after the `Limits` field (`Error::limit_kind()`), so callers branch on it +without parsing the message. #### T2a: Oversized Input - Input size checked before parsing against `max_input_size`