diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c44e70..5cde8de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Added +- XLSB (Excel binary workbook) support in the `spreadsheet` feature, via a new bounds-checked + reader: values, shared strings, dates and durations, hidden-sheet skipping, row/column caps and + the workbook title. Previously XLSB returned `UnsupportedFormat`. - `markkit mcp`: Model Context Protocol server over stdio exposing the `markkit` tool (`tools/list`, `tools/call`). URL fetching keeps the SSRF guard; `--no-url` keeps it offline, the `extract` limit flags and `--max-output-chars` bound each call. diff --git a/README.md b/README.md index cb347e7..d8d31a9 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ package for browsers and edge runtimes. - **HTML** – Web pages and HTML documents; nav, header, footer, scripts, styles and forms stripped by default - **DOCX** – Headings, lists, tables, links, emphasis -- **Spreadsheets** – XLSX/XLSM, XLS and ODS workbooks to markdown tables, one section per sheet +- **Spreadsheets** – XLSX/XLSM, XLSB, XLS and ODS workbooks to markdown tables, one section per sheet - **Plain text / Markdown** – Text to paragraphs; Markdown normalized through the same IR (raw HTML never passed through) - **Format detection** – Content type, then file extension, then magic bytes - **URL fetching** – Fetch and extract in one call (`http_client` feature) @@ -32,11 +32,11 @@ package for browsers and edge runtimes. |---|---|---| | HTML / XHTML | Supported | `html` (default) | | DOCX | Supported | `docx` | -| XLSX, XLSM, XLS, ODS | Supported | `spreadsheet` | +| XLSX, XLSM, XLSB, XLS, ODS | Supported | `spreadsheet` | | Plain text | Supported | always on | | Markdown | Supported | `markdown` | | PDF (text layer) | Supported | `pdf` | -| DOC, PPTX, XLSB | Detected, rejected explicitly | – | +| DOC, PPTX | Detected, rejected explicitly | – | ## Quick Start diff --git a/crates/markkit/src/extract.rs b/crates/markkit/src/extract.rs index be2dbfb..9f7dd28 100644 --- a/crates/markkit/src/extract.rs +++ b/crates/markkit/src/extract.rs @@ -230,7 +230,7 @@ impl Markkit { .map(|e| e.markdown) } - /// Convert a document (HTML, PDF, DOCX, XLSX/XLS/ODS, plain text, Markdown) to Markdown with metadata. + /// Convert a document (HTML, PDF, DOCX, XLSX/XLSB/XLS/ODS, plain text, Markdown) to Markdown with metadata. /// /// The format is detected from `hint.content_type`, then the extension of /// `hint.file_name`, then magic bytes. Parsing is CPU-bound and runs inline; diff --git a/crates/markkit/src/lib.rs b/crates/markkit/src/lib.rs index c44c321..edb7249 100644 --- a/crates/markkit/src/lib.rs +++ b/crates/markkit/src/lib.rs @@ -33,7 +33,7 @@ //! //! - **HTML** (`html`, default): Convert HTML to markdown with configurable rules //! - **DOCX** (`docx`): Convert Word documents -//! - **Spreadsheet** (`spreadsheet`): Convert XLSX/XLSM, XLS and ODS workbooks to tables +//! - **Spreadsheet** (`spreadsheet`): Convert XLSX/XLSM, XLSB, XLS and ODS workbooks to tables //! - **Plain text** (always on): `text/plain`, `.txt` //! - **Markdown** (`markdown`): `text/markdown`, `.md`, normalized through the IR //! - **PDF** (`pdf`): Text layer of PDF documents (no OCR); encrypted PDFs are diff --git a/crates/markkit/src/spreadsheet.rs b/crates/markkit/src/spreadsheet.rs index 4b0c99d..f4645c7 100644 --- a/crates/markkit/src/spreadsheet.rs +++ b/crates/markkit/src/spreadsheet.rs @@ -16,7 +16,9 @@ //! executed or rendered; dialog/macro/chart sheets are skipped. //! - Hidden sheets are skipped by default (content invisible to a human reviewer //! is a prompt-injection vector); opt in with `include_hidden_sheets`. -//! - XLSB is detected but not supported (no grid pre-scan implemented). +//! - XLSB is read by our own bounds-checked BIFF12 reader ([`xlsb`]), streaming +//! cells into the same capped grid; calamine's XLSB reader panics on +//! malformed records. use std::collections::BTreeMap; use std::io::{Cursor, Read, Seek}; @@ -31,6 +33,8 @@ use crate::format::Format; use crate::ir::{Document, Node, Source, TableCell}; use crate::limits::Budget; +mod xlsb; + pub(crate) struct SheetOutput { pub doc: Document, pub format: Format, @@ -59,6 +63,7 @@ const ODS_MAX_COLS: u64 = 16_384; #[derive(Clone, Copy)] enum Container { Xlsx, + Xlsb, Ods, Xls, } @@ -76,15 +81,17 @@ pub(crate) fn parse( let container = if has(&zip, "xl/workbook.xml") { Container::Xlsx } else if has(&zip, "xl/workbook.bin") { - return Err(Error::UnsupportedFormat( - "XLSB (binary Excel workbook) is not supported".into(), - )); + Container::Xlsb } else if has(&zip, "mimetype") || has(&zip, "content.xml") { Container::Ods } else { - return Err(err("ZIP container is not an XLSX or ODS workbook")); + return Err(err("ZIP container is not an XLSX, XLSB or ODS workbook")); }; zip.validate_all().map_err(archive_err)?; + if let Container::Xlsb = container { + let title = read_dc_title(&mut zip, "docProps/core.xml", config, budget)?; + return parse_xlsb(&mut zip, config, budget, title); + } let title = if let Container::Ods = container { let content = zip .read("content.xml") @@ -105,7 +112,7 @@ pub(crate) fn parse( prescan_xls(&mut comp, config)?; (Container::Xls, xls_title(&mut comp)) } else { - return Err(err("not a spreadsheet (expected XLSX, XLS or ODS)")); + return Err(err("not a spreadsheet (expected XLSX, XLSB, XLS or ODS)")); }; budget.check_time()?; let bytes = patched.as_deref().unwrap_or(bytes); @@ -150,9 +157,66 @@ pub(crate) fn parse( read_ranges(&mut wb, sel, &mut out, config, budget)?; out.finish(Format::Xls, title) } + Container::Xlsb => unreachable!("XLSB returns early"), } } +fn parse_xlsb( + zip: &mut ZipGuard<'_>, + config: &MarkkitBuilder, + budget: &Budget, + title: Option, +) -> Result { + let wb = xlsb::open(zip, config.limits.nesting_depth(), budget)?; + budget.check_time()?; + let meta: Vec = wb + .sheets + .iter() + .map(|s| calamine::Sheet { + name: s.name.clone(), + // Chart / dialog / macro sheets and dangling relationships have no + // worksheet part and are skipped like other non-worksheets. + typ: if s.part.is_some() { + SheetType::WorkSheet + } else { + SheetType::ChartSheet + }, + visible: match s.visibility { + xlsb::Visibility::Visible => SheetVisible::Visible, + xlsb::Visibility::Hidden => SheetVisible::Hidden, + }, + }) + .collect(); + let mut out = Collector::new(config); + let selected = out.select(&meta); + for ((name, include), sheet) in selected.into_iter().zip(&wb.sheets) { + let Some(part) = sheet.part.as_deref().filter(|_| include) else { + continue; + }; + let Some(data) = zip.read(part).map_err(archive_err)? else { + out.warn(format!("sheet_error: sheet '{name}': missing part {part}")); + continue; + }; + let mut grid = Grid::new(config); + let res = (|| { + for cell in wb.cells(&data) { + if budget.expired() { + break; + } + let (r, c, v) = cell?; + grid.add(r, c, fmt_xlsb(&v, wb.is_1904)); + } + Ok::<_, Error>(()) + })(); + budget.check_time()?; + match res { + Ok(()) => out.push_sheet(&name, grid), + Err(e) => out.warn(format!("sheet_error: sheet '{name}': {e}")), + } + } + out.finish(Format::Xlsb, title) +} + fn read_ranges( wb: &mut R, sel: Vec<(String, bool)>, @@ -238,6 +302,26 @@ fn fmt_data(d: &Data) -> String { } } +fn fmt_xlsb(v: &xlsb::Value, is_1904: bool) -> String { + use calamine::{ExcelDateTime, ExcelDateTimeType}; + match v { + xlsb::Value::Number(f) => fmt_float(*f), + xlsb::Value::DateTime(f) => fmt_datetime(&ExcelDateTime::new( + *f, + ExcelDateTimeType::DateTime, + is_1904, + )), + xlsb::Value::Duration(f) => fmt_datetime(&ExcelDateTime::new( + *f, + ExcelDateTimeType::TimeDelta, + is_1904, + )), + xlsb::Value::Text(s) => s.clone(), + xlsb::Value::Bool(b) => b.to_string(), + xlsb::Value::Error(e) => (*e).to_string(), + } +} + fn fmt_ref(d: &DataRef<'_>) -> String { match d { DataRef::SharedString(s) => (*s).to_string(), diff --git a/crates/markkit/src/spreadsheet/xlsb.rs b/crates/markkit/src/spreadsheet/xlsb.rs new file mode 100644 index 0000000..06e68c9 --- /dev/null +++ b/crates/markkit/src/spreadsheet/xlsb.rs @@ -0,0 +1,640 @@ +//! XLSB (Excel binary workbook, BIFF12 records in an OPC ZIP) reader. +//! +//! Decisions: +//! - Own reader, not calamine's: calamine's XLSB path indexes record buffers +//! and shared-string tables without bounds checks, so malformed records panic +//! (fatal under `panic = "abort"`). Every read here is checked; malformed +//! records are skipped or fail the part with an error, never a panic. +//! - Parts are read through [`ZipGuard`] (bounded decompression) after the +//! whole archive is validated, so no part can exceed archive limits. +//! - Like the XLSX path, cells stream into the capped [`Grid`](super) — no dense +//! grid, whatever `BrtWsDim` claims. +//! - Only cached cell values are read: formulas are never evaluated, VBA +//! (`vbaProject.bin`) is never read. +//! - Dates: a numeric cell is a date when its XF's number format is a built-in +//! date/time id or a custom code with date/time tokens ([MS-XLSB] 2.4.353). + +use crate::archive::ZipGuard; +use crate::error::Error; +use crate::limits::Budget; + +fn err(msg: impl std::fmt::Display) -> Error { + Error::SpreadsheetExtract(format!("XLSB: {msg}")) +} + +// ─── Records ──────────────────────────────────────────────────────── + +/// Iterate BIFF12 records `(type, payload)`. Type: 1–2 bytes, 7 bits each; +/// size: 1–4 bytes, 7 bits each ([MS-XLSB] 2.1.4). Stops with an error on a +/// truncated record. +pub(crate) fn records(data: &[u8]) -> impl Iterator> { + let mut pos = 0usize; + let mut failed = false; + std::iter::from_fn(move || { + if failed || pos >= data.len() { + return None; + } + let mut next = || { + let b = *data.get(pos)?; + pos += 1; + Some(b) + }; + let rec = (|| { + let b0 = next()?; + let mut typ = u16::from(b0 & 0x7F); + if b0 & 0x80 != 0 { + typ |= u16::from(next()? & 0x7F) << 7; + } + let mut len = 0usize; + for i in 0..4 { + let b = next()?; + len |= usize::from(b & 0x7F) << (7 * i); + if b & 0x80 == 0 { + break; + } + } + Some((typ, len)) + })(); + let Some((typ, len)) = rec else { + failed = true; + return Some(Err(err("truncated record header"))); + }; + let Some(payload) = pos.checked_add(len).and_then(|end| data.get(pos..end)) else { + failed = true; + return Some(Err(err(format!("record 0x{typ:04X} overruns its part")))); + }; + pos += len; + Some(Ok((typ, payload))) + }) +} + +fn u16_at(d: &[u8], at: usize) -> Option { + Some(u16::from_le_bytes( + d.get(at..at.checked_add(2)?)?.try_into().ok()?, + )) +} + +fn u32_at(d: &[u8], at: usize) -> Option { + Some(u32::from_le_bytes( + d.get(at..at.checked_add(4)?)?.try_into().ok()?, + )) +} + +fn f64_at(d: &[u8], at: usize) -> Option { + Some(f64::from_le_bytes( + d.get(at..at.checked_add(8)?)?.try_into().ok()?, + )) +} + +/// `XLWideString` at `at`: u32 character count + UTF-16LE. Returns the string +/// and the offset just past it. `0xFFFFFFFF` (nullable form) reads as empty. +fn wide_str(d: &[u8], at: usize) -> Option<(String, usize)> { + let n = u32_at(d, at)?; + if n == u32::MAX { + return Some((String::new(), at + 4)); + } + let start = at + 4; + let end = start.checked_add(usize::try_from(n).ok()?.checked_mul(2)?)?; + Some((super::utf16le(d.get(start..end)?), end)) +} + +// ─── Workbook ─────────────────────────────────────────────────────── + +/// Sheet visibility from `BrtBundleSh.hsState`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Visibility { + Visible, + Hidden, +} + +#[derive(Debug)] +pub(crate) struct SheetRef { + pub name: String, + /// Archive path of the sheet part; `None` for non-worksheet sheets + /// (chart, dialog, macro) or a dangling relationship. + pub part: Option, + pub visibility: Visibility, +} + +#[derive(Debug, Default)] +pub(crate) struct Workbook { + pub sheets: Vec, + pub is_1904: bool, + strings: Vec, + /// Per cell XF: number format kind. + xf_kinds: Vec, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum NumKind { + Number, + DateTime, + Duration, +} + +const WORKSHEET_REL: &str = "/relationships/worksheet"; + +pub(crate) fn open( + zip: &mut ZipGuard<'_>, + max_depth: usize, + budget: &Budget, +) -> Result { + let a = super::archive_err; + let workbook = zip + .read("xl/workbook.bin") + .map_err(a)? + .ok_or_else(|| err("missing xl/workbook.bin"))?; + let rels = zip + .read("xl/_rels/workbook.bin.rels") + .map_err(a)? + .map(|x| parse_rels(&x, max_depth, budget)) + .unwrap_or_default(); + let mut wb = Workbook::default(); + read_workbook(&workbook, &rels, &mut wb)?; + if let Some(sst) = zip.read("xl/sharedStrings.bin").map_err(a)? { + wb.strings = read_shared_strings(&sst)?; + } + if let Some(styles) = zip.read("xl/styles.bin").map_err(a)? { + wb.xf_kinds = read_styles(&styles); + } + Ok(wb) +} + +/// `(Id, Type, Target)` from `xl/_rels/workbook.bin.rels`; malformed XML → empty. +fn parse_rels(xml: &[u8], max_depth: usize, budget: &Budget) -> Vec<(String, String, String)> { + let Ok(root) = crate::xml::parse(xml, max_depth, budget) else { + return Vec::new(); + }; + let mut out = Vec::new(); + let mut stack = vec![&root]; + while let Some(el) = stack.pop() { + for c in el.elements() { + if c.name == "Relationship" { + if let (Some(id), Some(t), Some(target)) = + (c.attr("Id"), c.attr("Type"), c.attr("Target")) + { + out.push((id.to_string(), t.to_string(), target.to_string())); + } + } else { + stack.push(c); + } + } + } + out +} + +/// Resolve a relationship target relative to `xl/`. +fn resolve_target(target: &str) -> String { + match target.strip_prefix('/') { + Some(abs) => abs.to_string(), + None => format!("xl/{target}"), + } +} + +fn read_workbook( + data: &[u8], + rels: &[(String, String, String)], + wb: &mut Workbook, +) -> Result<(), Error> { + for rec in records(data) { + let (typ, d) = rec?; + match typ { + // BrtWbProp: flags, bit 0 = f1904. + 0x0099 => wb.is_1904 = u32_at(d, 0).is_some_and(|f| f & 1 != 0), + // BrtBundleSh: hsState, iTabID, strRelID, strName. + 0x009C => { + let malformed = || err("malformed sheet entry in workbook.bin"); + let state = u32_at(d, 0).ok_or_else(malformed)?; + let (rel_id, next) = wide_str(d, 8).ok_or_else(malformed)?; + let (name, _) = wide_str(d, next).ok_or_else(malformed)?; + let part = rels + .iter() + .find(|(id, t, _)| *id == rel_id && t.ends_with(WORKSHEET_REL)) + .map(|(_, _, target)| resolve_target(target)); + wb.sheets.push(SheetRef { + name, + part, + visibility: if state == 0 { + Visibility::Visible + } else { + Visibility::Hidden + }, + }); + } + // BrtEndBundleShs + 0x0090 => break, + _ => {} + } + } + Ok(()) +} + +fn read_shared_strings(data: &[u8]) -> Result, Error> { + let mut out = Vec::new(); + for rec in records(data) { + let (typ, d) = rec?; + // BrtSSTItem: RichStr = flags u8 + XLWideString (+ optional runs). + if typ == 0x0013 { + out.push(wide_str(d, 1).map(|(s, _)| s).unwrap_or_default()); + } + } + Ok(out) +} + +/// Number format kind per cell XF. Best effort: malformed styles → no dates. +fn read_styles(data: &[u8]) -> Vec { + let mut custom = std::collections::HashMap::new(); + let mut kinds = Vec::new(); + let mut in_cell_xfs = false; + for rec in records(data) { + let Ok((typ, d)) = rec else { break }; + match typ { + // BrtFmt: ifmt u16 + stFmtCode. + 0x002C => { + if let (Some(id), Some((code, _))) = (u16_at(d, 0), wide_str(d, 2)) { + custom.insert(id, custom_kind(&code)); + } + } + 0x0269 => in_cell_xfs = true, + 0x026A => in_cell_xfs = false, + // BrtXF: ixfeParent u16, iFmt u16. + 0x002F if in_cell_xfs => { + let id = u16_at(d, 2).unwrap_or(0); + kinds.push(custom.get(&id).copied().unwrap_or_else(|| builtin_kind(id))); + } + _ => {} + } + } + kinds +} + +fn builtin_kind(id: u16) -> NumKind { + match id { + 14..=22 | 27..=36 | 45 | 47 | 50..=58 => NumKind::DateTime, + 46 => NumKind::Duration, + _ => NumKind::Number, + } +} + +/// Classify a custom number format code: date/time tokens outside quoted +/// literals, escapes and `[...]` sections (except elapsed `[h]`, `[m]`, `[s]`). +fn custom_kind(code: &str) -> NumKind { + // Only the first section (positive numbers) matters. + let mut chars = code.chars().peekable(); + let (mut date, mut elapsed) = (false, false); + while let Some(c) = chars.next() { + match c { + ';' => break, + '"' => { + for q in chars.by_ref() { + if q == '"' { + break; + } + } + } + '\\' | '_' | '*' => { + chars.next(); + } + '[' => { + let mut inner = String::new(); + for b in chars.by_ref() { + if b == ']' { + break; + } + inner.push(b); + } + let lower = inner.to_ascii_lowercase(); + if !lower.is_empty() && lower.chars().all(|c| matches!(c, 'h' | 'm' | 's')) { + elapsed = true; + } + } + 'y' | 'Y' | 'd' | 'D' | 'h' | 'H' | 's' | 'S' | 'm' | 'M' => date = true, + _ => {} + } + } + if elapsed { + NumKind::Duration + } else if date && !code.eq_ignore_ascii_case("general") { + NumKind::DateTime + } else { + NumKind::Number + } +} + +// ─── Cells ────────────────────────────────────────────────────────── + +/// A cell value as read from a BIFF12 cell record. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum Value { + Number(f64), + DateTime(f64), + Duration(f64), + Text(String), + Bool(bool), + Error(&'static str), +} + +fn error_text(code: u8) -> &'static str { + match code { + 0x00 => "#NULL!", + 0x07 => "#DIV/0!", + 0x0F => "#VALUE!", + 0x17 => "#REF!", + 0x1D => "#NAME?", + 0x24 => "#NUM!", + 0x2A => "#N/A", + 0x2B => "#GETTING_DATA", + _ => "#ERROR!", + } +} + +/// `RkNumber` ([MS-XLSB] 2.5.122). +fn rk(v: u32) -> f64 { + let n = if v & 2 != 0 { + f64::from((v as i32) >> 2) + } else { + f64::from_bits(u64::from(v & 0xFFFF_FFFC) << 32) + }; + if v & 1 != 0 { n / 100.0 } else { n } +} + +impl Workbook { + /// Stream a worksheet's cells as `(row, col, value)`. Records outside + /// `BrtBeginSheetData`…`BrtEndSheetData` are ignored; malformed cell records + /// are skipped; a truncated part ends the stream with an error. + pub(crate) fn cells<'d>( + &'d self, + data: &'d [u8], + ) -> impl Iterator> + 'd { + let mut recs = records(data); + let mut in_data = false; + let mut row = 0u32; + std::iter::from_fn(move || { + loop { + let (typ, d) = match recs.next()? { + Ok(r) => r, + Err(e) => return Some(Err(e)), + }; + match typ { + 0x0091 => in_data = true, + 0x0092 => return None, + 0x0000 if in_data => row = u32_at(d, 0).unwrap_or(row), + 0x0002..=0x000B if in_data => { + if let Some(cell) = self.cell(typ, d) { + return Some(Ok((row, cell.0, cell.1))); + } + } + _ => {} + } + } + }) + } + + fn cell(&self, typ: u16, d: &[u8]) -> Option<(u32, Value)> { + // Cell: column u32, iStyleRef 24 bits, flags 8 bits; value at offset 8. + let col = u32_at(d, 0)?; + let style = (u32_at(d, 4)? & 0x00FF_FFFF) as usize; + let num = |n: f64| match self.xf_kinds.get(style) { + Some(NumKind::DateTime) => Value::DateTime(n), + Some(NumKind::Duration) => Value::Duration(n), + _ => Value::Number(n), + }; + let v = match typ { + 0x0002 => num(rk(u32_at(d, 8)?)), + 0x0003 | 0x000B => Value::Error(error_text(*d.get(8)?)), + 0x0004 | 0x000A => Value::Bool(*d.get(8)? != 0), + 0x0005 | 0x0009 => num(f64_at(d, 8)?), + 0x0006 | 0x0008 => Value::Text(wide_str(d, 8)?.0), + 0x0007 => Value::Text(self.strings.get(u32_at(d, 8)? as usize)?.clone()), + _ => return None, + }; + Some((col, v)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn rec(out: &mut Vec, typ: u16, data: &[u8]) { + if typ < 0x80 { + out.push(typ as u8); + } else { + out.push((typ & 0x7F) as u8 | 0x80); + out.push((typ >> 7) as u8); + } + let mut n = data.len(); + loop { + let b = (n & 0x7F) as u8; + n >>= 7; + if n == 0 { + out.push(b); + break; + } + out.push(b | 0x80); + } + out.extend_from_slice(data); + } + + fn ws(s: &str) -> Vec { + let mut v = (s.encode_utf16().count() as u32).to_le_bytes().to_vec(); + v.extend(s.encode_utf16().flat_map(u16::to_le_bytes)); + v + } + + #[test] + fn record_varints() { + let mut d = Vec::new(); + rec(&mut d, 0x0269, &[1, 2, 3]); + rec(&mut d, 0x0013, &vec![7; 300]); + let got: Vec<_> = records(&d).map(|r| r.unwrap()).collect(); + assert_eq!(got[0], (0x0269, &[1u8, 2, 3][..])); + assert_eq!(got[1].0, 0x0013); + assert_eq!(got[1].1.len(), 300); + } + + #[test] + fn truncated_records_error_once() { + let mut d = Vec::new(); + rec(&mut d, 0x0013, &[1, 2, 3, 4]); + d.truncate(d.len() - 1); + let got: Vec<_> = records(&d).collect(); + assert_eq!(got.len(), 1); + assert!(got[0].is_err()); + // Header cut mid-varint. + assert!(records(&[0x80]).next().unwrap().is_err()); + assert!( + records(&[0x13, 0xFF, 0xFF, 0xFF, 0xFF]) + .next() + .unwrap() + .is_err() + ); + } + + #[test] + fn wide_strings_are_bounds_checked() { + assert_eq!(wide_str(&ws("hé"), 0), Some(("hé".to_string(), 8))); + assert_eq!(wide_str(&u32::MAX.to_le_bytes(), 0).unwrap().0, ""); + let mut long = 1_000_000u32.to_le_bytes().to_vec(); + long.extend_from_slice(b"ab"); + assert_eq!(wide_str(&long, 0), None); + assert_eq!(wide_str(&[1, 0], 0), None); + assert_eq!(wide_str(&ws("x"), usize::MAX - 1), None); + } + + #[test] + fn rk_numbers() { + assert_eq!(rk((42 << 2) | 2), 42.0); + assert_eq!(rk(((-7i32 << 2) as u32) | 2), -7.0); + assert_eq!(rk((1234 << 2) | 3), 12.34); + let bits = (1.5f64.to_bits() >> 32) as u32; + assert_eq!(rk(bits), 1.5); + } + + #[test] + fn number_format_classification() { + assert_eq!(custom_kind("yyyy-mm-dd"), NumKind::DateTime); + assert_eq!(custom_kind("h:mm AM/PM"), NumKind::DateTime); + assert_eq!(custom_kind("[h]:mm:ss"), NumKind::Duration); + assert_eq!(custom_kind("0.00"), NumKind::Number); + assert_eq!(custom_kind("#,##0 \"days\""), NumKind::Number); + assert_eq!(custom_kind("[Red]0.0"), NumKind::Number); + assert_eq!(custom_kind("[$-409]d-mmm"), NumKind::DateTime); + assert_eq!(custom_kind("General"), NumKind::Number); + assert_eq!(custom_kind("0;\"d\"m"), NumKind::Number); + assert_eq!(builtin_kind(14), NumKind::DateTime); + assert_eq!(builtin_kind(46), NumKind::Duration); + assert_eq!(builtin_kind(2), NumKind::Number); + } + + fn cell_rec(col: u32, style: u32, value: &[u8]) -> Vec { + let mut d = col.to_le_bytes().to_vec(); + d.extend_from_slice(&style.to_le_bytes()); + d.extend_from_slice(value); + d + } + + #[test] + fn cells_stream_with_malformed_records_skipped() { + let wb = Workbook { + strings: vec!["shared".into()], + xf_kinds: vec![NumKind::Number, NumKind::DateTime], + ..Default::default() + }; + let mut s = Vec::new(); + rec(&mut s, 0x0005, &cell_rec(0, 0, &1.0f64.to_le_bytes())); // before sheet data: ignored + rec(&mut s, 0x0091, &[]); + rec(&mut s, 0x0000, &3u32.to_le_bytes()); + rec(&mut s, 0x0005, &cell_rec(1, 1, &45000.0f64.to_le_bytes())); + rec(&mut s, 0x0007, &cell_rec(2, 0, &0u32.to_le_bytes())); + rec(&mut s, 0x0007, &cell_rec(3, 0, &99u32.to_le_bytes())); // bad isst: skipped + rec(&mut s, 0x0006, &cell_rec(4, 0, &ws("inline"))); + rec(&mut s, 0x0006, &cell_rec(5, 0, &[0xFF, 0xFF, 0xFF, 0x7F])); // overlong: skipped + rec(&mut s, 0x0004, &[1, 2]); // short: skipped + rec(&mut s, 0x0003, &cell_rec(6, 0, &[0x07])); + rec(&mut s, 0x0004, &cell_rec(7, 0, &[1])); + rec( + &mut s, + 0x0002, + &cell_rec(8, 7, &((5u32 << 2) | 2).to_le_bytes()), + ); // style out of range + rec(&mut s, 0x0092, &[]); + rec(&mut s, 0x0005, &cell_rec(0, 0, &2.0f64.to_le_bytes())); // after: ignored + let got: Vec<_> = wb.cells(&s).map(|c| c.unwrap()).collect(); + assert_eq!( + got, + vec![ + (3, 1, Value::DateTime(45000.0)), + (3, 2, Value::Text("shared".into())), + (3, 4, Value::Text("inline".into())), + (3, 6, Value::Error("#DIV/0!")), + (3, 7, Value::Bool(true)), + (3, 8, Value::Number(5.0)), + ] + ); + } + + #[test] + fn workbook_sheets_resolve_relationships() { + let rels = vec![ + ( + "rId1".into(), + format!("http://x{WORKSHEET_REL}"), + "worksheets/sheet1.bin".into(), + ), + ( + "rId2".into(), + "http://x/relationships/chartsheet".into(), + "chartsheets/c.bin".into(), + ), + ( + "rId3".into(), + format!("http://x{WORKSHEET_REL}"), + "/xl/worksheets/s3.bin".into(), + ), + ]; + let sheet = |state: u32, rel: &str, name: &str| { + let mut d = state.to_le_bytes().to_vec(); + d.extend_from_slice(&0u32.to_le_bytes()); + d.extend(ws(rel)); + d.extend(ws(name)); + d + }; + let mut d = Vec::new(); + rec(&mut d, 0x0099, &1u32.to_le_bytes()); + rec(&mut d, 0x009C, &sheet(0, "rId1", "Data")); + rec(&mut d, 0x009C, &sheet(1, "rId2", "Chart")); + rec(&mut d, 0x009C, &sheet(2, "rId3", "Secret")); + rec(&mut d, 0x009C, &sheet(0, "rId9", "Dangling")); + rec(&mut d, 0x0090, &[]); + let mut wb = Workbook::default(); + read_workbook(&d, &rels, &mut wb).unwrap(); + assert!(wb.is_1904); + let parts: Vec<_> = wb.sheets.iter().map(|s| s.part.as_deref()).collect(); + assert_eq!( + parts, + vec![ + Some("xl/worksheets/sheet1.bin"), + None, + Some("xl/worksheets/s3.bin"), + None + ] + ); + assert_eq!(wb.sheets[2].visibility, Visibility::Hidden); + + let mut bad = Vec::new(); + rec(&mut bad, 0x009C, &[0, 0, 0, 0]); + assert!(read_workbook(&bad, &rels, &mut Workbook::default()).is_err()); + } + + // These call the reader directly, outside the `catch_unwind` guard, so a + // panic fails the test. + proptest::proptest! { + #![proptest_config(proptest::prelude::ProptestConfig::with_cases(512))] + + #[test] + fn arbitrary_parts_never_panic(data in proptest::collection::vec(proptest::prelude::any::(), 0..512)) { + let _ = records(&data).count(); + let _ = read_workbook(&data, &[], &mut Workbook::default()); + let _ = read_shared_strings(&data); + let _ = read_styles(&data); + let wb = Workbook { strings: vec!["s".into()], xf_kinds: vec![NumKind::DateTime], ..Default::default() }; + let mut sheet = Vec::new(); + rec(&mut sheet, 0x0091, &[]); + sheet.extend_from_slice(&data); + let _ = wb.cells(&sheet).count(); + } + + #[test] + fn arbitrary_cell_payloads_never_panic( + typ in 0u16..0x000C, + payload in proptest::collection::vec(proptest::prelude::any::(), 0..40), + ) { + let wb = Workbook { strings: vec!["s".into()], xf_kinds: vec![NumKind::Duration], ..Default::default() }; + let mut sheet = Vec::new(); + rec(&mut sheet, 0x0091, &[]); + rec(&mut sheet, typ, &payload); + let _ = wb.cells(&sheet).count(); + } + } +} diff --git a/crates/markkit/tests/common/mod.rs b/crates/markkit/tests/common/mod.rs index 375b814..7b7b9a0 100644 --- a/crates/markkit/tests/common/mod.rs +++ b/crates/markkit/tests/common/mod.rs @@ -484,6 +484,117 @@ pub fn xls(sheets: &[XlsSheet]) -> Vec { xls_from_stream(&biff8_workbook(sheets)) } +// ─── XLSB (BIFF12 records in an OPC ZIP) ──────────────────────────── + +/// Append a BIFF12 record (1–2 byte type, 7-bit varint size). +pub fn biff12(out: &mut Vec, typ: u16, data: &[u8]) { + if typ < 0x80 { + out.push(typ as u8); + } else { + out.push((typ & 0x7F) as u8 | 0x80); + out.push((typ >> 7) as u8); + } + let mut n = data.len(); + loop { + let b = (n & 0x7F) as u8; + n >>= 7; + if n == 0 { + out.push(b); + break; + } + out.push(b | 0x80); + } + out.extend_from_slice(data); +} + +/// `XLWideString`. +pub fn wide(s: &str) -> Vec { + let units: Vec = s.encode_utf16().collect(); + let mut v = (units.len() as u32).to_le_bytes().to_vec(); + v.extend(units.iter().flat_map(|u| u.to_le_bytes())); + v +} + +/// Worksheet part with inline strings (`BrtCellSt`) and numbers (`BrtCellReal`). +pub fn xlsb_sheet(rows: &[Vec]) -> Vec { + let mut d = Vec::new(); + biff12(&mut d, 0x0081, &[]); // BrtBeginSheet + biff12(&mut d, 0x0094, &[0; 16]); // BrtWsDim + biff12(&mut d, 0x0091, &[]); // BrtBeginSheetData + for (r, row) in rows.iter().enumerate() { + biff12(&mut d, 0x0000, &(r as u32).to_le_bytes()); + for (c, cell) in row.iter().enumerate() { + let mut rec = (c as u32).to_le_bytes().to_vec(); + rec.extend_from_slice(&0u32.to_le_bytes()); + match cell { + Cell::S(s) => { + rec.extend(wide(s)); + biff12(&mut d, 0x0006, &rec); + } + Cell::Str(s) => { + rec.extend(wide(s)); + biff12(&mut d, 0x0006, &rec); + } + Cell::N(n) => { + rec.extend_from_slice(&n.to_le_bytes()); + biff12(&mut d, 0x0005, &rec); + } + Cell::B(b) => { + rec.push(u8::from(*b)); + biff12(&mut d, 0x0004, &rec); + } + Cell::Empty => {} + } + } + } + biff12(&mut d, 0x0092, &[]); // BrtEndSheetData + biff12(&mut d, 0x0082, &[]); // BrtEndSheet + d +} + +/// Build an XLSB from sheets (hidden sheets honoured), plus extra raw entries. +pub fn xlsb_with(sheets: &[Sheet], extra: &[(&str, &[u8])]) -> Vec { + let mut wb = Vec::new(); + biff12(&mut wb, 0x0083, &[]); // BrtBeginBook + biff12(&mut wb, 0x008F, &[]); // BrtBeginBundleShs + let mut rels = String::new(); + let mut entries: Vec<(String, Vec)> = Vec::new(); + for (i, s) in sheets.iter().enumerate() { + let n = i + 1; + let mut rec = u32::from(s.hidden).to_le_bytes().to_vec(); + rec.extend_from_slice(&(n as u32).to_le_bytes()); + rec.extend(wide(&format!("rId{n}"))); + rec.extend(wide(&s.name)); + biff12(&mut wb, 0x009C, &rec); + rels.push_str(&format!(r#""#)); + entries.push((format!("xl/worksheets/sheet{n}.bin"), xlsb_sheet(&s.rows))); + } + biff12(&mut wb, 0x0090, &[]); // BrtEndBundleShs + biff12(&mut wb, 0x0084, &[]); // BrtEndBook + let rels = format!( + r#"{rels}"# + ); + let root_rels = r#""#; + let mut all: Vec<(String, Vec)> = vec![ + ("_rels/.rels".into(), root_rels.as_bytes().to_vec()), + ("xl/workbook.bin".into(), wb), + ("xl/_rels/workbook.bin.rels".into(), rels.into_bytes()), + ]; + all.extend(entries); + for (n, d) in extra { + all.push((n.to_string(), d.to_vec())); + } + let refs: Vec<(&str, &[u8])> = all + .iter() + .map(|(n, d)| (n.as_str(), d.as_slice())) + .collect(); + zip_bytes(&refs) +} + +pub fn xlsb(sheets: &[Sheet]) -> Vec { + xlsb_with(sheets, &[]) +} + // ─── PDF ──────────────────────────────────────────────────────────── /// Zlib-compress (for `/FlateDecode` streams). diff --git a/crates/markkit/tests/document_security.rs b/crates/markkit/tests/document_security.rs index ce64334..9e594db 100644 --- a/crates/markkit/tests/document_security.rs +++ b/crates/markkit/tests/document_security.rs @@ -24,6 +24,7 @@ fn hints() -> Vec> { DocumentHint::content_type(XLSX_CT), DocumentHint::content_type("application/vnd.ms-excel"), DocumentHint::content_type("application/vnd.oasis.opendocument.spreadsheet"), + DocumentHint::content_type("application/vnd.ms-excel.sheet.binary.macroEnabled.12"), ] } @@ -106,6 +107,57 @@ fn xlsx_zip_bomb_sheet_rejected() { assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); } +#[cfg(feature = "spreadsheet")] +#[test] +fn xlsb_zip_bomb_sheet_rejected() { + let mut rows = vec![vec![Cell::S("x")]]; + rows.extend((0..1).map(|_| vec![Cell::Str("y".repeat(4 * 1024 * 1024))])); + let bytes = xlsb(&[Sheet::new("S", rows)]); + assert!(bytes.len() < 256 * 1024); + let m = Markkit::builder() + .max_archive_entry_size(1024 * 1024) + .build(); + let err = m + .extract_document_sync(&bytes, DocumentHint::default()) + .unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); +} + +#[cfg(feature = "spreadsheet")] +#[test] +fn xlsb_huge_declared_dimensions_do_not_allocate() { + // BrtWsDim claims A1:XFD1048576; cells are streamed, never a dense grid. + let mut sheet = Vec::new(); + biff12(&mut sheet, 0x0081, &[]); + let mut dim = Vec::new(); + for v in [0u32, 1_048_575, 0, 16_383] { + dim.extend_from_slice(&v.to_le_bytes()); + } + biff12(&mut sheet, 0x0094, &dim); + biff12(&mut sheet, 0x0091, &[]); + for (row, col) in [(0u32, 0u32), (1_048_575, 16_383)] { + biff12(&mut sheet, 0x0000, &row.to_le_bytes()); + let mut c = col.to_le_bytes().to_vec(); + c.extend_from_slice(&0u32.to_le_bytes()); + c.extend(wide("far")); + biff12(&mut sheet, 0x0006, &c); + } + biff12(&mut sheet, 0x0092, &[]); + let base = xlsb(&[Sheet::new("S", vec![])]); + let bytes = rezip( + &base, + &["xl/worksheets/sheet1.bin"], + &[("xl/worksheets/sheet1.bin", &sheet)], + ); + let e = Markkit::builder() + .build() + .extract_document_sync(&bytes, DocumentHint::default()) + .unwrap(); + assert!(e.markdown.contains("far")); + assert!(e.markdown.len() < 64 * 1024); + assert!(e.warnings.iter().any(|w| w.starts_with("truncated_sheet:"))); +} + #[cfg(feature = "spreadsheet")] #[test] fn xlsx_bomb_in_unreferenced_entry_rejected() { @@ -337,6 +389,19 @@ fn valid_fixtures() -> Vec> { &[("docProps/core.xml", core)], ); v.push(rezip(&book, &["_rels/.rels"], &[])); + v.push(xlsb(&[Sheet::new( + "S", + vec![vec![Cell::S("a"), Cell::N(1.0)], vec![Cell::B(true)]], + )])); + for name in ["date", "issues"] { + v.push( + std::fs::read(format!( + "{}/tests/fixtures/xlsb/{name}.xlsb", + env!("CARGO_MANIFEST_DIR") + )) + .unwrap(), + ); + } } v } @@ -397,7 +462,7 @@ proptest! { /// Byte flips in valid documents never panic. #[test] fn mutated_fixtures_never_panic( - idx in 0usize..6, + idx in 0usize..9, flips in proptest::collection::vec((any::(), any::()), 1..16), ) { let fixtures = valid_fixtures(); diff --git a/crates/markkit/tests/fixtures/xlsb/README.md b/crates/markkit/tests/fixtures/xlsb/README.md new file mode 100644 index 0000000..e4cc917 --- /dev/null +++ b/crates/markkit/tests/fixtures/xlsb/README.md @@ -0,0 +1,8 @@ +# XLSB fixtures + +Real Excel binary workbooks, copied from the calamine test suite +(, MIT license, +Copyright 2016-2026 Johann Tuffe). + +- `date.xlsb`: dates and a `[h]:mm:ss` duration +- `issues.xlsb`: mixed types, shared strings, formulas, special characters, a VBA project diff --git a/crates/markkit/tests/fixtures/xlsb/date.xlsb b/crates/markkit/tests/fixtures/xlsb/date.xlsb new file mode 100644 index 0000000..4a206a2 Binary files /dev/null and b/crates/markkit/tests/fixtures/xlsb/date.xlsb differ diff --git a/crates/markkit/tests/fixtures/xlsb/issues.xlsb b/crates/markkit/tests/fixtures/xlsb/issues.xlsb new file mode 100644 index 0000000..5633201 Binary files /dev/null and b/crates/markkit/tests/fixtures/xlsb/issues.xlsb differ diff --git a/crates/markkit/tests/limits.rs b/crates/markkit/tests/limits.rs index 046240f..eb1239b 100644 --- a/crates/markkit/tests/limits.rs +++ b/crates/markkit/tests/limits.rs @@ -266,4 +266,21 @@ mod spreadsheet { assert!(limit_msg(run(&m, &sheet())).contains("max_output_size")); assert!(run(&Markkit::builder().build(), &sheet()).is_ok()); } + + #[test] + fn xlsb_zero_timeout_rejects() { + let rows = (0..200) + .map(|i| vec![Cell::Str(format!("row {i}")), Cell::N(i as f64)]) + .collect(); + let bytes = xlsb(&[Sheet::new("S", rows)]); + let hint = DocumentHint::file_name("t.xlsb"); + let m = Markkit::builder().timeout(Duration::ZERO).build(); + assert!(limit_msg(m.extract_document_sync(&bytes, hint)).contains("timeout")); + assert!( + Markkit::builder() + .build() + .extract_document_sync(&bytes, hint) + .is_ok() + ); + } } diff --git a/crates/markkit/tests/spreadsheet_integration.rs b/crates/markkit/tests/spreadsheet_integration.rs index bc9eb4b..457e593 100644 --- a/crates/markkit/tests/spreadsheet_integration.rs +++ b/crates/markkit/tests/spreadsheet_integration.rs @@ -385,13 +385,150 @@ fn xlsx_without_package_relationships_still_parses() { assert_eq!(e.markdown, "## S\n\n| h |\n| --- |\n| v |\n"); } +// ─── XLSB ─────────────────────────────────────────────────────────── + +const XLSB_CT: &str = "application/vnd.ms-excel.sheet.binary.macroEnabled.12"; + +fn fixture(name: &str) -> Vec { + std::fs::read(format!( + "{}/tests/fixtures/{name}", + env!("CARGO_MANIFEST_DIR") + )) + .unwrap() +} + #[test] fn xlsb_content_type_is_not_xlsx() { - let ct = "application/vnd.ms-excel.sheet.binary.macroEnabled.12"; - assert_eq!(Format::from_content_type(ct), Format::Xlsb); + assert_eq!(Format::from_content_type(XLSB_CT), Format::Xlsb); assert_eq!(Format::from_file_name("book.xlsb"), Format::Xlsb); } +#[test] +fn xlsb_generated_workbook() { + let bytes = xlsb(&[ + Sheet::new( + "People", + vec![ + vec![S("Name"), S("Age"), S("Member")], + vec![S("Ann"), N(30.0), B(true)], + vec![S("Bob|Jr"), N(41.5), B(false)], + ], + ), + Sheet::new("Empty", vec![]), + ]); + for hint in [ + DocumentHint::default(), + DocumentHint::content_type(XLSB_CT), + DocumentHint::file_name("book.xlsb"), + // Mislabelled: the container is sniffed. + DocumentHint::content_type(XLSX_CT), + ] { + let e = Markkit::builder() + .build() + .extract_document_sync(&bytes, hint) + .unwrap(); + assert_eq!(e.format, Format::Xlsb, "{hint:?}"); + assert_eq!( + e.markdown, + "## People\n\n| Name | Age | Member |\n| --- | --- | --- |\n| Ann | 30 | true |\n| Bob\\|Jr | 41.5 | false |\n" + ); + assert_eq!(e.stats.sheets, Some(1)); + assert_eq!(e.stats.rows, Some(2)); + } +} + +#[test] +fn xlsb_real_workbook_with_dates_and_duration() { + let e = extract(&fixture("xlsb/date.xlsb")); + assert_eq!(e.format, Format::Xlsb); + assert_eq!( + e.markdown, + "## Sheet1\n\n| 2021-01-01 | 15 |\n| --- | --- |\n| 2021-01-02 | 16 |\n| 255:10:10 | 17 |\n" + ); +} + +#[test] +fn xlsb_real_workbook_mixed_types_and_shared_strings() { + let e = extract(&fixture("xlsb/issues.xlsb")); + assert!(e.warnings.is_empty(), "{:?}", e.warnings); + assert!(e.markdown.starts_with( + "## datatypes\n\n| 1 |\n| --- |\n| 1.5 |\n| ab |\n| false |\n| test |\n| 2016-10-20 |\n" + )); + assert!( + e.markdown + .contains("## issue2\n\n| 1 | a |\n| --- | --- |\n| 2 | b |\n| 3 | c |\n") + ); + assert!(e.markdown.contains("| ☺ |")); + assert!(e.markdown.contains("| àâéêèçöïî«» |")); + // VBA project is present in the file but never rendered. + assert!(!e.markdown.to_lowercase().contains("vba")); +} + +#[test] +fn xlsb_hidden_sheet_skipped_by_default() { + let mut hidden = Sheet::new("Secret", vec![vec![S("h")], vec![S("pw")]]); + hidden.hidden = true; + let bytes = xlsb(&[Sheet::new("Open", vec![vec![S("h")], vec![S("v")]]), hidden]); + let e = extract(&bytes); + assert!(!e.markdown.contains("Secret"), "{}", e.markdown); + assert!( + e.warnings + .iter() + .any(|w| w.starts_with("hidden_sheet_skipped:")) + ); + let m = Markkit::builder().include_hidden_sheets(true).build(); + assert!(extract_with(&m, &bytes).markdown.contains("## Secret")); +} + +#[test] +fn xlsb_caps_and_title() { + let rows: Vec> = (0..20).map(|i| vec![N(f64::from(i)); 10]).collect(); + let core = b"Binary Book"; + let bytes = xlsb_with(&[Sheet::new("Big", rows)], &[("docProps/core.xml", core)]); + let m = Markkit::builder() + .max_sheet_rows(3) + .max_sheet_cols(2) + .build(); + let e = extract_with(&m, &bytes); + assert_eq!(e.title.as_deref(), Some("Binary Book")); + assert_eq!(e.stats.rows, Some(3)); + assert_eq!( + e.warnings + .iter() + .filter(|w| w.starts_with("truncated_sheet:")) + .count(), + 2, + "{:?}", + e.warnings + ); +} + +#[test] +fn xlsb_missing_and_truncated_sheet_parts_warn() { + let bytes = xlsb(&[Sheet::new("S", vec![vec![S("h")], vec![S("v")]])]); + // Sheet part removed. + let gone = rezip(&bytes, &["xl/worksheets/sheet1.bin"], &[]); + let e = extract(&gone); + assert!( + e.warnings.iter().any(|w| w.starts_with("sheet_error:")), + "{:?}", + e.warnings + ); + // Sheet part cut mid-record (the end records take 6 bytes). + let sheet = xlsb_sheet(&[vec![S("h")], vec![S("value")]]); + let cut = rezip( + &bytes, + &["xl/worksheets/sheet1.bin"], + &[("xl/worksheets/sheet1.bin", &sheet[..sheet.len() - 8])], + ); + let e = extract(&cut); + assert!( + e.warnings.iter().any(|w| w.starts_with("sheet_error:")), + "{:?}", + e.warnings + ); +} + // ─── Provenance ────────────────────────────────────────────────────── #[test] diff --git a/knowledge/formats/format-support.md b/knowledge/formats/format-support.md index c464f4a..7b8c3b3 100644 --- a/knowledge/formats/format-support.md +++ b/knowledge/formats/format-support.md @@ -23,11 +23,13 @@ tags: ### Tier 2 (Feature-gated) - **DOCX** (`docx`): Word documents via a direct `zip` + `quick-xml` reader (no docx-rs) - **XLSX/XLSM, XLS, ODS** (`spreadsheet`): Workbooks to Markdown tables via calamine +- **XLSB** (`spreadsheet`): Excel binary workbooks via markkit's own bounds-checked BIFF12 + reader (calamine's XLSB reader panics on malformed records) - **Markdown** (`markdown`): Re-parsed with pulldown-cmark into the IR and rendered back (`text/markdown`, `text/x-markdown`, `.md`, `.markdown`, `.mdown`, `.mkd`, `.mkdn`) - **PDF** (`pdf`): Text layer via hayro-syntax (pure Rust); no OCR, encrypted PDFs rejected ### Detected, not supported -- **DOC** (Word 97-2003 binary), **PPTX**, **XLSB**: explicit `UnsupportedFormat` errors +- **DOC** (Word 97-2003 binary), **PPTX**: explicit `UnsupportedFormat` errors ### Tier 3 (Planned) - **EPUB**: E-book extraction @@ -63,7 +65,7 @@ concatenate cleanly); an empty document renders `""`. This is intended and stabl |---|---| | HTML | `og:title` / JSON-LD / `` (see [HTML Extraction](html-extraction.md)) | | DOCX | `docProps/core.xml` `dc:title` (via `_rels/.rels`), else the first `Title`-style paragraph | -| XLSX | `docProps/core.xml` `dc:title` | +| XLSX, XLSB | `docProps/core.xml` `dc:title` | | ODS | `meta.xml` `dc:title` | | XLS | `\u{5}SummaryInformation` property set, `PIDSI_TITLE` (code page aware: 1200 UTF-16, UTF-8, else Latin-1) | | PDF | `/Info /Title` (UTF-16BE with BOM, else PDFDocEncoding) | @@ -134,6 +136,11 @@ numbering through the main part's relationships. emits a `truncated_sheet` warning. - Hidden / very hidden sheets are skipped with `hidden_sheet_skipped` unless `include_hidden_sheets(true)`. Chart, dialog and macro sheets are skipped. +- XLSB: sheets from `xl/workbook.bin` (`BrtBundleSh`) resolved through + `xl/_rels/workbook.bin.rels`; shared strings from `xl/sharedStrings.bin`; dates and + durations from the cell XF's number format in `xl/styles.bin` (built-in date ids or custom + codes with date/time tokens; `[h]`-style elapsed codes are durations). A missing or + truncated sheet part emits `sheet_error` and the other sheets still render. - Only cached cell values are read: formulas are not evaluated, VBA/XLM macros are never executed or rendered. diff --git a/knowledge/log.md b/knowledge/log.md index 6c58dd7..58ade41 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,6 +2,7 @@ ## 2026-10-27 +* **XLSB support (#11)**: Excel binary workbooks are read by a new bounds-checked BIFF12 reader (`spreadsheet/xlsb.rs`) instead of being rejected; calamine's XLSB reader was not used because it panics on malformed records. Same caps, hidden-sheet and macro rules as XLSX. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md) T6c. * **MCP server**: `markkit mcp` serves `MarkkitTool` over MCP stdio (initialize, ping, tools/list, tools/call); line size capped from `--max-size`, tool failures as `isError` results. New [MCP Server](integrations/mcp-server.md) concept; MCP messages row in the [Threat Model](security/threat-model.md). * **Spreadsheet follow-ups (#11)**: single-row sheets render as data under column letters; XLSX/ODS/XLS titles come from document properties; XLSB content type no longer maps to XLSX; XLSX without `_rels/.rels` parses; trailing-newline output documented as intended; release profile switched to `panic = "unwind"` so `catch_unwind` is effective in markkit's own binaries. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md) T6c. * **Plain text and Markdown input**: text is always on and goes through the IR; Markdown (`markdown` feature) is re-parsed with pulldown-cmark rather than passed through, so raw HTML never reaches output. Neither is sniffed from bytes. Shares the call's `Limits` budget (timeout, nodes, nesting depth). New threat T7 and invariant S18. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md). diff --git a/knowledge/operations/testing.md b/knowledge/operations/testing.md index d3f2dfe..eddf07e 100644 --- a/knowledge/operations/testing.md +++ b/knowledge/operations/testing.md @@ -27,7 +27,7 @@ tags: | `crates/markkit/tests/limits.rs` | Every `Limits` field enforced for HTML, DOCX, spreadsheets; small-stack depth ceiling | | `crates/markkit/tests/pdf_integration.rs` | PDF text layout, fonts/encodings, forms, page cap | | `crates/markkit/tests/pdf_security.rs` | PDF bombs, interpretation budget, encrypted/unverifiable input, malformed PDFs | -| `crates/markkit/tests/common/` | In-memory DOCX/XLSX/XLS/ODS/PDF fixture builders | +| `crates/markkit/tests/common/` | In-memory DOCX/XLSX/XLSB/XLS/ODS/PDF fixture builders; `fixtures/xlsb/` holds two real XLSB files from calamine (MIT) | | `crates/markkit-cli/tests/cli_integration.rs` | CLI flags and output | | `crates/markkit-cli/tests/mcp_integration.rs` | `markkit mcp` stdio session end to end | | `crates/markkit/benches/extraction.rs` | Criterion micro-benchmarks | diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 5a2c934..2eaa002 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -224,7 +224,7 @@ the proxy, or use `allow_host`. - `html_security::very_long_text_node` (100KB text node) - `html_proptest::no_panic_on_arbitrary_input` -### T6: Malicious Office Documents (DOCX / XLSX / XLS / ODS) +### T6: Malicious Office Documents (DOCX / XLSX / XLSB / XLS / ODS) **Description**: Crafted containers aimed at exhausting memory/CPU, crashing the process, or smuggling content past a human reviewer. @@ -263,6 +263,12 @@ process, or smuggling content past a human reviewer. `panic = "unwind"`, which the workspace release profile uses so the CLI and the Python/Node bindings survive a parser panic; embedders building with `abort` lose this layer). +- XLSB is parsed by markkit's own BIFF12 reader, not calamine's (which indexes record + buffers and the shared-string table unchecked). Every field read is bounds-checked: + record sizes against the part, wide-string lengths with checked arithmetic, shared-string + and XF indexes by lookup. Malformed cell records are skipped; a truncated part fails that + sheet with `sheet_error`. Cells stream into the capped grid, so `BrtWsDim` claims never + allocate. Proptests drive the reader directly, outside `catch_unwind`. - XLS `SummaryInformation` title parsing is bounds-checked (1 MiB stream cap, checked offsets, 1024-property cap); failures yield no title, never an error. @@ -290,6 +296,10 @@ process, or smuggling content past a human reviewer. `spreadsheet_integration::xlsx_sparse_far_cell_does_not_allocate_grid` - `docx_billion_laughs_not_expanded`, `docx_external_entity_not_resolved`, `docx_deep_nesting_does_not_overflow_stack` +- `xlsb_zip_bomb_sheet_rejected`, `xlsb_huge_declared_dimensions_do_not_allocate`, + `spreadsheet::xlsb::tests::arbitrary_parts_never_panic`, + `spreadsheet::xlsb::tests::arbitrary_cell_payloads_never_panic`, + `spreadsheet_integration::xlsb_hidden_sheet_skipped_by_default` - `spreadsheet::tests::summary_title_rejects_malformed`, `spreadsheet_integration::xlsx_malformed_core_properties_do_not_fail_extraction` - `truncated_fixtures_never_panic`, `random_bytes_never_panic`,