diff --git a/CHANGELOG.md b/CHANGELOG.md index 132f985..75e9801 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/). - `markkit mcp`: Model Context Protocol server over stdio exposing the `markkit` tool (`tools/list`, `tools/call`). URL fetching keeps the SSRF guard; `--no-url` keeps it offline, the `extract` limit flags and `--max-output-chars` bound each call. +- CSV and TSV input (`text/csv`, `text/tab-separated-values`, `.csv`, `.tsv`), always on: one GFM + table, RFC 4180 quoting, `;`/`|`/tab delimiter detection, bounded by `max_sheet_rows` / + `max_sheet_cols` (`truncated_table` warning). New `Format::Tsv`. - Plain text input (`text/plain`, `.txt`), always on: paragraphs and hard breaks through the IR, UTF-8/UTF-16 BOM decoding, control characters dropped, `invalid_utf8` warning on lossy decode. - Markdown input (`markdown` feature, in `all_formats`; `text/markdown`, `.md`): re-parsed with diff --git a/README.md b/README.md index d8d31a9..dbfb1f6 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ package for browsers and edge runtimes. - **HTML** – Web pages and HTML documents; nav, header, footer, scripts, styles and forms stripped by default - **DOCX** – Headings, lists, tables, links, emphasis - **Spreadsheets** – XLSX/XLSM, XLSB, XLS and ODS workbooks to markdown tables, one section per sheet +- **CSV / TSV** – One markdown table; `;` and `|` delimiters detected, RFC 4180 quoting - **Plain text / Markdown** – Text to paragraphs; Markdown normalized through the same IR (raw HTML never passed through) - **Format detection** – Content type, then file extension, then magic bytes - **URL fetching** – Fetch and extract in one call (`http_client` feature) @@ -34,6 +35,7 @@ package for browsers and edge runtimes. | DOCX | Supported | `docx` | | XLSX, XLSM, XLSB, XLS, ODS | Supported | `spreadsheet` | | Plain text | Supported | always on | +| CSV, TSV | Supported | always on | | Markdown | Supported | `markdown` | | PDF (text layer) | Supported | `pdf` | | DOC, PPTX | Detected, rejected explicitly | – | diff --git a/crates/markkit-cli/tests/cli_integration.rs b/crates/markkit-cli/tests/cli_integration.rs index 4692455..3d3cdd9 100644 --- a/crates/markkit-cli/tests/cli_integration.rs +++ b/crates/markkit-cli/tests/cli_integration.rs @@ -117,6 +117,11 @@ fn extract_text_and_markdown_files() { "Title\n=====\n\n* a\n\n\n", "# Title\n\n- a\n", ), + ( + "data.csv", + "a;b\n1;2\n", + "| a | b |\n| --- | --- |\n| 1 | 2 |\n", + ), ]; for (name, input, expected) in cases { let file_path = dir.path().join(name); diff --git a/crates/markkit-js/src/lib.rs b/crates/markkit-js/src/lib.rs index 8b05e1c..7fe7fbc 100644 --- a/crates/markkit-js/src/lib.rs +++ b/crates/markkit-js/src/lib.rs @@ -93,7 +93,7 @@ pub struct Metadata { pub struct Extraction { pub markdown: String, pub title: Option, - /// Detected format (`html`, `docx`, `xlsx`, `xls`, `ods`, `text`, `markdown`). + /// Detected format (`html`, `docx`, `xlsx`, `xls`, `ods`, `text`, `markdown`, `csv`, `tsv`). pub format: String, pub warnings: Vec, pub stats: ExtractionStats, diff --git a/crates/markkit-wasm/README.md b/crates/markkit-wasm/README.md index dcc03bd..9459c4d 100644 --- a/crates/markkit-wasm/README.md +++ b/crates/markkit-wasm/README.md @@ -1,8 +1,8 @@ # @everruns/markkit-wasm Markkit compiled to WebAssembly: extract clean Markdown from HTML, DOCX, -spreadsheets (XLSX/XLSM, XLS, ODS), plain text and Markdown in the **browser -and any JavaScript runtime**: +spreadsheets (XLSX/XLSM, XLS, ODS), CSV/TSV, plain text and Markdown in the +**browser and any JavaScript runtime**: edge workers (Cloudflare Workers, Vercel Edge, Deno Deploy), Node, Deno and Bun. Single-threaded `wasm-bindgen` build: no `SharedArrayBuffer`, no `COOP`/`COEP` diff --git a/crates/markkit-wasm/__test__/markkit-wasm.test.mjs b/crates/markkit-wasm/__test__/markkit-wasm.test.mjs index 0458d6e..85800ad 100644 --- a/crates/markkit-wasm/__test__/markkit-wasm.test.mjs +++ b/crates/markkit-wasm/__test__/markkit-wasm.test.mjs @@ -85,6 +85,15 @@ test("extractDocument reads plain text and Markdown", () => { assert.ok(!md.markdown.includes("script")); }); +test("extractDocument reads CSV and TSV", () => { + const mk = new Markkit(); + const csv = mk.extractDocument(new TextEncoder().encode("a;b\n1;2\n"), { fileName: "t.csv" }); + assert.equal(csv.format, "csv"); + assert.equal(csv.markdown, "| a | b |\n| --- | --- |\n| 1 | 2 |\n"); + const tsv = mk.extractDocument(new TextEncoder().encode("x\ty\n"), { fileName: "t.tsv" }); + assert.equal(tsv.format, "tsv"); +}); + test("extractBytes and unsupported formats", () => { const mk = new Markkit(); assert.equal(mk.extractBytes(new TextEncoder().encode("

x

"), "text/html"), "x\n"); diff --git a/crates/markkit/src/csv.rs b/crates/markkit/src/csv.rs new file mode 100644 index 0000000..4c648bc --- /dev/null +++ b/crates/markkit/src/csv.rs @@ -0,0 +1,257 @@ +//! CSV / TSV input → one GFM table. +//! +//! Decisions: +//! - Always compiled, hand-written RFC 4180 reader (no dependency). Like plain +//! text, selected only by content type or extension, never sniffed. +//! - Delimiter: tab for TSV; for CSV the candidate (`,` `;` `\t` `|`) seen most +//! often outside quotes in the first record, default `,`. +//! - Quoted fields may span lines and use `""` escapes. A stray quote inside an +//! unquoted field is literal; an unterminated quote takes the rest of the +//! input as one field and emits `malformed_csv`. +//! - First non-empty record is the header; fully empty records are dropped; +//! short rows are padded. Cells are trimmed and kept raw in the IR; the +//! renderer escapes pipes and flattens line breaks in table context (S17). +//! - Bounded by `max_sheet_rows` / `max_sheet_cols`: the reader stops after the +//! row cap and never stores columns past the column cap (`truncated_table`). +//! Each record ticks the call's `timeout` budget. + +use crate::config::MarkkitBuilder; +use crate::ir::{Document, Node, TableCell}; +use crate::limits::Budget; + +/// Parsed CSV/TSV document. +pub(crate) struct CsvOutput { + pub doc: Document, + pub warnings: Vec, + /// Data rows (header excluded). + pub rows: usize, + /// 1 when a table was produced, else 0. + pub tables: usize, +} + +/// Parse delimited text. `tab` forces the TSV delimiter. +pub(crate) fn parse( + text: &str, + tab: bool, + config: &MarkkitBuilder, + budget: &Budget, +) -> crate::Result { + let delim = if tab { '\t' } else { sniff_delimiter(text) }; + let max_cols = config.limits.max_sheet_cols.max(1); + let max_rows = config.limits.max_sheet_rows; + let mut warnings = Vec::new(); + + let mut reader = Reader::new(text, delim, max_cols); + let mut records: Vec> = Vec::new(); + let mut truncated_rows = false; + while let Some(rec) = reader.next_record() { + if budget.expired() { + return Err(budget.timeout_error()); + } + if rec.iter().all(|f| f.trim().is_empty()) { + continue; + } + // Header + `max_rows` data rows. + if records.len() > max_rows { + truncated_rows = true; + break; + } + records.push(rec); + } + if reader.unterminated { + warnings.push("malformed_csv: unterminated quoted field".into()); + } + if truncated_rows { + warnings.push(format!( + "truncated_table: table truncated to {max_rows} rows" + )); + } + if reader.truncated_cols { + warnings.push(format!( + "truncated_table: table truncated to {max_cols} columns" + )); + } + + let width = records.iter().map(Vec::len).max().unwrap_or(0); + let mut rows = records.into_iter().map(|rec| { + let mut cells: Vec = rec + .iter() + .map(|f| { + let f = f.trim(); + TableCell::new(if f.is_empty() { + vec![] + } else { + vec![Node::Text(f.to_string())] + }) + }) + .collect(); + cells.resize_with(width, || TableCell::new(vec![])); + cells + }); + let Some(headers) = rows.next() else { + return Ok(CsvOutput { + doc: Document::new(vec![]), + warnings, + rows: 0, + tables: 0, + }); + }; + let rows: Vec<_> = rows.collect(); + let n = rows.len(); + Ok(CsvOutput { + doc: Document::new(vec![Node::Table { + alignments: Vec::new(), + headers, + rows, + source: None, + }]), + warnings, + rows: n, + tables: 1, + }) +} + +fn sniff_delimiter(text: &str) -> char { + const CANDIDATES: [char; 4] = [',', ';', '\t', '|']; + let mut counts = [0usize; 4]; + let mut quoted = false; + let mut seen = false; + for c in text.chars() { + match c { + '"' => quoted = !quoted, + '\n' if !quoted => { + if seen { + break; + } + } + _ if !quoted => { + if let Some(i) = CANDIDATES.iter().position(|d| *d == c) { + counts[i] += 1; + } + if !c.is_whitespace() { + seen = true; + } + } + _ => {} + } + } + // First maximum wins, so ties prefer `,`. + let mut best = 0; + for i in 1..CANDIDATES.len() { + if counts[i] > counts[best] { + best = i; + } + } + CANDIDATES[best] +} + +struct Reader<'a> { + chars: std::iter::Peekable>, + delim: char, + max_cols: usize, + unterminated: bool, + truncated_cols: bool, +} + +impl<'a> Reader<'a> { + fn new(text: &'a str, delim: char, max_cols: usize) -> Self { + Self { + chars: text.chars().peekable(), + delim, + max_cols, + unterminated: false, + truncated_cols: false, + } + } + + /// Next record, or `None` at end of input. Input is LF-normalized. + fn next_record(&mut self) -> Option> { + self.chars.peek()?; + let mut rec = Vec::new(); + let mut field = String::new(); + let mut quoted = false; + let mut at_start = true; + loop { + let Some(c) = self.chars.next() else { + if quoted { + self.unterminated = true; + } + self.push_field(&mut rec, &mut field); + return Some(rec); + }; + if quoted { + if c == '"' { + if self.chars.peek() == Some(&'"') { + self.chars.next(); + field.push('"'); + } else { + quoted = false; + } + } else { + field.push(c); + } + continue; + } + if c == self.delim { + self.push_field(&mut rec, &mut field); + at_start = true; + } else if c == '\n' { + self.push_field(&mut rec, &mut field); + return Some(rec); + } else if c == '"' && at_start && field.trim().is_empty() { + field.clear(); + quoted = true; + at_start = false; + } else { + field.push(c); + if !c.is_whitespace() { + at_start = false; + } + } + } + } + + fn push_field(&mut self, rec: &mut Vec, field: &mut String) { + if rec.len() < self.max_cols { + rec.push(std::mem::take(field)); + } else { + self.truncated_cols = true; + field.clear(); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn records(text: &str, delim: char) -> Vec> { + let mut r = Reader::new(text, delim, 100); + std::iter::from_fn(|| r.next_record()).collect() + } + + #[test] + fn reader_quotes_and_escapes() { + assert_eq!( + records("a,\"b,c\",\"d \"\"q\"\"\"\n\"multi\nline\",x\"y,\n", ','), + vec![vec!["a", "b,c", "d \"q\""], vec!["multi\nline", "x\"y", ""],] + ); + } + + #[test] + fn reader_unterminated_quote() { + let mut r = Reader::new("a,\"open\nrest", ',', 100); + assert_eq!(r.next_record().unwrap(), vec!["a", "open\nrest"]); + assert!(r.unterminated); + assert!(r.next_record().is_none()); + } + + #[test] + fn delimiter_sniffing() { + assert_eq!(sniff_delimiter("a;b;c\n1,5;2;3"), ';'); + assert_eq!(sniff_delimiter("\"x;y\",b\n"), ','); + assert_eq!(sniff_delimiter("a\tb\n"), '\t'); + assert_eq!(sniff_delimiter("a|b|c\n"), '|'); + assert_eq!(sniff_delimiter("\n\nsingle\n"), ','); + } +} diff --git a/crates/markkit/src/extract.rs b/crates/markkit/src/extract.rs index 9f7dd28..9cab775 100644 --- a/crates/markkit/src/extract.rs +++ b/crates/markkit/src/extract.rs @@ -22,7 +22,8 @@ pub struct Extraction { pub format: Format, /// Non-fatal issues, each prefixed with a stable code /// (`truncated_sheet:`, `hidden_sheet_skipped:`, `sheet_error:`, - /// `truncated_pdf:`, `pdf_unmapped_font:`, `pdf_no_text:`, `invalid_utf8:`). + /// `truncated_pdf:`, `pdf_unmapped_font:`, `pdf_no_text:`, `invalid_utf8:`, + /// `malformed_csv:`, `truncated_table:`). pub warnings: Vec, /// Format-specific counters. pub stats: ExtractionStats, @@ -65,11 +66,11 @@ pub struct ExtractionStats { pub pages: Option, /// Sheets rendered (spreadsheets; empty and skipped sheets excluded). pub sheets: Option, - /// Data rows rendered across all sheets (header rows excluded). + /// Data rows rendered across all sheets / the CSV table (header rows excluded). pub rows: Option, /// Non-empty paragraphs, headings and list items (DOCX, plain text, Markdown). pub paragraphs: Option, - /// Tables (DOCX, Markdown). + /// Tables (DOCX, Markdown, CSV/TSV). pub tables: Option, } @@ -230,7 +231,8 @@ impl Markkit { .map(|e| e.markdown) } - /// Convert a document (HTML, PDF, DOCX, XLSX/XLSB/XLS/ODS, plain text, Markdown) to Markdown with metadata. + /// Convert a document (HTML, PDF, DOCX, XLSX/XLSB/XLS/ODS, plain text, Markdown, CSV/TSV) + /// to Markdown with metadata. /// /// The format is detected from `hint.content_type`, then the extension of /// `hint.file_name`, then magic bytes. Parsing is CPU-bound and runs inline; @@ -310,7 +312,8 @@ impl Markkit { ) -> crate::Result { self.config.limits.check_input(bytes.len())?; - match Format::detect(bytes, hint) { + let format = Format::detect(bytes, hint); + match format { Format::Html => self.parse_html_doc(bytes, budget), Format::Pdf => self.parse_pdf(bytes, budget), Format::Docx => self.parse_docx(bytes, budget), @@ -319,6 +322,7 @@ impl Markkit { } Format::PlainText => self.parse_plain_text(bytes, budget), Format::Markdown => self.parse_markdown(bytes, budget), + Format::Csv | Format::Tsv => self.parse_csv(bytes, format, budget), Format::Doc => Err(Error::UnsupportedFormat( "legacy Word 97-2003 (.doc) documents are not supported".into(), )), @@ -361,6 +365,24 @@ impl Markkit { Ok(p) } + fn parse_csv( + &self, + bytes: &[u8], + format: Format, + budget: &Budget, + ) -> crate::Result { + let decoded = crate::text::decode(bytes); + let out = crate::csv::parse(&decoded.text, format == Format::Tsv, &self.config, budget)?; + let mut doc = out.doc; + self.config.transforms.apply(&mut doc)?; + let mut p = ParsedDocument::new(doc, format); + p.warnings = decoded.warnings; + p.warnings.extend(out.warnings); + p.stats.rows = Some(out.rows); + p.stats.tables = Some(out.tables); + Ok(p) + } + #[cfg(feature = "markdown")] fn parse_markdown(&self, bytes: &[u8], budget: &Budget) -> crate::Result { let decoded = crate::text::decode(bytes); diff --git a/crates/markkit/src/format.rs b/crates/markkit/src/format.rs index fea7b95..614c363 100644 --- a/crates/markkit/src/format.rs +++ b/crates/markkit/src/format.rs @@ -30,8 +30,11 @@ pub enum Format { Ods, /// Office Open XML presentation (`.pptx`). Detected but not supported. Pptx, + /// Comma (or `;` / `|`) separated values (`.csv`). Csv, - /// Plain text (`text/plain`, `.txt`). Never sniffed from bytes. + /// Tab separated values (`.tsv`, `.tab`). + Tsv, + /// Plain text (`text/plain`, `.txt`). Never sniffed from bytes (nor CSV/TSV/Markdown). /// Serialized as `"text"`, same as `as_str`. #[serde(rename = "text")] PlainText, @@ -87,6 +90,7 @@ impl Format { Format::Ods => "ods", Format::Pptx => "pptx", Format::Csv => "csv", + Format::Tsv => "tsv", Format::PlainText => "text", Format::Markdown => "markdown", Format::Unknown => "unknown", @@ -115,7 +119,9 @@ impl Format { Format::Ods } else if ct.contains("presentationml") || ct.contains("ms-powerpoint") { Format::Pptx - } else if ct.contains("text/csv") { + } else if ct.contains("tab-separated-values") { + Format::Tsv + } else if ct.contains("text/csv") || ct.contains("application/csv") { Format::Csv } else if ct.contains("text/plain") { Format::PlainText @@ -138,7 +144,8 @@ impl Format { "xls" | "xlt" => Format::Xls, "ods" => Format::Ods, "pptx" | "pptm" | "ppt" => Format::Pptx, - "csv" | "tsv" => Format::Csv, + "csv" => Format::Csv, + "tsv" | "tab" => Format::Tsv, "txt" | "text" => Format::PlainText, "md" | "markdown" | "mdown" | "mkd" | "mkdn" => Format::Markdown, _ => Format::Unknown, diff --git a/crates/markkit/src/lib.rs b/crates/markkit/src/lib.rs index edb7249..ab35a0f 100644 --- a/crates/markkit/src/lib.rs +++ b/crates/markkit/src/lib.rs @@ -35,6 +35,7 @@ //! - **DOCX** (`docx`): Convert Word documents //! - **Spreadsheet** (`spreadsheet`): Convert XLSX/XLSM, XLSB, XLS and ODS workbooks to tables //! - **Plain text** (always on): `text/plain`, `.txt` +//! - **CSV / TSV** (always on): `text/csv`, `text/tab-separated-values`, `.csv`, `.tsv`, one table //! - **Markdown** (`markdown`): `text/markdown`, `.md`, normalized through the IR //! - **PDF** (`pdf`): Text layer of PDF documents (no OCR); encrypted PDFs are //! rejected with [`Error::UnsupportedFormat`] @@ -50,6 +51,7 @@ #[cfg(any(feature = "docx", feature = "spreadsheet"))] mod archive; pub mod config; +mod csv; #[cfg(feature = "docx")] mod docx; pub mod error; diff --git a/crates/markkit/tests/csv_integration.rs b/crates/markkit/tests/csv_integration.rs new file mode 100644 index 0000000..846da61 --- /dev/null +++ b/crates/markkit/tests/csv_integration.rs @@ -0,0 +1,208 @@ +//! CSV / TSV → Markdown table integration and security tests. + +use markkit::format::Format; +use markkit::{DocumentHint, Error, Markkit}; + +fn extract_with(m: &Markkit, bytes: &[u8], hint: DocumentHint<'_>) -> markkit::Extraction { + m.extract_document_sync(bytes, hint).unwrap() +} + +fn extract(bytes: &[u8], hint: DocumentHint<'_>) -> markkit::Extraction { + extract_with(&Markkit::builder().build(), bytes, hint) +} + +#[test] +fn csv_basic_table() { + let e = extract( + b"name,qty,note\nApple,3,\"red, sweet\"\nPear,10\n", + DocumentHint::file_name("fruit.csv"), + ); + assert_eq!(e.format, Format::Csv); + assert_eq!( + e.markdown, + "| name | qty | note |\n| --- | --- | --- |\n| Apple | 3 | red, sweet |\n| Pear | 10 | |\n" + ); + assert_eq!(e.stats.rows, Some(2)); + assert_eq!(e.stats.tables, Some(1)); + assert!(e.warnings.is_empty()); +} + +#[test] +fn tsv_by_extension_and_content_type() { + for hint in [ + DocumentHint::file_name("data.tsv"), + DocumentHint::content_type("text/tab-separated-values"), + ] { + let e = extract(b"a\tb,c\n1\t2\n", hint); + assert_eq!(e.format, Format::Tsv); + assert_eq!(e.markdown, "| a | b,c |\n| --- | --- |\n| 1 | 2 |\n"); + } +} + +#[test] +fn csv_content_types() { + for ct in ["text/csv; charset=utf-8", "application/csv"] { + let e = extract(b"a\n1", DocumentHint::content_type(ct)); + assert_eq!(e.format, Format::Csv, "{ct}"); + } +} + +#[test] +fn csv_semicolon_delimiter_sniffed() { + let e = extract( + b"\xEF\xBB\xBFcity;price\r\nKyiv;1,5\r\n", + DocumentHint::file_name("eu.csv"), + ); + assert_eq!( + e.markdown, + "| city | price |\n| --- | --- |\n| Kyiv | 1,5 |\n" + ); +} + +#[test] +fn csv_cells_escaped_and_flattened() { + let e = extract( + b"h\n\"a|b\nc\"\n\n,,\n spaced \n", + DocumentHint::file_name("x.csv"), + ); + assert_eq!(e.markdown, "| h |\n| --- |\n| a\\|b c |\n| spaced |\n"); +} + +#[test] +fn csv_empty_input() { + for input in [&b""[..], b"\n\n", b",,,\n , \n"] { + let e = extract(input, DocumentHint::file_name("e.csv")); + assert_eq!(e.markdown, ""); + assert_eq!(e.stats.tables, Some(0)); + assert_eq!(e.stats.rows, Some(0)); + } +} + +#[test] +fn csv_never_sniffed_without_hint() { + let err = Markkit::builder() + .build() + .extract_document_sync(b"a,b\n1,2\n", DocumentHint::default()) + .unwrap_err(); + assert!(matches!(err, Error::UnsupportedFormat(_)), "{err:?}"); +} + +// ─── Security (threat model T7) ───────────────────────────────────── + +#[test] +fn security_row_cap_stops_reading() { + let m = Markkit::builder().max_sheet_rows(3).build(); + let input = "h\n".to_string() + &"r\n".repeat(1_000_000); + let e = extract_with(&m, input.as_bytes(), DocumentHint::file_name("a.csv")); + assert_eq!(e.stats.rows, Some(3)); + assert_eq!(e.markdown.lines().count(), 5); + assert_eq!( + e.warnings, + vec!["truncated_table: table truncated to 3 rows"] + ); +} + +#[test] +fn security_column_cap_never_stores_extra_fields() { + let m = Markkit::builder().max_sheet_cols(2).build(); + let input = ",".repeat(5_000_000); + let e = extract_with( + &m, + format!("a{input}x").as_bytes(), + DocumentHint::file_name("a.csv"), + ); + assert_eq!(e.markdown, "| a | |\n| --- | --- |\n"); + assert_eq!( + e.warnings, + vec!["truncated_table: table truncated to 2 columns"] + ); +} + +#[test] +fn security_default_caps_bound_output() { + // 300 columns x 20 000 rows: defaults keep 256 columns and 10 000 rows. + let row = "x,".repeat(299) + "x\n"; + let input = row.repeat(20_001); + let e = extract(input.as_bytes(), DocumentHint::file_name("big.csv")); + assert_eq!(e.stats.rows, Some(10_000)); + let header = e.markdown.lines().next().unwrap(); + assert_eq!(header.matches(" x ").count(), 256); + assert_eq!(e.warnings.len(), 2); +} + +#[test] +fn security_unterminated_quote_warns() { + let input = format!("a,\"{}", "x\n".repeat(100_000)); + let e = extract(input.as_bytes(), DocumentHint::file_name("a.csv")); + assert_eq!(e.markdown.lines().count(), 2); + assert_eq!(e.warnings, vec!["malformed_csv: unterminated quoted field"]); +} + +#[test] +fn security_input_size_limit() { + let m = Markkit::builder().max_input_size(4).build(); + let err = m + .extract_document_sync(b"a,b,c", DocumentHint::file_name("a.csv")) + .unwrap_err(); + assert!(matches!(err, Error::ResourceLimit(_)), "{err:?}"); +} + +#[test] +fn security_control_chars_and_invalid_utf8() { + let e = extract( + b"h\x1b[2J\x00\n\xff\x07v\n", + DocumentHint::file_name("a.tsv"), + ); + assert_eq!(e.markdown, "| h\\[2J |\n| --- |\n| \u{FFFD}v |\n"); + assert!(e.warnings[0].starts_with("invalid_utf8:")); +} + +#[test] +fn security_arbitrary_bytes_never_panic() { + let mut seed = 0x9E37_79B9_7F4A_7C15u64; + for _ in 0..300 { + let len = (seed % 512) as usize; + let bytes: Vec = (0..len) + .map(|_| { + seed ^= seed << 13; + seed ^= seed >> 7; + seed ^= seed << 17; + const ALPHABET: &[u8] = b",;\t|\"\n\r x\\\x00\xff"; + ALPHABET[(seed % ALPHABET.len() as u64) as usize] + }) + .collect(); + for name in ["a.csv", "a.tsv"] { + let e = extract(&bytes, DocumentHint::file_name(name)); + // Every table line is a well-formed row. + assert!( + e.markdown + .lines() + .all(|l| l.starts_with('|') && l.ends_with('|')), + "{name}: {bytes:?} -> {:?}", + e.markdown + ); + } + } +} + +#[test] +fn security_cell_markup_is_escaped() { + let e = extract( + b"h\n\n[x](javascript:alert(1))\n", + DocumentHint::file_name("a.csv"), + ); + assert!(e.markdown.contains("\\`, event handlers) or pathological nesting. **Impact**: Medium -- raw HTML passed through re-introduces T1; deep nesting can crash. **Mitigations**: -- Text and Markdown are never sniffed; a content type or extension must select them +- CSV/TSV reading stops at `max_sheet_rows`; fields past `max_sheet_cols` are never stored + (a 5 MB row of commas allocates two cells) +- Text, Markdown and CSV are never sniffed; a content type or extension must select them - Decoding is total: invalid UTF-8/UTF-16 is replaced (U+FFFD) with an `invalid_utf8` warning - C0 controls (except tab/newline) and DEL are dropped before parsing - Markdown is re-parsed (pulldown-cmark) into the IR; raw HTML is never copied. HTML blocks @@ -344,6 +346,8 @@ in a `.txt`) is emitted as `Node::Text` and escaped by the renderer (S17), like - `security_input_size_limit`, `security_output_bounded_by_input` - `security_text_and_markdown_share_limits` (timeout, `max_nodes`) - `security_plain_text_markup_is_escaped` +- `csv_integration::security_*`: row/column caps, unterminated quotes, control characters, + random bytes always yield well-formed table rows ### T8: Malicious PDFs @@ -465,7 +469,7 @@ style, is not detected. | PDF decoded total / interpreted content | 256 MiB | `max_pdf_decoded_size(usize)` | Pre-scan; interpretation budget | | PDF pages | 10 000 | `max_pdf_pages(usize)` | Page loop (truncate + warn) | | PDF form XObject depth | 8 | Hardcoded `MAX_FORM_DEPTH` | `Do` operator | -| Rows / columns per sheet | 10 000 / 256 | `max_sheet_rows`, `max_sheet_cols` | While collecting cells (truncate + warn) | +| Rows / columns per sheet or CSV table | 10 000 / 256 | `max_sheet_rows`, `max_sheet_cols` | While collecting cells (truncate + warn) | ## Malicious Input Categories @@ -512,7 +516,7 @@ style, is not detected. | S15 | No configuration of `max_nesting_depth` can overflow the stack | `limits::*hard_ceiling_survives_small_stack` | | S16 | Extraction stops with `ResourceLimit` once `timeout` passes (checked at least after each non-interruptible third-party step) | `limits::*zero_timeout_rejects`, `limits::tests::*` | | S17 | Rendered markdown has no raw HTML, no non-allowlisted link/image scheme, and document text cannot create Markdown structure | `output_safety::*` (pulldown-cmark oracle + proptests) | -| S18 | Text and Markdown output never contains C0 controls (except tab/newline); Markdown raw HTML (blocks, inline tags) is never copied | `text_integration::security_*` | +| S18 | Text, Markdown and CSV/TSV output never contains C0 controls (except tab/newline); Markdown raw HTML (blocks, inline tags) is never copied | `text_integration::security_*`, `csv_integration::security_*` | | S19 | No PDF stream is decoded past `max_pdf_stream_size`, by markkit or hayro | `pdf_security::*bomb*`, `pdf_security::chained_filters_are_measured_after_every_stage` | | S20 | Hidden content is not rendered by default | `hidden_content::*` | | S21 | HTML parse time is linear in input whatever the nesting; flattened deep subtrees never emit non-content (script/style/template) text | `html_security::deep_*_is_linear`, `script_and_style_bodies_do_not_leak_past_max_depth` |