Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
32f53d3
fix(spreadsheet): single-row sheets, workbook titles, XLSB detection
chaliy Oct 26, 2026
c913ef0
fix(spreadsheet): single-row sheets, workbook titles, XLSB detection
chaliy Oct 26, 2026
b7fad42
chore: merge stale branch head
chaliy Oct 26, 2026
7322499
fix(spreadsheet): satisfy clippy chunks_exact_to_as_chunks
chaliy Oct 27, 2026
119c757
chore: merge origin/main
chaliy Oct 27, 2026
465cb08
Merge remote-tracking branch 'origin/main' into claude/project-thread…
chaliy Oct 27, 2026
f4b06d9
Merge remote-tracking branch 'origin/main' into claude/project-thread…
chaliy Oct 27, 2026
86b7104
chore(knowledge): tidy log after merge
chaliy Oct 27, 2026
28f4d17
Merge remote-tracking branch 'origin/main' into claude/project-thread…
chaliy Oct 27, 2026
3a62597
Merge remote-tracking branch 'origin/main' into claude/project-thread…
chaliy Oct 27, 2026
e0365e9
Merge remote-tracking branch 'origin/main' into claude/project-thread…
chaliy Oct 27, 2026
a6a7d7d
feat(spreadsheet): read XLSB workbooks with a bounds-checked BIFF12 r…
chaliy Oct 27, 2026
f36f38e
chore: supersede merged PR history
chaliy Oct 27, 2026
7130864
chore: merge origin/main; XLSB honours the parse time budget
chaliy Oct 27, 2026
50b5aea
chore: merge origin/main
chaliy Oct 27, 2026
7b9703c
feat(pptx): convert PowerPoint presentations slide by slide
chaliy Oct 27, 2026
c422071
chore: supersede merged PR history
chaliy Oct 27, 2026
5e2469b
test(js): use legacy .ppt as the unsupported-format example
chaliy Oct 27, 2026
c058ec4
chore: merge origin/main
chaliy Oct 27, 2026
41e90ee
chore(knowledge): drop trailing whitespace
chaliy Oct 27, 2026
f4daf3b
feat(api): structured warnings; Markkit is Clone + Debug
chaliy Oct 27, 2026
6415396
feat(limits): structured ResourceLimit errors with LimitKind
chaliy Oct 27, 2026
e67a42f
chore: supersede merged PR history
chaliy Oct 27, 2026
0a2bb5a
Merge remote-tracking branch 'origin/main' into claude/project-thread…
chaliy Oct 27, 2026
c2a3a86
Merge remote-tracking branch 'origin/main' into claude/project-thread…
chaliy Oct 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

### Changed

- **Breaking:** `Error::ResourceLimit` carries a `LimitExceeded { kind: LimitKind, detail }` instead of
a `String`. `LimitKind` names the `Limits` field (`MaxInputSize`, `Timeout`, `MaxArchiveEntrySize`,
...); `Error::limit_kind()` returns it. `ResourceLimit(_)` patterns and the error message are unchanged.
- DOCX uses far less memory: body blocks are converted while `document.xml` is parsed and freed
right after, and the XML tree is compact (interned names and values, exact-size lists). A 1.5 MB
`document.xml` peaks at 3.9 MiB instead of 18.5 MiB, with half the allocations. Markdown
Expand All @@ -160,6 +163,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
no HTTP/TLS stack. Library users calling `extract_url` add `features = ["http_client"]`. The CLI and
the Python/Node packages still enable it, so their `extract_url` is unchanged.
- **Breaking:** `Error` is `#[non_exhaustive]` (new `Error::Transform`); matches need a `_` arm.
- **Breaking:** `Extraction::warnings` and `ParsedDocument::warnings` are `Vec<Warning>`
(`code: WarningCode` + `detail: String`) instead of `"code: detail"` strings, so callers can match
on the code. `Warning`'s `Display` keeps the old string form; JSON serializes
`{"code": "truncated_sheet", "detail": "..."}`. The agent tool output is unchanged.
Node and WASM `extractDocument` return `warnings` as `{ code, detail }` objects too.
- `Markkit` implements `Clone` and `Debug`.
- **Breaking (IR API):** `Node`, `Document`, `ListItem` and the new IR structs are `#[non_exhaustive]`;
matches need a `_` arm and structs are built with constructors. `Node::Table` cells are now
`TableCell` (`children`, `colspan`, `rowspan`) with an `alignments` field. Block variants gain a
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ println!("{:?} {:?} {:?}", doc.format, doc.title, doc.warnings);
println!("{}", doc.markdown);
```

`Extraction` carries `markdown`, `title`, `format`, `warnings` (e.g. `truncated_sheet: …`),
`Extraction` carries `markdown`, `title`, `format`, `warnings` (structured: `code` such as
`WarningCode::TruncatedSheet` plus a human-readable `detail`),
`stats` (sheets/rows/paragraphs/tables) and, for HTML, `metadata` (description, author,
published date, language, canonical URL, site name, lead image from `<meta>`, Open Graph
and JSON-LD). `extract_bytes` remains as a Markdown-only shortcut.
Expand Down Expand Up @@ -218,7 +219,8 @@ markkit = { version = "0.1", features = ["http_client"] }
### Resource limits

Every resource an input can drive is bounded by `markkit::Limits`. Defaults are
generous; violations return `Error::ResourceLimit` naming the limit.
generous; violations return `Error::ResourceLimit`, whose `LimitKind` (`err.limit_kind()`)
names the field that was exceeded.

| Limit | Default | Bounds |
|-------|---------|--------|
Expand Down
2 changes: 1 addition & 1 deletion crates/markkit-js/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ await extractHtml(html, { includeSelectors: ['article'], excludeSelectors: ['.ad
// URLs: non-public addresses are blocked by default.
await extractUrl('https://docs.example.com/page', { allowedHosts: ['docs.example.com'], maxRedirects: 2 });

// Documents with metadata (HTML, DOCX, XLSX/XLS/ODS).
// Documents with metadata (HTML, DOCX, PPTX, XLSX/XLSB/XLS/ODS). `warnings` are `{ code, detail }`.
const doc = extractDocumentSync(fs.readFileSync('report.docx'), { fileName: 'report.docx' }, { maxSheetRows: 500 });
console.log(doc.title, doc.format, doc.warnings, doc.markdown);
```
Expand Down
22 changes: 19 additions & 3 deletions crates/markkit-js/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,14 +88,23 @@ pub struct Metadata {
pub image: Option<String>,
}

/// Structured extraction warning.
#[napi(object)]
pub struct Warning {
/// Stable machine code (`truncated_sheet`, `hidden_slide_skipped`, ...).
pub code: String,
/// Human-readable detail.
pub detail: String,
}

/// Markdown plus metadata.
#[napi(object)]
pub struct Extraction {
pub markdown: String,
pub title: Option<String>,
/// Detected format (`html`, `docx`, `xlsx`, `xls`, `ods`, `text`, `markdown`, `csv`, `tsv`, `ipynb`).
/// Detected format (`html`, `docx`, `pptx`, `xlsx`, `xlsb`, `xls`, `ods`, `text`, `markdown`, `csv`, `tsv`, `ipynb`).
pub format: String,
pub warnings: Vec<String>,
pub warnings: Vec<Warning>,
pub stats: ExtractionStats,
pub metadata: Metadata,
}
Expand Down Expand Up @@ -222,7 +231,14 @@ fn to_js(e: markkit::Extraction) -> Extraction {
markdown: e.markdown,
title: e.title,
format: e.format.as_str().to_string(),
warnings: e.warnings,
warnings: e
.warnings
.into_iter()
.map(|w| Warning {
code: w.code.as_str().to_string(),
detail: w.detail,
})
.collect(),
stats: ExtractionStats {
pages: count(e.stats.pages),
sheets: count(e.stats.sheets),
Expand Down
8 changes: 7 additions & 1 deletion crates/markkit-wasm/js/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -61,11 +61,17 @@ export interface Metadata {
image: string | null;
}

export interface Warning {
/** Stable machine code (`truncated_sheet`, `hidden_slide_skipped`, ...). */
code: string;
detail: string;
}

export interface Extraction {
markdown: string;
title: string | null;
format: string;
warnings: string[];
warnings: Warning[];
stats: ExtractionStats;
metadata: Metadata;
}
Expand Down
25 changes: 15 additions & 10 deletions crates/markkit/src/archive.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,13 @@ use zip::ZipArchive;
use zip::result::ZipError;

use crate::config::MarkkitBuilder;
use crate::limits::{LimitExceeded, LimitKind};

/// Archive failure, split so callers can map limit violations to
/// [`crate::Error::ResourceLimit`] and everything else to a format error.
#[derive(Debug)]
pub(crate) enum ArchiveError {
Limit(String),
Limit(LimitExceeded),
Invalid(String),
}

Expand All @@ -42,10 +43,13 @@ impl<'a> ZipGuard<'a> {
pub fn open(bytes: &'a [u8], config: &MarkkitBuilder) -> Result<Self, ArchiveError> {
let archive = ZipArchive::new(Cursor::new(bytes))?;
if archive.len() > config.limits.max_archive_entries {
return Err(ArchiveError::Limit(format!(
"archive has {} entries, limit is {}",
archive.len(),
config.limits.max_archive_entries
return Err(ArchiveError::Limit(LimitExceeded::new(
LimitKind::MaxArchiveEntries,
format!(
"archive has {} entries, limit is {}",
archive.len(),
config.limits.max_archive_entries
),
)));
}
let names = archive
Expand Down Expand Up @@ -115,9 +119,9 @@ impl<'a> ZipGuard<'a> {
fn add_total(&mut self, n: u64) -> Result<(), ArchiveError> {
self.total = self.total.saturating_add(n);
if self.total > self.max_total {
return Err(ArchiveError::Limit(format!(
"archive decompresses to more than {} bytes",
self.max_total
return Err(ArchiveError::Limit(LimitExceeded::new(
LimitKind::MaxArchiveTotalSize,
format!("archive decompresses to more than {} bytes", self.max_total),
)));
}
Ok(())
Expand All @@ -136,8 +140,9 @@ fn copy_capped<R: Read, W: std::io::Write>(
let n = std::io::copy(&mut reader.take(cap.saturating_add(1)), out)
.map_err(|e| ArchiveError::Invalid(format!("cannot read entry '{name}': {e}")))?;
if n > cap {
return Err(ArchiveError::Limit(format!(
"entry '{name}' decompresses to more than {cap} bytes"
return Err(ArchiveError::Limit(LimitExceeded::new(
LimitKind::MaxArchiveEntrySize,
format!("entry '{name}' decompresses to more than {cap} bytes"),
)));
}
Ok(n)
Expand Down
17 changes: 11 additions & 6 deletions crates/markkit/src/csv.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ use crate::limits::Budget;
/// Parsed CSV/TSV document.
pub(crate) struct CsvOutput {
pub doc: Document,
pub warnings: Vec<String>,
pub warnings: Vec<crate::extract::Warning>,
/// Data rows (header excluded).
pub rows: usize,
/// 1 when a table was produced, else 0.
Expand Down Expand Up @@ -59,16 +59,21 @@ pub(crate) fn parse(
records.push(rec);
}
if reader.unterminated {
warnings.push("malformed_csv: unterminated quoted field".into());
warnings.push(crate::extract::Warning::new(
crate::extract::WarningCode::MalformedCsv,
"unterminated quoted field",
));
}
if truncated_rows {
warnings.push(format!(
"truncated_table: table truncated to {max_rows} rows"
warnings.push(crate::extract::Warning::new(
crate::extract::WarningCode::TruncatedTable,
format!("table truncated to {max_rows} rows"),
));
}
if reader.truncated_cols {
warnings.push(format!(
"truncated_table: table truncated to {max_cols} columns"
warnings.push(crate::extract::Warning::new(
crate::extract::WarningCode::TruncatedTable,
format!("table truncated to {max_cols} columns"),
));
}

Expand Down
13 changes: 8 additions & 5 deletions crates/markkit/src/docx.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ use crate::archive::{ArchiveError, ZipGuard};
use crate::config::MarkkitBuilder;
use crate::error::Error;
use crate::ir::{Document, Node, Source, TableCell};
use crate::limits::Budget;
use crate::limits::{Budget, LimitExceeded};
use crate::ooxml::{
Fmt, ListEntry, Relationship, Seg, build_inline, build_list, is_blank, parse_rels, plain_text,
rel_target_by_type, resolve_target, split_dir, table_cell,
Expand All @@ -36,7 +36,7 @@ use crate::xml::{self, Element, XmlError};
/// Result of parsing a DOCX package.
pub(crate) struct DocxOutput {
pub doc: Document,
pub warnings: Vec<String>,
pub warnings: Vec<crate::extract::Warning>,
pub title: Option<String>,
pub paragraphs: usize,
pub tables: usize,
Expand All @@ -53,7 +53,9 @@ fn archive_err(e: ArchiveError) -> Error {

fn xml_err(part: &str, e: XmlError) -> Error {
match e {
XmlError::Limit(m) => Error::ResourceLimit(format!("{part}: {m}")),
XmlError::Limit(m) => {
Error::ResourceLimit(LimitExceeded::new(m.kind, format!("{part}: {}", m.detail)))
}
XmlError::Invalid(m) => Error::DocxExtract(format!("{part}: {m}")),
}
}
Expand Down Expand Up @@ -693,8 +695,9 @@ pub(crate) fn parse(
doc: Document::new(body.blocks),
warnings: match ctx.hidden_runs.get() {
0 => Vec::new(),
n => vec![format!(
"hidden_content_skipped: {n} hidden text run(s) skipped"
n => vec![crate::extract::Warning::new(
crate::extract::WarningCode::HiddenContentSkipped,
format!("{n} hidden text run(s) skipped"),
)],
},
})
Expand Down
14 changes: 13 additions & 1 deletion crates/markkit/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,10 @@ pub enum Error {
#[error("Unsupported format: {0}")]
UnsupportedFormat(String),

/// A [`Limits`](crate::Limits) value was exceeded; match on
/// [`LimitExceeded::kind`](crate::LimitExceeded) (or use [`Error::limit_kind`]).
#[error("Resource limit exceeded: {0}")]
ResourceLimit(String),
ResourceLimit(crate::limits::LimitExceeded),

/// A custom [`crate::registry::FormatParser`] failed or panicked.
#[error("Parser error: {0}")]
Expand All @@ -48,3 +50,13 @@ pub enum Error {
#[error("I/O error: {0}")]
Io(#[from] std::io::Error),
}

impl Error {
/// The exceeded limit, for [`Error::ResourceLimit`].
pub fn limit_kind(&self) -> Option<crate::limits::LimitKind> {
match self {
Error::ResourceLimit(e) => Some(e.kind),
_ => None,
}
}
}
Loading
Loading