Repository navigation
feat(formats): add Jupyter notebook input - #50
Merged
Merged
Conversation
Plain text (always on) goes through the IR: paragraphs on blank lines, hard breaks on single newlines. Markdown (new `markdown` feature, in all_formats) is re-parsed with pulldown-cmark into the IR and rendered back, so raw HTML never passes through. Both decode UTF-8/UTF-16 BOMs, drop control characters and warn on lossy decoding. Neither is sniffed from bytes. Threat model T7, invariant S13.
…-3z7tjs # Conflicts: # CHANGELOG.md # README.md # crates/markkit/Cargo.toml # crates/markkit/src/extract.rs # crates/markkit/src/lib.rs # knowledge/log.md # knowledge/security/threat-model.md
…-3z7tjs # Conflicts: # knowledge/log.md
…-3z7tjs # Conflicts: # knowledge/log.md
…-3z7tjs # Conflicts: # .github/workflows/ci.yml # crates/markkit/Cargo.toml # crates/markkit/src/extract.rs # crates/markkit/src/lib.rs # knowledge/log.md # knowledge/security/threat-model.md
…-3z7tjs # Conflicts: # knowledge/log.md # knowledge/security/threat-model.md
Always-on RFC 4180 reader renders delimited text as one GFM table. Delimiter is tab for TSV and detected among , ; tab | for CSV. Reading is bounded by max_sheet_rows / max_sheet_cols (truncated_table warning); unterminated quotes emit malformed_csv. Adds Format::Tsv.
…-3z7tjs # Conflicts: # CHANGELOG.md # README.md # crates/markkit/src/extract.rs # knowledge/formats/format-support.md # knowledge/log.md # knowledge/security/threat-model.md
nbformat 4 notebooks render Markdown cells through the Markdown path, code cells as fenced blocks tagged with the kernel language, and text outputs and tracebacks (ANSI stripped) as text blocks; rich outputs are skipped. Adds Format::Notebook, Error::NotebookExtract and ExtractionStats::cells.
With no format feature, extract_document rendered unbounded, so always-on text, CSV and notebook input ignored max_output_size and the timeout (csv_integration::security_timeout_applies failed under --no-default-features). Bounded rendering is now unconditional, decoding is followed by a deadline check, and CI tests --no-default-features.
# Conflicts: # crates/markkit/src/extract.rs # crates/markkit/src/format.rs # crates/markkit/src/lib.rs # crates/markkit/src/limits.rs # crates/markkit/src/render/markdown.rs # knowledge/log.md # knowledge/operations/testing.md
This was referenced Oct 27, 2026
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
## What changed New `email` feature (in `all_formats`, so CLI and Python/Node bindings get it) converts `.eml` / `message/rfc822` files to Markdown: - Subject becomes the `#` heading and the title, followed by From / To / Cc / Date lines. - The plain-text body is preferred. When a message has only an HTML body, it goes through the HTML parser with the same stripping and hidden-content rules as web pages. - Attachments and nested messages are listed by name, type and size, never parsed. - New `Format::Eml` (`"eml"`), `Error::EmailExtract`, `ExtractionStats::attachments`. The parser shares the call's `Limits` budget (nodes, timeout, output cap). Header values are folded, so an encoded newline cannot inject a line. ## Why This is part of the cheap-formats roadmap initiative (text/Markdown #21, CSV/TSV #44 and ipynb #50 are already merged). Email is a common input for agents. ## Before / After ``` $ markkit extract --file a.eml # before Error: Unsupported format: unknown # after # Hi **From:** Ada \<ada@example.com> **To:** rene@example.com Hello \<b>there\</b> ``` I ran tests locally under all features, the default build, `--no-default-features`, `docx,spreadsheet,markdown,email` and `email` alone. Clippy passes with `-D warnings` on each, and the OKF check passes. ## Risk - Low: the feature is additive and opt-in; the default build is unchanged. - It depends on mail-parser 0.11 (`default-features = false`). 1.0 is a fresh rewrite, so it is not adopted yet. ## Checklist - [x] Tests added or updated: `email_integration` covers multipart, HTML-only, nested and detection, plus security tests for garbage input, header injection, deep multipart, attachment floods, input size and `security_limits_apply` (timeout, max_nodes, output cap) - [x] Backward compatibility considered: the change is additive. Threat T7 and invariant S18 now cover email. Knowledge, CHANGELOG and README are updated. <!-- claude-projects-attribution: {"github_login":"chaliy"} --> _Requested by **Михайло** · [project thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMJ85i1gQtgReoJ4AEgQoFq7)_ --- _Generated by [Claude Code](https://claude.ai/code/session_011vT4a1LnUCTWbKHD7BAsFM)_
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
extract_documentandparse_documentnow accept Jupyter notebooks (.ipynb,application/x-ipynb+json, nbformat 4). This applies to the CLI, Python, Node and WASM alike. It is the third "cheap formats" PR, after #21 and #44.markdownfeature they become plain-text paragraphs.streamandtext/plainbecome```textblocks.text/markdowngoes through the Markdown path.ename: evalueplus the traceback, with ANSI escapes stripped.metadata.title, otherwise the first#heading.timeoutand counts againstmax_nodes. Markdown cells share the same budget.Format::Notebook(serialized as"ipynb", the same asas_str()),Error::NotebookExtract,stats.cells.Also fixes a limits bug from #44
Reported by the IR thread. With no format feature enabled (
--no-default-features),extract_documentrendered without bounds. Always-on text, CSV and notebook input therefore ignoredmax_output_sizeand the final deadline check, andcsv_integration::security_timeout_appliesfailed.The fix:
cargo test -p markkit --no-default-features.Why
This is the roadmap initiative "cheap formats". Notebooks are a common input for agents, and their outputs often hold the result a reader needs.
Before / After
Before:
After:
Tests:
tests/notebook_integration.rshas 11 tests: mapping, ANSI stripping, metadata, detection, and security cases.max_nodeslimits.markkittests pass in each of these feature sets:--no-default-features,markdown,docx,spreadsheet,markdown, default, and all features.check-okfis clean.Risk
Checklist
🤖 Generated with Claude Code