Skip to content

feat(formats): add Jupyter notebook input - #50

Merged
chaliy merged 17 commits into
mainfrom
claude/project-thread-3z7tjs
Oct 27, 2026
Merged

chaliy merged 17 commits into
mainfrom
claude/project-thread-3z7tjs

Conversation

@chaliy

@chaliy chaliy commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Requested by Михайло · project thread

What changed

extract_document and parse_document now accept Jupyter notebooks (.ipynb, application/x-ipynb+json, nbformat 4). This applies to the CLI, Python, Node and WASM alike. It is the third "cheap formats" PR, after #21 and #44.

  • Always on, with no new dependencies (uses serde_json). Notebooks are never sniffed from bytes.
  • Markdown cells go through the Markdown parser, so raw HTML is never copied. Without the markdown feature they become plain-text paragraphs.
  • Code cells become fenced blocks tagged with the kernel language (sanitized). Raw cells become untagged blocks.
  • Outputs:
    • stream and text/plain become ```text blocks.
    • text/markdown goes through the Markdown path.
    • Errors render as ename: evalue plus the traceback, with ANSI escapes stripped.
    • Images, HTML and widget outputs are skipped.
  • Title: metadata.title, otherwise the first # heading.
  • Limits: each cell checks the call's timeout and counts against max_nodes. Markdown cells share the same budget.
  • New API: Format::Notebook (serialized as "ipynb", the same as as_str()), Error::NotebookExtract, stats.cells.

Also fixes a limits bug from #44

Reported by the IR thread. With no format feature enabled (--no-default-features), extract_document rendered without bounds. Always-on text, CSV and notebook input therefore ignored max_output_size and the final deadline check, and csv_integration::security_timeout_applies failed.

The fix:

  • Bounded rendering is now unconditional.
  • Decoding is followed by a deadline check.
  • CI now runs cargo test -p markkit --no-default-features.

Why

This is the roadmap initiative "cheap formats". Notebooks are a common input for agents, and their outputs often hold the result a reader needs.

Before / After

Before:

$ markkit extract --file analysis.ipynb
Error: Unsupported format
$ cargo test -p markkit --no-default-features --test csv_integration
test security_timeout_applies ... FAILED

After:

$ markkit extract --file analysis.ipynb
# Analysis

```python
print(1 + 1)
```

```text
2
```
$ cargo test -p markkit --no-default-features
(all pass)

Tests:

  • tests/notebook_integration.rs has 11 tests: mapping, ANSI stripping, metadata, detection, and security cases.
  • The security cases cover malformed or 1000-deep JSON, wrong value types, raw HTML, control characters, and the timeout and max_nodes limits.
  • There are unit tests and a WASM node test.
  • Clippy and markkit tests pass in each of these feature sets: --no-default-features, markdown, docx,spreadsheet,markdown, default, and all features.
  • check-okf is clean.

Risk

  • Low. The notebook support only adds behavior, and rich outputs are dropped by design.
  • The limits fix only affects builds without any format feature, which previously ignored those limits.

Checklist

  • Tests added or updated
  • Backward compatibility considered (CHANGELOG Added entry, knowledge T7/S18 extended, log entry for the limits fix)

🤖 Generated with Claude Code

chaliy added 14 commits October 26, 2026 23:35
Plain text (always on) goes through the IR: paragraphs on blank lines,
hard breaks on single newlines. Markdown (new `markdown` feature, in
all_formats) is re-parsed with pulldown-cmark into the IR and rendered
back, so raw HTML never passes through. Both decode UTF-8/UTF-16 BOMs,
drop control characters and warn on lossy decoding. Neither is sniffed
from bytes. Threat model T7, invariant S13.
…-3z7tjs

# Conflicts:
#	CHANGELOG.md
#	README.md
#	crates/markkit/Cargo.toml
#	crates/markkit/src/extract.rs
#	crates/markkit/src/lib.rs
#	knowledge/log.md
#	knowledge/security/threat-model.md
…-3z7tjs

# Conflicts:
#	.github/workflows/ci.yml
#	crates/markkit/Cargo.toml
#	crates/markkit/src/extract.rs
#	crates/markkit/src/lib.rs
#	knowledge/log.md
#	knowledge/security/threat-model.md
…-3z7tjs

# Conflicts:
#	knowledge/log.md
#	knowledge/security/threat-model.md
Always-on RFC 4180 reader renders delimited text as one GFM table.
Delimiter is tab for TSV and detected among , ; tab | for CSV. Reading
is bounded by max_sheet_rows / max_sheet_cols (truncated_table warning);
unterminated quotes emit malformed_csv. Adds Format::Tsv.
…-3z7tjs

# Conflicts:
#	CHANGELOG.md
#	README.md
#	crates/markkit/src/extract.rs
#	knowledge/formats/format-support.md
#	knowledge/log.md
#	knowledge/security/threat-model.md
nbformat 4 notebooks render Markdown cells through the Markdown path,
code cells as fenced blocks tagged with the kernel language, and text
outputs and tracebacks (ANSI stripped) as text blocks; rich outputs are
skipped. Adds Format::Notebook, Error::NotebookExtract and
ExtractionStats::cells.
@chaliy chaliy self-assigned this Oct 27, 2026
With no format feature, extract_document rendered unbounded, so always-on
text, CSV and notebook input ignored max_output_size and the timeout
(csv_integration::security_timeout_applies failed under
--no-default-features). Bounded rendering is now unconditional, decoding is
followed by a deadline check, and CI tests --no-default-features.
# Conflicts:
#	crates/markkit/src/extract.rs
#	crates/markkit/src/format.rs
#	crates/markkit/src/lib.rs
#	crates/markkit/src/limits.rs
#	crates/markkit/src/render/markdown.rs
#	knowledge/log.md
#	knowledge/operations/testing.md
@chaliy
chaliy merged commit 6cc85dd into main Oct 27, 2026
13 checks passed
chaliy added a commit that referenced this pull request Oct 27, 2026
## What changed
New `email` feature (in `all_formats`, so CLI and Python/Node bindings
get it) converts `.eml` / `message/rfc822` files to Markdown:
- Subject becomes the `#` heading and the title, followed by From / To /
Cc / Date lines.
- The plain-text body is preferred. When a message has only an HTML
body, it goes through the HTML parser with the same stripping and
hidden-content rules as web pages.
- Attachments and nested messages are listed by name, type and size,
never parsed.
- New `Format::Eml` (`"eml"`), `Error::EmailExtract`,
`ExtractionStats::attachments`.

The parser shares the call's `Limits` budget (nodes, timeout, output
cap). Header values are folded, so an encoded newline cannot inject a
line.

## Why
This is part of the cheap-formats roadmap initiative (text/Markdown #21,
CSV/TSV #44 and ipynb #50 are already merged). Email is a common input
for agents.

## Before / After
```
$ markkit extract --file a.eml
# before
Error: Unsupported format: unknown
# after
# Hi

**From:** Ada \<ada@example.com>  
**To:** rene@example.com

Hello \<b>there\</b>
```
I ran tests locally under all features, the default build,
`--no-default-features`, `docx,spreadsheet,markdown,email` and `email`
alone. Clippy passes with `-D warnings` on each, and the OKF check
passes.

## Risk
- Low: the feature is additive and opt-in; the default build is
unchanged.
- It depends on mail-parser 0.11 (`default-features = false`). 1.0 is a
fresh rewrite, so it is not adopted yet.

## Checklist
- [x] Tests added or updated: `email_integration` covers multipart,
HTML-only, nested and detection, plus security tests for garbage input,
header injection, deep multipart, attachment floods, input size and
`security_limits_apply` (timeout, max_nodes, output cap)
- [x] Backward compatibility considered: the change is additive. Threat
T7 and invariant S18 now cover email. Knowledge, CHANGELOG and README
are updated.

<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMJ85i1gQtgReoJ4AEgQoFq7)_

---
_Generated by [Claude
Code](https://claude.ai/code/session_011vT4a1LnUCTWbKHD7BAsFM)_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant