Repository navigation
feat(formats): add CSV and TSV input - #44
Merged
Merged
Conversation
Plain text (always on) goes through the IR: paragraphs on blank lines, hard breaks on single newlines. Markdown (new `markdown` feature, in all_formats) is re-parsed with pulldown-cmark into the IR and rendered back, so raw HTML never passes through. Both decode UTF-8/UTF-16 BOMs, drop control characters and warn on lossy decoding. Neither is sniffed from bytes. Threat model T7, invariant S13.
…-3z7tjs # Conflicts: # CHANGELOG.md # README.md # crates/markkit/Cargo.toml # crates/markkit/src/extract.rs # crates/markkit/src/lib.rs # knowledge/log.md # knowledge/security/threat-model.md
…-3z7tjs # Conflicts: # knowledge/log.md
…-3z7tjs # Conflicts: # knowledge/log.md
…-3z7tjs # Conflicts: # .github/workflows/ci.yml # crates/markkit/Cargo.toml # crates/markkit/src/extract.rs # crates/markkit/src/lib.rs # knowledge/log.md # knowledge/security/threat-model.md
…-3z7tjs # Conflicts: # knowledge/log.md # knowledge/security/threat-model.md
Always-on RFC 4180 reader renders delimited text as one GFM table. Delimiter is tab for TSV and detected among , ; tab | for CSV. Reading is bounded by max_sheet_rows / max_sheet_cols (truncated_table warning); unterminated quotes emit malformed_csv. Adds Format::Tsv.
…-3z7tjs # Conflicts: # CHANGELOG.md # README.md # crates/markkit/src/extract.rs # knowledge/formats/format-support.md # knowledge/log.md # knowledge/security/threat-model.md
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
This was referenced Oct 27, 2026
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMJ85i1gQtgReoJ4AEgQoFq7)_
## What changed
`extract_document` and `parse_document` now accept Jupyter notebooks
(`.ipynb`, `application/x-ipynb+json`, nbformat 4). This applies to the
CLI, Python, Node and WASM alike. It is the third "cheap formats" PR,
after #21 and #44.
- **Always on**, with no new dependencies (uses serde_json). Notebooks
are never sniffed from bytes.
- **Markdown cells** go through the Markdown parser, so raw HTML is
never copied. Without the `markdown` feature they become plain-text
paragraphs.
- **Code cells** become fenced blocks tagged with the kernel language
(sanitized). **Raw cells** become untagged blocks.
- **Outputs:**
- `stream` and `text/plain` become ```` ```text ```` blocks.
- `text/markdown` goes through the Markdown path.
- Errors render as `ename: evalue` plus the traceback, with ANSI escapes
stripped.
- Images, HTML and widget outputs are skipped.
- **Title:** `metadata.title`, otherwise the first `# ` heading.
- **Limits:** each cell checks the call's `timeout` and counts against
`max_nodes`. Markdown cells share the same budget.
- **New API:** `Format::Notebook` (serialized as `"ipynb"`, the same as
`as_str()`), `Error::NotebookExtract`, `stats.cells`.
### Also fixes a limits bug from #44
Reported by the IR thread. With no format feature enabled
(`--no-default-features`), `extract_document` rendered without bounds.
Always-on text, CSV and notebook input therefore ignored
`max_output_size` and the final deadline check, and
`csv_integration::security_timeout_applies` failed.
The fix:
- Bounded rendering is now unconditional.
- Decoding is followed by a deadline check.
- CI now runs `cargo test -p markkit --no-default-features`.
## Why
This is the roadmap initiative "cheap formats". Notebooks are a common
input for agents, and their outputs often hold the result a reader
needs.
## Before / After
Before:
```
$ markkit extract --file analysis.ipynb
Error: Unsupported format
$ cargo test -p markkit --no-default-features --test csv_integration
test security_timeout_applies ... FAILED
```
After:
````
$ markkit extract --file analysis.ipynb
# Analysis
```python
print(1 + 1)
```
```text
2
```
$ cargo test -p markkit --no-default-features
(all pass)
````
Tests:
- `tests/notebook_integration.rs` has 11 tests: mapping, ANSI stripping,
metadata, detection, and security cases.
- The security cases cover malformed or 1000-deep JSON, wrong value
types, raw HTML, control characters, and the timeout and `max_nodes`
limits.
- There are unit tests and a WASM node test.
- Clippy and `markkit` tests pass in each of these feature sets:
`--no-default-features`, `markdown`, `docx,spreadsheet,markdown`,
default, and all features.
- `check-okf` is clean.
## Risk
- Low. The notebook support only adds behavior, and rich outputs are
dropped by design.
- The limits fix only affects builds without any format feature, which
previously ignored those limits.
## Checklist
- [x] Tests added or updated
- [x] Backward compatibility considered (CHANGELOG Added entry,
knowledge T7/S18 extended, log entry for the limits fix)
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 tasks done
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
## What changed New `email` feature (in `all_formats`, so CLI and Python/Node bindings get it) converts `.eml` / `message/rfc822` files to Markdown: - Subject becomes the `#` heading and the title, followed by From / To / Cc / Date lines. - The plain-text body is preferred. When a message has only an HTML body, it goes through the HTML parser with the same stripping and hidden-content rules as web pages. - Attachments and nested messages are listed by name, type and size, never parsed. - New `Format::Eml` (`"eml"`), `Error::EmailExtract`, `ExtractionStats::attachments`. The parser shares the call's `Limits` budget (nodes, timeout, output cap). Header values are folded, so an encoded newline cannot inject a line. ## Why This is part of the cheap-formats roadmap initiative (text/Markdown #21, CSV/TSV #44 and ipynb #50 are already merged). Email is a common input for agents. ## Before / After ``` $ markkit extract --file a.eml # before Error: Unsupported format: unknown # after # Hi **From:** Ada \<ada@example.com> **To:** rene@example.com Hello \<b>there\</b> ``` I ran tests locally under all features, the default build, `--no-default-features`, `docx,spreadsheet,markdown,email` and `email` alone. Clippy passes with `-D warnings` on each, and the OKF check passes. ## Risk - Low: the feature is additive and opt-in; the default build is unchanged. - It depends on mail-parser 0.11 (`default-features = false`). 1.0 is a fresh rewrite, so it is not adopted yet. ## Checklist - [x] Tests added or updated: `email_integration` covers multipart, HTML-only, nested and detection, plus security tests for garbage input, header injection, deep multipart, attachment floods, input size and `security_limits_apply` (timeout, max_nodes, output cap) - [x] Backward compatibility considered: the change is additive. Threat T7 and invariant S18 now cover email. Knowledge, CHANGELOG and README are updated. <!-- claude-projects-attribution: {"github_login":"chaliy"} --> _Requested by **Михайло** · [project thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMJ85i1gQtgReoJ4AEgQoFq7)_ --- _Generated by [Claude Code](https://claude.ai/code/session_011vT4a1LnUCTWbKHD7BAsFM)_
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
extract_documentandparse_documentnow accept CSV (text/csv,application/csv,.csv) and TSV (text/tab-separated-values,.tsv,.tab). This works in every runtime: the CLI, Python, Node and WASM. It is the second "cheap formats" PR, after #21.""for a literal quote.,;tab|occurs most outside quotes in the first record, so European;files work.max_sheet_rows, and fields pastmax_sheet_colsare never stored. Either cap emits atruncated_tablewarning. An unterminated quote emitsmalformed_csv. Each record checks the call'stimeout.Format::Tsv.stats.rowsandstats.tablesare set.Why
This is part of the roadmap's "cheap formats" initiative. CSV was detected, but returned
UnsupportedFormat.Before / After
Before:
After:
Tests:
tests/csv_integration.rshas 16 tests: mapping, delimiter detection, quoting, and padding.check-okfare clean.Risk
Checklist
Generated by Claude Code