Repository navigation
feat(ir): IR v2 with footnotes, definition lists, task lists, figures, table spans - #24
Merged
Merged
Conversation
…, table spans - Node: FootnoteReference/FootnoteDefinition, DefinitionList, Figure; ListItem.checked; Table cells as TableCell (colspan/rowspan) + alignments - Document.ir_version (IR_VERSION = 2); IR types #[non_exhaustive]; Node::is_block/is_inline - HTML: GitHub/Pandoc footnotes, <dl>, checkbox items, captioned figures, table spans/alignment, row-header <th> kept - Renderer: GFM + footnote/deflist extensions; span grid layout bounded by MAX_TABLE_GRID_CELLS (T2e, S13)
This was referenced Oct 26, 2026
chaliy
added a commit
that referenced
this pull request
Oct 27, 2026
<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_01MUtksmx61ypcr6wBFEWsbM?thread=cmsg_01MUtksmx61ypcr6wBFEWsbMTNX4RpiXUKhUJtNtdtM6pk)_
## What changed
Second step of IR v2: provenance for citations, and IR access for every
format.
- **`ir::Source`** on every block (except `ThematicBreak`), opt-in with
`MarkkitBuilder::track_sources(true)`:
- HTML: absolute XPath of the element (`/html[1]/body[1]/div[1]/p[2]`)
- DOCX: 0-based body block index (`w:p`/`w:tbl` order; lists take the
first item's index)
- Spreadsheets: sheet name on heading + table, A1 range of the table
(`B2:C3`)
- `page` reserved for PDF
- **`Markkit::parse_document[_sync]`** returns `ParsedDocument` (IR +
title, format, warnings, stats) for HTML, DOCX and spreadsheets;
`ParsedDocument::render()` gives the `Extraction`. `extract_document` is
now `parse_document` + `render`, so the output is unchanged.
- Off by default: no extra work, no `source` in JSON, Markdown identical
with or without tracking.
## Why
Roadmap item "Richer IR with provenance": RAG users need to cite where a
chunk came from. Until now DOCX and spreadsheet IR could not be reached
at all; only Markdown came out.
## Before / After
Before: `parse_html` only; no positions.
After:
```rust
let m = Markkit::builder().track_sources(true).build();
let p = m.parse_document_sync(&xlsx, DocumentHint::default())?;
p.document.nodes[1].source() // Some(Source { sheet: Some("Data"), range: Some("B2:C3"), .. })
```
```json
{"Paragraph":{"children":[{"Text":"b"}],"source":{"path":"/html[1]/body[1]/div[1]/p[2]"}}}
```
## Risk
- Low–Medium. IR API break (pre-1.0, same release as #24): block
variants gain a `source` field, so patterns need `..`; `Node::paragraph`
/ `Node::heading` constructors added.
- Perf/DoS: XPath sibling indexes are cached per parent, O(depth) per
block. `html_security::source_tracking_many_siblings_is_linear` covers
200k siblings.
- New dependency edge `ego-tree` (already in the tree via scraper),
gated by `html`.
## Checklist
- [x] Tests added or updated (HTML XPath + nested list/blockquote,
off-by-default/JSON, markdown unchanged, DOCX indexes, XLSX sheet/range,
perf)
- [x] Backward compatibility considered (`extract_*` output unchanged;
CHANGELOG notes IR break)
- [x] Knowledge updated (`foundations/ir.md`,
`foundations/architecture.md`, log), docs/getting-started
---
_Generated by [Claude
Code](https://claude.ai/code/session_014kBa8kqnewZALvymUgcVBj)_
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
First step of the IR v2 + extensibility roadmap.
FootnoteReference/FootnoteDefinition), definition lists, task-list items (ListItem.checked), figures with captions, table column alignment andcolspan/rowspan(TableCell).Document.ir_version(IR_VERSION = 2) is serialized;Node,Document,ListItemand the new structs are#[non_exhaustive];Node::is_block()/is_inline()replace per-module lists. Default/unset fields are omitted from JSON.<dl>, leading checkbox in<li>,<figure>+<figcaption>, tablecolspan/rowspan/align/text-align. Row-header<th>cells in body rows are no longer dropped.[^1],Term/: def,- [x]). Spans are laid out on a grid (content top-left, covered slots empty). DOCXgridSpannow goes through the same path.Breaking (IR API, pre-1.0): external
matchonNodeneeds a_arm;Node::Tablecells areTableCell. Markdown output for existing inputs is unchanged except where it was wrong (see below).Why
Roadmap: richer IR for LLM-ready output and a stable, versioned JSON contract before adding transforms, element rules and a parser registry.
Before / After
Input: checkbox list,
<dl>, GitHub footnote, table withcolspan/rowspan/align.Before:
After:
Risk
#[non_exhaustive],TableCell). Bindings don't expose IR, so Python/Node APIs are unaffected.[A-Za-z0-9_.-]so they can't close[^…](invariant S13). Tests:html_security::hostile_colspan_rowspan_bounded,non_numeric_spans_ignored,deep_definition_lists_bounded,html_integration::footnote_label_cannot_break_out.Checklist
ir_versionmissing in JSON reads as 1; CHANGELOG notes the IR break)foundations/ir.md,formats/html-extraction.md, threat model T2e/S13, log)Generated by Claude Code