Repository navigation
fix(html): keep the source blocks — a locator per item, each block's own text - #906
Merged
Merged
Conversation
…own text An HTML conversion kept no trace of the elements its items were made from: `TreeItem` has `prov` (page geometry) and `source` (a track segment), but nothing names the original `<p>`, its `id`, or where in the document it stood, so two paragraphs with the same text were indistinguishable and a caller keyed to the source's blocks could not tie a converted item back (docling-project#903). And a block's text could only be read through a serializer, with the serializer's separators in it: `export_to_text()` keeps the blank line that follows a paragraph, so the slice between two paragraphs is the paragraph plus `"\n\n"` (docling-project#904). The item tree now carries the source's blocks (`ItemTree::blocks`): every element that produced an item, every non-inline element with text of its own (text nodes reached through inline elements only), and every empty `<p>`, heading, `<li>`, `<dt>`/`<dd>`, cell, `<pre>` or `<caption>`, in document order, as a `SourceBlock` — tag, `id`, `class`, the element path (`html/body[1]/div[3]/p[1]`: each step the tag and the 1-based position among the parent's same-tag children, so an anonymous paragraph is located too), the block's own whitespace-normalized text, and the items made from it. `TreeItem::block` points every item — an inline group's runs, a table's rich-cell groups included — back at its block. A block that produced no item says why (`BlockOmission`): hidden (`display: none`, `<script>`), empty, flattened into a table's grid (with the table item), merged into an enclosing block's item (a `<p>` inside a `<li>`). Every text node of the source belongs to exactly one block, so the blocks' texts together are the document's text. `ItemTree::source_text([i, j])` reads selected blocks with one `\n` between them and nothing else — the contract docling-project#904 asks for; the default Markdown and plain-text serializations are untouched. The walk records the element each item is created for (the flushed buffer's element, the `<li>`, the `<img>`, the `<table>`, …) and a linear post-pass over the DOM enumerates the blocks. On the 7 MB EUR-Lex consolidated text the issues were measured on (14 519 items, 15 626 blocks) the pass adds about 0.8 s to a debug-build conversion of 7.9 s. Rust API only: the JSON export is docling's schema, which has no slot for it, and stays byte-identical (the regression suite and the HTML conformance tests pass unchanged). Tests: the issues' two reductions (the located `<p class="norm">` and its 84-byte reference text; the two paragraphs joined by one newline), an inline group's runs, every omission reason, and the IAS 12 excerpt of that EUR-Lex text (`tests/fixtures/html_blocks/eurlex_ias12_excerpt.xhtml`: headings with ids, anonymous paragraphs, a flattened table, a hidden and an empty paragraph) — the coverage invariant, paragraph 79 by its locator with the reference text, and the `export_to_text()` slice the issue reports, unchanged. Closes docling-project#903 Closes docling-project#904 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ND9Uzdq6iouWJivti5dX8T
This was referenced Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An HTML conversion kept no trace of the elements its items were made from:
TreeItemhasprov(page geometry) andsource(a track segment), but nothing names the original<p>, itsid, or where in the document it stood, so two paragraphs with the same text were indistinguishable and a caller keyed to the source's blocks could not tie a converted item back (#903). And a block's text could only be read through a serializer, with the serializer's separators in it:export_to_text()keeps the blank line that follows a paragraph, so the slice between two paragraphs is the paragraph plus"\n\n"(#904).The item tree now carries the source's blocks (
ItemTree::blocks): every element that produced an item, every non-inline element with text of its own (text nodes reached through inline elements only), and every empty<p>, heading,<li>,<dt>/<dd>, cell,<pre>or<caption>, in document order, as aSourceBlock— tag,id,class, the element path (html/body[1]/div[3]/p[1]: each step the tag and the 1-based position among the parent's same-tag children, so an anonymous paragraph is located too), the block's own whitespace-normalized text, and the items made from it.TreeItem::blockpoints every item — an inline group's runs, a table's rich-cell groups included — back at its block. A block that produced no item says why (BlockOmission): hidden (display: none,<script>), empty, flattened into a table's grid (with the table item), merged into an enclosing block's item (a<p>inside a<li>). Every text node of the source belongs to exactly one block, so the blocks' texts together are the document's text.ItemTree::source_text([i, j])reads selected blocks with one\nbetween them and nothing else — the contract #904 asks for; the default Markdown and plain-text serializations are untouched.The walk records the element each item is created for (the flushed buffer's element, the
<li>, the<img>, the<table>, …) and a linear post-pass over the DOM enumerates the blocks. On the 7 MB EUR-Lex consolidated text the issues were measured on (14 519 items, 15 626 blocks) the pass adds about 0.8 s to a debug-build conversion of 7.9 s. Rust API only: the JSON export is docling's schema, which has no slot for it, and stays byte-identical (the regression suite and the HTML conformance tests pass unchanged).Tests: the issues' two reductions (the located
<p class="norm">and its 84-byte reference text; the two paragraphs joined by one newline), an inline group's runs, every omission reason, and the IAS 12 excerpt of that EUR-Lex text (tests/fixtures/html_blocks/eurlex_ias12_excerpt.xhtml: headings with ids, anonymous paragraphs, a flattened table, a hidden and an empty paragraph) — the coverage invariant, paragraph 79 by its locator with the reference text, and theexport_to_text()slice the issue reports, unchanged.Closes #903
Closes #904
Claude-Session: https://claude.ai/code/session_01ND9Uzdq6iouWJivti5dX8T