Skip to content

fix(html): keep the source blocks — a locator per item, each block's own text - #906

Merged
artiz merged 1 commit into
docling-project:masterfrom
artiz:claude/html-source-locators
Oct 28, 2026
Merged

artiz merged 1 commit into
docling-project:masterfrom
artiz:claude/html-source-locators

Conversation

@artiz

@artiz artiz commented Oct 28, 2026

Copy link
Copy Markdown
Member

An HTML conversion kept no trace of the elements its items were made from: TreeItem has prov (page geometry) and source (a track segment), but nothing names the original <p>, its id, or where in the document it stood, so two paragraphs with the same text were indistinguishable and a caller keyed to the source's blocks could not tie a converted item back (#903). And a block's text could only be read through a serializer, with the serializer's separators in it: export_to_text() keeps the blank line that follows a paragraph, so the slice between two paragraphs is the paragraph plus "\n\n" (#904).

The item tree now carries the source's blocks (ItemTree::blocks): every element that produced an item, every non-inline element with text of its own (text nodes reached through inline elements only), and every empty <p>, heading, <li>, <dt>/<dd>, cell, <pre> or <caption>, in document order, as a SourceBlock — tag, id, class, the element path (html/body[1]/div[3]/p[1]: each step the tag and the 1-based position among the parent's same-tag children, so an anonymous paragraph is located too), the block's own whitespace-normalized text, and the items made from it. TreeItem::block points every item — an inline group's runs, a table's rich-cell groups included — back at its block. A block that produced no item says why (BlockOmission): hidden (display: none, <script>), empty, flattened into a table's grid (with the table item), merged into an enclosing block's item (a <p> inside a <li>). Every text node of the source belongs to exactly one block, so the blocks' texts together are the document's text. ItemTree::source_text([i, j]) reads selected blocks with one \n between them and nothing else — the contract #904 asks for; the default Markdown and plain-text serializations are untouched.

The walk records the element each item is created for (the flushed buffer's element, the <li>, the <img>, the <table>, …) and a linear post-pass over the DOM enumerates the blocks. On the 7 MB EUR-Lex consolidated text the issues were measured on (14 519 items, 15 626 blocks) the pass adds about 0.8 s to a debug-build conversion of 7.9 s. Rust API only: the JSON export is docling's schema, which has no slot for it, and stays byte-identical (the regression suite and the HTML conformance tests pass unchanged).

Tests: the issues' two reductions (the located <p class="norm"> and its 84-byte reference text; the two paragraphs joined by one newline), an inline group's runs, every omission reason, and the IAS 12 excerpt of that EUR-Lex text (tests/fixtures/html_blocks/eurlex_ias12_excerpt.xhtml: headings with ids, anonymous paragraphs, a flattened table, a hidden and an empty paragraph) — the coverage invariant, paragraph 79 by its locator with the reference text, and the export_to_text() slice the issue reports, unchanged.

Closes #903
Closes #904

Claude-Session: https://claude.ai/code/session_01ND9Uzdq6iouWJivti5dX8T

…own text

An HTML conversion kept no trace of the elements its items were made from:
`TreeItem` has `prov` (page geometry) and `source` (a track segment), but
nothing names the original `<p>`, its `id`, or where in the document it
stood, so two paragraphs with the same text were indistinguishable and a
caller keyed to the source's blocks could not tie a converted item back
(docling-project#903). And a block's text could only be read through a serializer, with
the serializer's separators in it: `export_to_text()` keeps the blank line
that follows a paragraph, so the slice between two paragraphs is the
paragraph plus `"\n\n"` (docling-project#904).

The item tree now carries the source's blocks (`ItemTree::blocks`): every
element that produced an item, every non-inline element with text of its
own (text nodes reached through inline elements only), and every empty
`<p>`, heading, `<li>`, `<dt>`/`<dd>`, cell, `<pre>` or `<caption>`, in
document order, as a `SourceBlock` — tag, `id`, `class`, the element path
(`html/body[1]/div[3]/p[1]`: each step the tag and the 1-based position
among the parent's same-tag children, so an anonymous paragraph is located
too), the block's own whitespace-normalized text, and the items made from
it. `TreeItem::block` points every item — an inline group's runs, a table's
rich-cell groups included — back at its block. A block that produced no
item says why (`BlockOmission`): hidden (`display: none`, `<script>`),
empty, flattened into a table's grid (with the table item), merged into an
enclosing block's item (a `<p>` inside a `<li>`). Every text node of the
source belongs to exactly one block, so the blocks' texts together are the
document's text. `ItemTree::source_text([i, j])` reads selected blocks with
one `\n` between them and nothing else — the contract docling-project#904 asks for; the
default Markdown and plain-text serializations are untouched.

The walk records the element each item is created for (the flushed
buffer's element, the `<li>`, the `<img>`, the `<table>`, …) and a linear
post-pass over the DOM enumerates the blocks. On the 7 MB EUR-Lex
consolidated text the issues were measured on (14 519 items, 15 626
blocks) the pass adds about 0.8 s to a debug-build conversion of 7.9 s.
Rust API only: the JSON export is docling's schema, which has no slot for
it, and stays byte-identical (the regression suite and the HTML
conformance tests pass unchanged).

Tests: the issues' two reductions (the located `<p class="norm">` and its
84-byte reference text; the two paragraphs joined by one newline), an
inline group's runs, every omission reason, and the IAS 12 excerpt of that
EUR-Lex text (`tests/fixtures/html_blocks/eurlex_ias12_excerpt.xhtml`:
headings with ids, anonymous paragraphs, a flattened table, a hidden and an
empty paragraph) — the coverage invariant, paragraph 79 by its locator with
the reference text, and the `export_to_text()` slice the issue reports,
unchanged.

Closes docling-project#903
Closes docling-project#904

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ND9Uzdq6iouWJivti5dX8T
@artiz
artiz merged commit a2fa967 into docling-project:master Oct 28, 2026
13 checks passed
@artiz
artiz deleted the claude/html-source-locators branch October 28, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant