Skip to content

feat: make inline serializers concatenate runs faithfully - #693

Open
cau-git wants to merge 19 commits into
mainfrom
cau/serializer-whitespace-fixes
Open

cau-git wants to merge 19 commits into
mainfrom
cau/serializer-whitespace-fixes

Conversation

@cau-git

@cau-git cau-git commented Jul 27, 2026 •

Copy link
Copy Markdown
Member

Update: merged main, review fixes

main (b501ec80, v2.104.0) is merged into this branch. The conflicts were resolved to the versions reviewed on a rebased copy, and the newer commits were replayed on top, so no force push was needed.

New since the last push:

  • Text before an inline group. A text item's own text and its inline-group child are joined with " ", as on main. Children of a text item are block structure. With "", headings merged with their first formatted paragraph (ResultsWe use bold words.). A producer that wants text flush against runs puts that text in the group as a run.
  • Markdown lists. A list item's text no longer merges with its inline group (- Intromore → - Intro more).
  • Plain-text reader. It now joins inline runs faithfully.
  • Migration fixes from an independent review:
    • A separator inserted between code or formula runs is kept when exporting or iterating a single page (page_no=…).
    • Empty hosts and directly nested inline groups now count as runs. Base core's own DocLang reader writes this structure, so real saved JSON is affected.
  • Goldens re-baselined (120 files of net test-data diff against main).

Summary

InlineGroup runs now carry their own significant whitespace, and the serializers concatenate them faithfully instead of joining with a hard " ".

The join was the defect. It invented spaces the model never asked for, and while the separator lived in the serializer, a producer fix could never remove them:

source before after
2<sup>nd</sup> 2 nd 2nd
O<sub>2</sub>. O 2 . O2.
**Advanced Topics** in *Machine Learning* **Advanced Topics** in *Machine Learning* **Advanced Topics** in *Machine Learning*
This is *italic text*. This is *italic text* . This is *italic text*.

The ODF backend already emitted runs that follow the contract, so its committed Markdown goldens show the reverse artifact: X 2 + Y 2 = Z. That is the shortest proof that the serializer, not the producer, was wrong.

What changed

Serializers. Markdown, HTML, LaTeX, DocLang and the plain-text reader join inline parts with "". DocLang keeps its separator only between the element head and the body; <content> remains its whitespace channel.

Whitespace hoisting. In DOCX and HTML the boundary space often sits inside a formatted run. Emitting it as-is produces **bold ** tail, which CommonMark renders as literal asterisks. Markdown, HTML and LaTeX therefore split each run into leading whitespace, core text and trailing whitespace, apply the complete decoration stack to the core, and put the whitespace back outside. A single formatter can't do this alone, because the hyperlink would capture the space again ([**bold** ](url)). DocLang opts out on purpose and keeps the whitespace inside the markup, in <content>.

DocLang deserializer.

  • _get_text stripped every fragment before joining, which collapsed Advanced <bold>Topics</bold> into AdvancedTopics. It now trims only the outer edges of the block, and never trims whitespace delivered by <content>.
  • Bare text nodes use the standard XML rule: a whitespace run that contains a newline is pretty-print indentation, and one without a newline is content. Serialized DocLang uses <content> for runs that carry whitespace, so this rule only matters for hand-written and VLM-emitted input.
  • Element-head children (<caption>, <description>, <summary>, <custom>) no longer leak into the visible body text. They were also modelled a second time in item.meta.

Compatibility

Saved .json files contain runs without spaces, and coordinating releases can't fix a file on disk. The schema minor is bumped to 1.11.0. A model_validator(mode="before") re-inserts the legacy separator for documents stamped below that version. A 1.11→1.10 downgrade projector is registered with the #703 framework. It strips inline runs to the shape old producers emitted, so a 1.10 client renders a projected document the way it renders an old one (H 2 O, but no double spaces and no **bold **).

The goal is narrow: keep the old serializer's visible output, including its extra-space bugs. An old document doesn't hold enough information to recover the real whitespace: ["H", "2", "O"] and ["Normal", "italic"] have the same shape but need different boundaries.

["Normal", "italic"] -> ["Normal ", "italic"] -> "Normal italic"
["H", "2", "O"]      -> ["H ", "2 ", "O"]     -> "H 2 O"        # legacy bug retained
["left ", "right"]   -> ["left  ", "right"]   -> "left  right"  # legacy double space retained
  • It must run with mode="before", because check_version_is_compatible rewrites version to CURRENT_VERSION and so destroys the evidence.
  • Input without a version is left alone rather than guessed at.
  • It works on a copy, so the caller's dict is never changed and validating the same raw value twice can't stack separators.
  • A separator never lands inside a delimiter. Code and formula runs are wrapped in ` and $, so the space goes to the plain neighbour. When both neighbours are delimited, the space becomes a plain run of its own, because `ab`` and$a$$b$` are both corrupt. That run takes the left run's provenance, so page filters keep it exactly when they keep both runs.
  • Empty hosts and nested inline groups count as runs.
  • One-way and idempotent; it warns once per process.

Limitations

  • Live conversion with an old docling backend. The backend builds the document in memory at the current version, so there is no old stamp to key the migration on. This is an accepted, documented risk, reduced by the paired docling change.
  • DocLang / .dclx. DocLang written by docling-core before schema 1.11.0 is loaded as current, because DocLang carries no schema version and old and new output have the same structure for tight runs. Inline runs from such archives join without the old separator (Normal**bold**tail). Re-convert the source, or use JSON (which is migrated on load) for long-term storage. In the other direction, the old core reads new archives with the old " " join (H 2 O, and double spaces at runs that carry whitespace).
  • DocTags. load_from_doctags can't restore run whitespace, because the DocTags serializer already stripped it. Legacy DocTags inline groups export with merged words. This is documented on the method.

Behaviour changes to expect

  • Chunk text changes for HierarchicalChunker. ChunkingDocSerializer defines no inline serializer, so it inherits the Markdown join. This changes embeddings, so anyone with a persisted vector index will see silent retrieval drift on upgrade. It needs its own line in the release notes. HybridChunker serializes each item separately and joins items with delim; that behaviour already exists and is tracked separately.
  • Markdown, plain-text, HTML, LaTeX and DocLang exports are re-baselined.
  • DocTags output is unchanged and out of scope.

The failure mode inverts

Worth flagging for reviewers: today, a producer that leaves out boundary whitespace produces extra spaces. That is ugly but harmless, and it has gone unnoticed for as long as it has existed. After this change the same omission produces merged words. The cost of a mistake goes from cosmetic to severe, and it falls on third-party backends and on code that builds DoclingDocuments by hand. The add_inline_group docstring now states the contract so the next backend author sees it.

Testing

1160 passed, 10 skipped, 1 xfailed, plus the compat-coverage gate, ty and ruff. Regenerating goldens with DOCLING_GEN_TEST_DATA=1 changes nothing.

The new tests/test_inline_whitespace_contract.py holds the contract as a table: one set of inline-run sequences checked against md, itxt, html, latex and doclang. The table includes whitespace inside a formatted run, the same inside a formatted run with a hyperlink, and a whitespace-only run. The file also covers:

  • DocLang round-trip identity over the same table, both pretty-printed and compact;
  • load-time migration of old-schema documents, including every pairing of plain, code and formula runs, page filters, empty hosts and nested groups;
  • that the migration doesn't change caller-owned input and gives the same result on repeated validation;
  • host text versus its inline group, including Markdown list items;
  • chunk text, pinned so the embedding drift can't come back unnoticed.

Pre-contract fixtures keep their old version stamp on purpose. Their output is unchanged, so they act as the regression test for the migration.

Release notes

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

❌ DCO Check Failed

Hi @cau-git, your pull request has failed the Developer Certificate of Origin (DCO) check.

This repository supports remediation commits, so you can fix this without rewriting history — but you must follow the required message format.


🛠 Quick Fix: Add a remediation commit

Run this command:

git commit --allow-empty -s -m "DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com>

I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 7600953bc08ca8b33ea758c8a8192897515c4d86
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: f0e26b6722e2c1b8c543a8b68f04641772f3ef5a"
git push

🔧 Advanced: Sign off each commit directly

For the latest commit:

git commit --amend --signoff
git push --force-with-lease

For multiple commits:

git rebase --signoff origin/main
git push --force-with-lease

More info: DCO check report

@mergify

mergify Bot commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@cau-git cau-git changed the title Cau/serializer whitespace fixes feat: make inline serializers concatenate runs faithfully Jul 27, 2026
@codecov

codecov Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.17834% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
docling_core/types/doc/document.py 93.42% 5 Missing ⚠️
docling_core/transforms/deserializer/doclang.py 97.43% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

`_get_children_simple_text_block` guarded against a second *text* fragment
but the Element branch just overwrote `result`, so
`<text>2<superscript>nd</superscript></text>` returned "nd" and the leading
`2` was silently discarded. Guard both branches so the element is routed to
`_parse_inline_group`, which already handles mixed children.
… faithfully

The Markdown, HTML, LaTeX and DocLang inline serializers joined runs with a
hard `" "`, inventing spaces the model never asked for: `2nd` -> `2 nd`,
`O2.` -> `O 2 .`, `italic text.` -> `italic text .`. Those artifacts can never
be removed while the separator lives in the serializer.

Serializers:
- join inline parts with `""`; DocLang keeps its delimiter between the element
  head and the body only, where `<content>` is the whitespace channel
- hoist edge whitespace around the *complete* decoration stack (formatting and
  hyperlink together) for Markdown, HTML and LaTeX. Decorating it verbatim
  emits `**bold ** tail`, which CommonMark renders as literal asterisks;
  hoisting inside one formatter would let the hyperlink recapture it. DocLang
  opts out: it keeps whitespace inside the markup, in `<content>`.

DocLang deserializer:
- `_get_text` stripped every fragment and joined with `""`, collapsing
  `Advanced <bold>Topics</bold>` into `AdvancedTopics`. It now trims the outer
  block boundary only, and never trims whitespace delivered by `<content>`.
- bare text nodes use the standard XML heuristic: a whitespace run containing a
  newline is pretty-print indentation, one without it is content. This is what
  hand-written and VLM-emitted DocLang needs, since it will not use `<content>`.
- element-head children (`<caption>`, `<description>`, `<summary>`, `<custom>`)
  no longer bleed into body text, where they were also double-modelled in
  `item.meta`.

Data at rest: schema minor bumped to 1.11.0 and a `mode="before"` validator
re-injects the legacy separator for documents stamped below it, preserving their
previous *visible* output including its extra-space bugs (`H 2 O` stays
`H 2 O`). It must run before `check_version_is_compatible`, which rewrites
`version` and destroys the evidence. Versionless input is left alone. The
validator works on a copy, so the caller's dict is untouched and re-validating
the same raw value cannot compound separators.

A separator can never land inside a delimiter. Code and formula runs are wrapped
in `` ` ``/`$` by the serializers, so the space goes on the plain neighbour; when
both neighbours are delimited it becomes a plain run of its own, because
`` `a``b` `` and `$a$$b$` are both corrupt.

Chunk text changes too, since ChunkingDocSerializer inherits the Markdown join
-- anyone with a persisted vector index gets silent retrieval drift, so that
needs its own line in the release notes.

DocTags code is untouched. Its golden gains one empty `<text></text>` for the
legacy fixture because that document now carries an extra whitespace run, which
DocTags renders empty by its own pre-existing strip-per-run behaviour.

Ships as a minor, not a major: `feat!:` would make python-semantic-release
(7.34.6, both repos) bump to 3.0, and a 3.0 is not affordable on either package.
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 64404b3
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 66040cb

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…ct bump

Integrates the 1.11.0 schema bump with the downgrade-projector framework
(#703) landed upstream. 1.11 and 1.10 are structurally identical; the only
difference is that 1.11 inline runs own their boundary whitespace, so the
projector just re-stamps the version (a 1.10 client re-applies its own join,
yielding the documented harmless double space).

- add @register_projector(from_minor=11, to_minor=10) + TestProjector_1_11_to_1_10
- add the auto-generated 1.11 JSON Schema snapshot
- re-format the 1.11 SCHEMA_VERSION_HISTORY entry (ruff)
- update the stale dclq CLI expectation (old join produced 'public  class';
  faithful join now yields 'public class')

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
cau-git force-pushed the cau/serializer-whitespace-fixes branch from 79ebc53 to e4e745a Compare August 18, 2026 13:25
typer >=0.25 vendors its own copy of click (typer._click) and raises those
exceptions instead of the external click package's. The dclq CLI referenced
the external click, so on typer >=0.25 bad input escaped the exit-code
handlers and the context-flag callback (-C/-A/-B/--context-scope) silently
no-op'd because the isinstance(param, click.Option) check no longer matched.

Import the exception and Parameter types from typer._click, duck-type the
option check, raise the [dclq] typer floor to >=0.25, and drop the now-unused
external click dependency.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@nanhanq1

nanhanq1 commented Oct 6, 2026

Copy link
Copy Markdown

Rebased this onto current main (it was 60 commits behind) and opened #843 against main: same five commits, conflicts resolved, ruff check / ruff format --check clean and the affected tests showing no regression against a main baseline. #843 supersedes this one — feel free to close it whenever you like, nothing needed from me.

PeterStaar-IBM
PeterStaar-IBM previously approved these changes Oct 22, 2026

@PeterStaar-IBM PeterStaar-IBM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

cau-git and others added 7 commits October 26, 2026 09:00
Conflicts resolved to the versions reviewed on the rebased branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…roups

<content> is the explicit whitespace-preserving text wrapper, so inside an
inline group it is a run, not a nested item: it must inherit the formatting
accumulated while descending through <bold>/<italic>/... . Dispatching it
through _dispatch_element dropped that formatting, which surfaced after the
rebase as <bold><content>x </content></bold> coming back unformatted and
broke the doclang round-trip for whitespace_inside_bold / _link.

Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>
The whitespace-contract bump was first written against v2.86.0; main is now
at v2.104.0, so record v2.105.0 as the placeholder release that ships schema
1.11.0. Adjust to the actual version at release time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Regenerated with DOCLING_GEN_TEST_DATA=1. The only change is the schema
version stamp 1.10.0 -> 1.11.0 in goldens added or moved on main
(including the six test/ -> tests/ renames and misplaced_list_items.out.yaml).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
_collect_subtree_text (TextItem._get_plain_text, FloatingItem.caption_text,
nested rich table cells) still joined the runs of an InlineGroup with " ",
so mixed captions read "Table 1 : results". It now concatenates the runs of
an inline group; every other join, including a host text item's own text
and its children, keeps its space. Children of a text item are block
structure (producers put a section's paragraphs under its heading), so the
host text stays apart from its inline group, as on main; add_inline_group
documents this.

Tests that asserted the invented run spacing now expect the contract
output, fixtures that built runs without boundary whitespace now carry it,
and a new test pins the host-text boundary for headings and captions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
DocTags stores inline runs trimmed, so the boundary whitespace the inline
whitespace contract expects is not in the source. Document the limitation
instead of guessing separators back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
The Markdown list serializer glued an inline group onto the preceding
list item with no separator, so a list item "Intro" whose child group
starts with "more" exported "- Intromore". Captions, headings and the
plain-text reader keep a space between host text and its inline group;
the list serializer now does the same when the group is the child of a
text item with its own text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
cau-git and others added 4 commits October 26, 2026 09:01
… page filters

A legacy document with two adjacent code or formula runs gets a standalone
" " run between them on load. It had no provenance, so filter(page_nrs=...)
and export_to_markdown(page_no=...) dropped it and rendered `$a$$b$`.

The separator now takes the left run's location when both runs are on the
same page, so it survives a page filter exactly when both runs do. Runs on
different pages keep the unlocated separator, which matches the old join.
Inline group locations in DocLang and DocTags are a union of the runs, so
they do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…acy runs

The legacy separator migration counted only direct text children with
non-empty text as runs. An empty-text item whose content is a child inline
group (a formula or footnote host) and an inline group nested directly in
another were skipped, so `left`, host(`x`), `right` loaded as
`left $x$right` instead of the old `left $x$ right`. The DocLang
deserializer writes both shapes for nested mixed-content elements.

A child now counts as a run when it or its subtree has text. Only a plain
run with its own text takes the separator; an empty host must keep its
empty text to render its group, so it is treated like a code or formula run
and the separator stays outside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…parators

DocLang has no schema version, so a .dclx written before schema 1.11.0
cannot be told apart from a current one and is not migrated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
This reverts 6a8b837; the change is unrelated to the whitespace contract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@mergify

mergify Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit —

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@nanhanq1

Copy link
Copy Markdown

Heads-up on the red DCO check — the remediation commit already on this branch is aimed at the wrong commits, so it can't clear it.

f768d757 ("DCO Remediation Commit for Christoph Auer") covers:

  • 64404b3709d9e3ba7caf08957cf2ef19df29459f
  • 66040cb7b96f77699c9d74051500d645e7e758ed

Those are pre-rebase SHAs; neither is in the branch any more. The two commits that are actually missing a sign-off right now are:

  • 7600953bc08ca8b33ea758c8a8192897515c4d86 — fix(doclang): preserve leading text before a lone formatting tag
  • f0e26b6722e2c1b8c543a8b68f04641772f3ef5a — feat: inline runs carry their own whitespace; serializers concatenate…

Everything else in the branch is signed off, including the merge of main (be82796c) and @nanhanq1's 1fcf2e9f, so a remediation commit covering just these two SHAs should be enough — no force push needed:

DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com>

I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 7600953bc08ca8b33ea758c8a8192897515c4d86
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: f0e26b6722e2c1b8c543a8b68f04641772f3ef5a

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

The other blocker is the Mergify "Require two reviewer for test updates" protection, which is waiting on two reviews — nothing to fix in the branch for that one.

cau-git and others added 3 commits October 26, 2026 10:53
project_to validated the projected dict under its target stamp. For 1.10.0 that ran the load-time separator migration on runs that already carry their whitespace, so 'Normal ', 'bold', ' tail' came back as 'Normal  ', 'bold ', ' tail'. Validate under the source stamp and restore the target stamp afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
A 1.10 client joins inline runs with " ", so runs that keep their boundary whitespace rendered with double spaces and with the space inside the markup (**bold **, which CommonMark does not render as bold). The projector now strips each inline-group child's text to the shape old producers emitted. Whitespace-only runs become "", which old serializers skip, so no refs move. orig and text outside inline groups are left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Between two code or formula runs the migration inserts a plain " " run. The 1.11 -> 1.10 projector strips it to "", and the next migration skipped the empty run and inserted another, so every projection round trip added a run. Refill an empty plain run between the two neighbours instead. Old serializers skipped empty runs, so legacy output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants