feat: inline whitespace contract for inline runs (rebase of #693 onto main) by nanhanq1 · Pull Request #843 · docling-project/docling-core

nanhanq1 · 2026-10-06T08:47:37Z

Rebases cau/serializer-whitespace-fixes (the branch behind #693) onto the current main
(3e42dba), so #693 stops being 60 commits behind and can land on today's tree.

This supersedes #693 rather than trying to update its branch: #693's head is
cau/serializer-whitespace-fixes, and a PR into that branch from a branch rebased onto main
would drag all of main along with it and then collide with the branch's original copies of these
same five commits. Same content, rebased and green, offered directly against main — close #693
in favour of this one if that is easier.

Conflicts

11 conflicted paths:

docling_core/transforms/deserializer/doclang.py — kept both sides: main's
_repair_unescaped_xml_text alongside this branch's _normalize_bare_text and the trim=
handling at the call site (a run inside an inline group owns its edge whitespace, a block host
does not).
docling_core/transforms/serializer/markdown.py — merged the import lines.
pyproject.toml, examples/, docling_core/compat.py, tests/* — adopted upstream's
test/ -> tests/ rename (chore: rename test folder to tests #739).
6 golden .json / .yaml files — three-way merged against the merge base, then re-validated.

Three extra commits on top

1. fix(doclang): keep inherited formatting on <content> runs in inline groups

<content> is the explicit whitespace-preserving wrapper, so inside an inline group it is a run,
not a nested item, and has to inherit the formatting accumulated while descending through
<bold> / <italic> / ... . Routing it through _dispatch_element dropped that formatting. This
only surfaced after the rebase: <bold><content>x </content></bold> came back unformatted and
broke test_doclang_roundtrip_is_lossless[whitespace_inside_bold] and [..._link].

2. style: use PEP 604 unions instead of Optional[...] (ruff UP045)

main now lints with ruff 0.16.5 at target-version = "py310", where UP045 rejects
Optional[X]. Without this the branch fails ruff check.

3. test: align expectations with the inline whitespace contract

the run following a CDATA section owns its leading space, so
<![CDATA[a & b < c]]> & d yields ["a & b < c", " & d"];
dropped the duplicate loop-shaped test_leading_text_before_lone_formatting_tag_is_preserved,
which silently shadowed main's parametrized test of the same name;
the LaTeX inline-group fixture runs now carry their own boundary whitespace, since the
serializer no longer invents a separator.

Verification

ruff check and ruff format --check: clean.
A/B against main in the same environment on test_deserializer_doclang.py,
test_latex_serialization.py, test_compat.py: main 11 failed / 168 passed, this branch
11 failed / 172 passed, and the two failure sets are identical (diffed both ways, empty). The
11 are an artifact of the sandbox I ran in: doclang's Schematron validation writes its temp
file to C:\WINDOWS\TEMP, which is blocked here, so Saxon reports SXXP0003. Not a regression.
tests/test_inline_whitespace_contract.py (new with this branch) passes except
test_chunk_text_follows_the_contract, which needs the optional huggingface_hub extra that
is not installed here.

Merge order

This has to land before docling-project/docling#3891 — new core + old backends means inline runs
lose their boundary whitespace and words get merged.

Happy to squash the three extra commits into the branch's own commits, or to re-target this at
main, if either is preferred.

github-actions · 2026-10-06T08:47:51Z

✅ DCO Check Passed

Thanks @nanhanq1, all your commits are properly signed off. 🎉

mergify · 2026-10-06T08:50:25Z

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

	Protection	Waiting on
🔴	Require two reviewer for test updates	👀 reviews
🟢	Enforce conventional commit	—

🔴 Require two reviewer for test updates

Waiting for

#approved-reviews-by >= 2

This rule is failing.

When test data is updated, we require two reviewers

#approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:$.+$)?(!)?:

nanhanq1 · 2026-10-06T10:21:50Z

Run CI is sitting at action_required (first PR from this fork), so no test job has been created yet. Could a maintainer approve the workflow run when convenient?

`_get_children_simple_text_block` guarded against a second *text* fragment but the Element branch just overwrote `result`, so `<text>2<superscript>nd</superscript></text>` returned "nd" and the leading `2` was silently discarded. Guard both branches so the element is routed to `_parse_inline_group`, which already handles mixed children. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

… faithfully The Markdown, HTML, LaTeX and DocLang inline serializers joined runs with a hard `" "`, inventing spaces the model never asked for: `2nd` -> `2 nd`, `O2.` -> `O 2 .`, `italic text.` -> `italic text .`. Those artifacts can never be removed while the separator lives in the serializer. Serializers: - join inline parts with `""`; DocLang keeps its delimiter between the element head and the body only, where `<content>` is the whitespace channel - hoist edge whitespace around the *complete* decoration stack (formatting and hyperlink together) for Markdown, HTML and LaTeX. Decorating it verbatim emits `**bold ** tail`, which CommonMark renders as literal asterisks; hoisting inside one formatter would let the hyperlink recapture it. DocLang opts out: it keeps whitespace inside the markup, in `<content>`. DocLang deserializer: - `_get_text` stripped every fragment and joined with `""`, collapsing `Advanced <bold>Topics</bold>` into `AdvancedTopics`. It now trims the outer block boundary only, and never trims whitespace delivered by `<content>`. - bare text nodes use the standard XML heuristic: a whitespace run containing a newline is pretty-print indentation, one without it is content. This is what hand-written and VLM-emitted DocLang needs, since it will not use `<content>`. - element-head children (`<caption>`, `<description>`, `<summary>`, `<custom>`) no longer bleed into body text, where they were also double-modelled in `item.meta`. Data at rest: schema minor bumped to 1.11.0 and a `mode="before"` validator re-injects the legacy separator for documents stamped below it, preserving their previous *visible* output including its extra-space bugs (`H 2 O` stays `H 2 O`). It must run before `check_version_is_compatible`, which rewrites `version` and destroys the evidence. Versionless input is left alone. The validator works on a copy, so the caller's dict is untouched and re-validating the same raw value cannot compound separators. A separator can never land inside a delimiter. Code and formula runs are wrapped in `` ` ``/`$` by the serializers, so the space goes on the plain neighbour; when both neighbours are delimited it becomes a plain run of its own, because `` `a``b` `` and `$a$$b$` are both corrupt. Chunk text changes too, since ChunkingDocSerializer inherits the Markdown join -- anyone with a persisted vector index gets silent retrieval drift, so that needs its own line in the release notes. DocTags code is untouched. Its golden gains one empty `<text></text>` for the legacy fixture because that document now carries an extra whitespace run, which DocTags renders empty by its own pre-existing strip-per-run behaviour. Ships as a minor, not a major: `feat!:` would make python-semantic-release (7.34.6, both repos) bump to 3.0, and a 3.0 is not affordable on either package. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 64404b3 I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 66040cb Signed-off-by: Christoph Auer <cau@zurich.ibm.com> Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

…ct bump Integrates the 1.11.0 schema bump with the downgrade-projector framework (docling-project#703) landed upstream. 1.11 and 1.10 are structurally identical; the only difference is that 1.11 inline runs own their boundary whitespace, so the projector just re-stamps the version (a 1.10 client re-applies its own join, yielding the documented harmless double space). - add @register_projector(from_minor=11, to_minor=10) + TestProjector_1_11_to_1_10 - add the auto-generated 1.11 JSON Schema snapshot - re-format the 1.11 SCHEMA_VERSION_HISTORY entry (ruff) - update the stale dclq CLI expectation (old join produced 'public class'; faithful join now yields 'public class') Signed-off-by: Christoph Auer <cau@zurich.ibm.com> Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

typer >=0.25 vendors its own copy of click (typer._click) and raises those exceptions instead of the external click package's. The dclq CLI referenced the external click, so on typer >=0.25 bad input escaped the exit-code handlers and the context-flag callback (-C/-A/-B/--context-scope) silently no-op'd because the isinstance(param, click.Option) check no longer matched. Import the exception and Parameter types from typer._click, duck-type the option check, raise the [dclq] typer floor to >=0.25, and drop the now-unused external click dependency. Signed-off-by: Christoph Auer <cau@zurich.ibm.com> Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

…roups <content> is the explicit whitespace-preserving text wrapper, so inside an inline group it is a run, not a nested item: it must inherit the formatting accumulated while descending through <bold>/<italic>/... . Dispatching it through _dispatch_element dropped that formatting, which surfaced after the rebase as <bold><content>x </content></bold> coming back unformatted and broke the doclang round-trip for whitespace_inside_bold / _link. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

main now lints with ruff 0.16.5 at target-version py310, where UP045 flags Optional[X] in favour of X | None. Without this the branch fails ruff check. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

- doclang: the run following a CDATA section owns its leading space, so "<![CDATA[a & b < c]]> & d" now yields ["a & b < c", " & d"]. - doclang: drop the duplicate loop-shaped test_leading_text_before_lone_formatting_tag_is_preserved, which main already covers with the parametrized version; keeping both silently shadowed main's. - latex: give the inline-group fixture runs their own boundary whitespace, since the serializer no longer invents a separator. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

nanhanq1 · 2026-10-14T09:33:52Z

Rebased onto main (cbb86ff) — the branch no longer conflicts, and DCO / dco_advisor are green.

Three things had to be reconciled with upstream work:

Deserializer _get_text. Upstream restructured it and added DocLangToken.CUSTOM to the skip set. The rebased version routes the skip through the existing _is_element_head_tag(), whose _ELEMENT_HEAD_TAGS already contains CUSTOM, so the upstream behaviour is preserved.
Inline serializer location. Upstream dropped params.add_location = False in DocLineInlineSerializer in favour of head_params (set in DocLangTextSerializer). Took upstream's approach and kept only the head_parts / inline_parts split.
Duplicate regression test. Upstream now carries a parametrized test_leading_text_before_lone_formatting_tag_is_preserved that covers the same shapes plus two more, so the loop-shaped duplicate from this branch is dropped in favour of upstream's.

Local gates on the rebased tree: ruff format --check, ruff check, and pytest green on every touched module (720 passed vs. 485 on unmodified main in the same sandbox). The residual sandbox failures are pre-existing and unrelated: the Saxon/Schematron backend cannot read its temp files here, and two chunker modules need transformers.

Two things still need a maintainer:

Run CI is at action_required again after the force-push (first PR from this fork) — approving it would let the test job run.
Mergify still flags Require two reviewer for test updates: this PR updates ground-truth data, so it needs two approvals.

nanhanq1 · 2026-10-15T07:05:05Z

Pinging maintainers for the two things this PR still needs. The branch is rebased onto main and no longer conflicts (MERGEABLE), and DCO / dco_advisor are green.

1. Run CI is at action_required for the current head 18a29f3 (first PR from this fork), so no test job has been created yet. One "Approve and run" would let it execute.

2. Mergify still blocks on Require two reviewer for test updates — this PR refreshes ground-truth data under tests/data, so it needs two approvals.

Recap of the rebase (details in the comment above):

Deserializer _get_text — upstream added DocLangToken.CUSTOM to the skip set; the rebased code routes the skip through the existing _is_element_head_tag(), whose _ELEMENT_HEAD_TAGS already contains CUSTOM.
Inline serializer location — took upstream's head_params approach and dropped the old params.add_location = False.
Duplicate regression test — dropped the loop-shaped duplicate in favour of upstream's parametrized test_leading_text_before_lone_formatting_tag_is_preserved.
Local gates (ruff format --check, ruff check, pytest) are green on every touched module.

@ceberam @vagenas @PeterStaar-IBM @cau-git @dolfim-ibm — would any of you have time to take a look, or point me at the right reviewers? Thanks!

nanhanq1 changed the base branch from cau/serializer-whitespace-fixes to main October 6, 2026 08:49

nanhanq1 force-pushed the rb693 branch from bd77f1f to 401bd72 Compare October 6, 2026 08:50

nanhanq1 changed the title ~~Rebase #693 onto main: resolve conflicts and fix <content> formatting loss~~ feat: inline whitespace contract for inline runs (rebase of #693 onto main) Oct 6, 2026

nanhanq1 mentioned this pull request Oct 6, 2026

feat: make inline serializers concatenate runs faithfully #693

Open

cau-git and others added 8 commits October 14, 2026 16:51

style: use PEP 604 unions instead of Optional[...] (ruff UP045)

08b7878

main now lints with ruff 0.16.5 at target-version py310, where UP045 flags Optional[X] in favour of X | None. Without this the branch fails ruff check. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

nanhanq1 force-pushed the rb693 branch from 401bd72 to 18a29f3 Compare October 14, 2026 09:32

yashikam19 mentioned this pull request Oct 22, 2026

feat(pdf): mark superscript and subscript text from glyph size and baseline docling-project/docling#4815

Open

2 of 3 tasks

Navigation Menu

Repository navigation

feat: inline whitespace contract for inline runs (rebase of #693 onto main) - #843

feat: inline whitespace contract for inline runs (rebase of #693 onto main)#843
nanhanq1 wants to merge 8 commits into
docling-project:mainfrom
nanhanq1:rb693

nanhanq1 commented Oct 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 6, 2026 •

edited

Loading

Uh oh!

mergify Bot commented Oct 6, 2026 •

edited

Loading

🟢 Enforce conventional commit

Uh oh!

nanhanq1 commented Oct 6, 2026

Uh oh!

nanhanq1 commented Oct 14, 2026

Uh oh!

nanhanq1 commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nanhanq1 commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Three extra commits on top

Verification

Merge order

Uh oh!

github-actions Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🔴 Require two reviewer for test updates

🟢 Enforce conventional commit

Uh oh!

nanhanq1 commented Oct 6, 2026

Uh oh!

nanhanq1 commented Oct 14, 2026

Uh oh!

nanhanq1 commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nanhanq1 commented Oct 6, 2026 •

edited

Loading

github-actions Bot commented Oct 6, 2026 •

edited

Loading

mergify Bot commented Oct 6, 2026 •

edited

Loading