Repository navigation
Conversation
|
❌ DCO Check Failed Hi @cau-git, your pull request has failed the Developer Certificate of Origin (DCO) check. This repository supports remediation commits, so you can fix this without rewriting history — but you must follow the required message format. 🛠 Quick Fix: Add a remediation commitRun this command: git commit --allow-empty -s -m "DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com>
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 7600953bc08ca8b33ea758c8a8192897515c4d86
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: f0e26b6722e2c1b8c543a8b68f04641772f3ef5a"
git push🔧 Advanced: Sign off each commit directlyFor the latest commit: git commit --amend --signoff
git push --force-with-leaseFor multiple commits: git rebase --signoff origin/main
git push --force-with-leaseMore info: DCO check report |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
`_get_children_simple_text_block` guarded against a second *text* fragment but the Element branch just overwrote `result`, so `<text>2<superscript>nd</superscript></text>` returned "nd" and the leading `2` was silently discarded. Guard both branches so the element is routed to `_parse_inline_group`, which already handles mixed children.
… faithfully The Markdown, HTML, LaTeX and DocLang inline serializers joined runs with a hard `" "`, inventing spaces the model never asked for: `2nd` -> `2 nd`, `O2.` -> `O 2 .`, `italic text.` -> `italic text .`. Those artifacts can never be removed while the separator lives in the serializer. Serializers: - join inline parts with `""`; DocLang keeps its delimiter between the element head and the body only, where `<content>` is the whitespace channel - hoist edge whitespace around the *complete* decoration stack (formatting and hyperlink together) for Markdown, HTML and LaTeX. Decorating it verbatim emits `**bold ** tail`, which CommonMark renders as literal asterisks; hoisting inside one formatter would let the hyperlink recapture it. DocLang opts out: it keeps whitespace inside the markup, in `<content>`. DocLang deserializer: - `_get_text` stripped every fragment and joined with `""`, collapsing `Advanced <bold>Topics</bold>` into `AdvancedTopics`. It now trims the outer block boundary only, and never trims whitespace delivered by `<content>`. - bare text nodes use the standard XML heuristic: a whitespace run containing a newline is pretty-print indentation, one without it is content. This is what hand-written and VLM-emitted DocLang needs, since it will not use `<content>`. - element-head children (`<caption>`, `<description>`, `<summary>`, `<custom>`) no longer bleed into body text, where they were also double-modelled in `item.meta`. Data at rest: schema minor bumped to 1.11.0 and a `mode="before"` validator re-injects the legacy separator for documents stamped below it, preserving their previous *visible* output including its extra-space bugs (`H 2 O` stays `H 2 O`). It must run before `check_version_is_compatible`, which rewrites `version` and destroys the evidence. Versionless input is left alone. The validator works on a copy, so the caller's dict is untouched and re-validating the same raw value cannot compound separators. A separator can never land inside a delimiter. Code and formula runs are wrapped in `` ` ``/`$` by the serializers, so the space goes on the plain neighbour; when both neighbours are delimited it becomes a plain run of its own, because `` `a``b` `` and `$a$$b$` are both corrupt. Chunk text changes too, since ChunkingDocSerializer inherits the Markdown join -- anyone with a persisted vector index gets silent retrieval drift, so that needs its own line in the release notes. DocTags code is untouched. Its golden gains one empty `<text></text>` for the legacy fixture because that document now carries an extra whitespace run, which DocTags renders empty by its own pre-existing strip-per-run behaviour. Ships as a minor, not a major: `feat!:` would make python-semantic-release (7.34.6, both repos) bump to 3.0, and a 3.0 is not affordable on either package.
…ct bump Integrates the 1.11.0 schema bump with the downgrade-projector framework (#703) landed upstream. 1.11 and 1.10 are structurally identical; the only difference is that 1.11 inline runs own their boundary whitespace, so the projector just re-stamps the version (a 1.10 client re-applies its own join, yielding the documented harmless double space). - add @register_projector(from_minor=11, to_minor=10) + TestProjector_1_11_to_1_10 - add the auto-generated 1.11 JSON Schema snapshot - re-format the 1.11 SCHEMA_VERSION_HISTORY entry (ruff) - update the stale dclq CLI expectation (old join produced 'public class'; faithful join now yields 'public class') Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
79ebc53 to
e4e745a
Compare
typer >=0.25 vendors its own copy of click (typer._click) and raises those exceptions instead of the external click package's. The dclq CLI referenced the external click, so on typer >=0.25 bad input escaped the exit-code handlers and the context-flag callback (-C/-A/-B/--context-scope) silently no-op'd because the isinstance(param, click.Option) check no longer matched. Import the exception and Parameter types from typer._click, duck-type the option check, raise the [dclq] typer floor to >=0.25, and drop the now-unused external click dependency. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
|
Rebased this onto current |
Conflicts resolved to the versions reviewed on the rebased branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…roups <content> is the explicit whitespace-preserving text wrapper, so inside an inline group it is a run, not a nested item: it must inherit the formatting accumulated while descending through <bold>/<italic>/... . Dispatching it through _dispatch_element dropped that formatting, which surfaced after the rebase as <bold><content>x </content></bold> coming back unformatted and broke the doclang round-trip for whitespace_inside_bold / _link. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>
The whitespace-contract bump was first written against v2.86.0; main is now at v2.104.0, so record v2.105.0 as the placeholder release that ships schema 1.11.0. Adjust to the actual version at release time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Regenerated with DOCLING_GEN_TEST_DATA=1. The only change is the schema version stamp 1.10.0 -> 1.11.0 in goldens added or moved on main (including the six test/ -> tests/ renames and misplaced_list_items.out.yaml). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
_collect_subtree_text (TextItem._get_plain_text, FloatingItem.caption_text, nested rich table cells) still joined the runs of an InlineGroup with " ", so mixed captions read "Table 1 : results". It now concatenates the runs of an inline group; every other join, including a host text item's own text and its children, keeps its space. Children of a text item are block structure (producers put a section's paragraphs under its heading), so the host text stays apart from its inline group, as on main; add_inline_group documents this. Tests that asserted the invented run spacing now expect the contract output, fixtures that built runs without boundary whitespace now carry it, and a new test pins the host-text boundary for headings and captions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
DocTags stores inline runs trimmed, so the boundary whitespace the inline whitespace contract expects is not in the source. Document the limitation instead of guessing separators back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
The Markdown list serializer glued an inline group onto the preceding list item with no separator, so a list item "Intro" whose child group starts with "more" exported "- Intromore". Captions, headings and the plain-text reader keep a space between host text and its inline group; the list serializer now does the same when the group is the child of a text item with its own text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
… page filters A legacy document with two adjacent code or formula runs gets a standalone " " run between them on load. It had no provenance, so filter(page_nrs=...) and export_to_markdown(page_no=...) dropped it and rendered `$a$$b$`. The separator now takes the left run's location when both runs are on the same page, so it survives a page filter exactly when both runs do. Runs on different pages keep the unlocated separator, which matches the old join. Inline group locations in DocLang and DocTags are a union of the runs, so they do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…acy runs The legacy separator migration counted only direct text children with non-empty text as runs. An empty-text item whose content is a child inline group (a formula or footnote host) and an inline group nested directly in another were skipped, so `left`, host(`x`), `right` loaded as `left $x$right` instead of the old `left $x$ right`. The DocLang deserializer writes both shapes for nested mixed-content elements. A child now counts as a run when it or its subtree has text. Only a plain run with its own text takes the separator; an empty host must keep its empty text to render its group, so it is treated like a code or formula run and the separator stays outside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…parators DocLang has no schema version, so a .dclx written before schema 1.11.0 cannot be told apart from a current one and is not migrated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
This reverts 6a8b837; the change is unrelated to the whitespace contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Merge Protections🔴 1 of 2 protections blocking · waiting on 👀 reviews
🔴 Require two reviewer for test updatesWaiting for
This rule is failing.When test data is updated, we require two reviewers
Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
|
Heads-up on the red DCO check — the remediation commit already on this branch is aimed at the wrong commits, so it can't clear it.
Those are pre-rebase SHAs; neither is in the branch any more. The two commits that are actually missing a sign-off right now are:
Everything else in the branch is signed off, including the merge of The other blocker is the Mergify "Require two reviewer for test updates" protection, which is waiting on two reviews — nothing to fix in the branch for that one. |
project_to validated the projected dict under its target stamp. For 1.10.0 that ran the load-time separator migration on runs that already carry their whitespace, so 'Normal ', 'bold', ' tail' came back as 'Normal ', 'bold ', ' tail'. Validate under the source stamp and restore the target stamp afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
A 1.10 client joins inline runs with " ", so runs that keep their boundary whitespace rendered with double spaces and with the space inside the markup (**bold **, which CommonMark does not render as bold). The projector now strips each inline-group child's text to the shape old producers emitted. Whitespace-only runs become "", which old serializers skip, so no refs move. orig and text outside inline groups are left alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Between two code or formula runs the migration inserts a plain " " run. The 1.11 -> 1.10 projector strips it to "", and the next migration skipped the empty run and inserted another, so every projection round trip added a run. Refill an empty plain run between the two neighbours instead. Old serializers skipped empty runs, so legacy output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Update: merged
main, review fixesmain(b501ec80, v2.104.0) is merged into this branch. The conflicts were resolved to the versions reviewed on a rebased copy, and the newer commits were replayed on top, so no force push was needed.7600953b(2<superscript>nd</superscript>lost its leading2) is gone from the diff: fix(doclang): keep label, provenance, layer and leading text of mixed-content elements #804 landed the same fix onmain.6a8b8374is reverted here. It is unrelated to the contract and moves to its own PR.<content>runs inside inline groups), with authorship kept. Thank you. feat: inline whitespace contract for inline runs (rebase of #693 onto main) #843 can be closed once this merges.New since the last push:
" ", as onmain. Children of a text item are block structure. With"", headings merged with their first formatted paragraph (ResultsWe use bold words.). A producer that wants text flush against runs puts that text in the group as a run.- Intromore→- Intro more).page_no=…).main).Summary
InlineGroupruns now carry their own significant whitespace, and the serializers concatenate them faithfully instead of joining with a hard" ".The join was the defect. It invented spaces the model never asked for, and while the separator lived in the serializer, a producer fix could never remove them:
2<sup>nd</sup>2 nd2ndO<sub>2</sub>.O 2 .O2.**Advanced Topics** in *Machine Learning***Advanced Topics** in *Machine Learning***Advanced Topics** in *Machine Learning*This is *italic text*.This is *italic text* .This is *italic text*.The ODF backend already emitted runs that follow the contract, so its committed Markdown goldens show the reverse artifact:
X 2 + Y 2 = Z. That is the shortest proof that the serializer, not the producer, was wrong.What changed
Serializers. Markdown, HTML, LaTeX, DocLang and the plain-text reader join inline parts with
"". DocLang keeps its separator only between the element head and the body;<content>remains its whitespace channel.Whitespace hoisting. In DOCX and HTML the boundary space often sits inside a formatted run. Emitting it as-is produces
**bold ** tail, which CommonMark renders as literal asterisks. Markdown, HTML and LaTeX therefore split each run into leading whitespace, core text and trailing whitespace, apply the complete decoration stack to the core, and put the whitespace back outside. A single formatter can't do this alone, because the hyperlink would capture the space again ([**bold** ](url)). DocLang opts out on purpose and keeps the whitespace inside the markup, in<content>.DocLang deserializer.
_get_textstripped every fragment before joining, which collapsedAdvanced <bold>Topics</bold>intoAdvancedTopics. It now trims only the outer edges of the block, and never trims whitespace delivered by<content>.<content>for runs that carry whitespace, so this rule only matters for hand-written and VLM-emitted input.<caption>,<description>,<summary>,<custom>) no longer leak into the visible body text. They were also modelled a second time initem.meta.Compatibility
Saved
.jsonfiles contain runs without spaces, and coordinating releases can't fix a file on disk. The schema minor is bumped to 1.11.0. Amodel_validator(mode="before")re-inserts the legacy separator for documents stamped below that version. A 1.11→1.10 downgrade projector is registered with the #703 framework. It strips inline runs to the shape old producers emitted, so a 1.10 client renders a projected document the way it renders an old one (H 2 O, but no double spaces and no**bold **).The goal is narrow: keep the old serializer's visible output, including its extra-space bugs. An old document doesn't hold enough information to recover the real whitespace:
["H", "2", "O"]and["Normal", "italic"]have the same shape but need different boundaries.mode="before", becausecheck_version_is_compatiblerewritesversiontoCURRENT_VERSIONand so destroys the evidence.`and$, so the space goes to the plain neighbour. When both neighbours are delimited, the space becomes a plain run of its own, because`ab`` andLimitations
.dclx. DocLang written by docling-core before schema 1.11.0 is loaded as current, because DocLang carries no schema version and old and new output have the same structure for tight runs. Inline runs from such archives join without the old separator (Normal**bold**tail). Re-convert the source, or use JSON (which is migrated on load) for long-term storage. In the other direction, the old core reads new archives with the old" "join (H 2 O, and double spaces at runs that carry whitespace).load_from_doctagscan't restore run whitespace, because the DocTags serializer already stripped it. Legacy DocTags inline groups export with merged words. This is documented on the method.Behaviour changes to expect
HierarchicalChunker.ChunkingDocSerializerdefines no inline serializer, so it inherits the Markdown join. This changes embeddings, so anyone with a persisted vector index will see silent retrieval drift on upgrade. It needs its own line in the release notes.HybridChunkerserializes each item separately and joins items withdelim; that behaviour already exists and is tracked separately.The failure mode inverts
Worth flagging for reviewers: today, a producer that leaves out boundary whitespace produces extra spaces. That is ugly but harmless, and it has gone unnoticed for as long as it has existed. After this change the same omission produces merged words. The cost of a mistake goes from cosmetic to severe, and it falls on third-party backends and on code that builds
DoclingDocuments by hand. Theadd_inline_groupdocstring now states the contract so the next backend author sees it.Testing
1160 passed, 10 skipped, 1 xfailed, plus the compat-coverage gate, ty and ruff. Regenerating goldens withDOCLING_GEN_TEST_DATA=1changes nothing.The new
tests/test_inline_whitespace_contract.pyholds the contract as a table: one set of inline-run sequences checked againstmd,itxt,html,latexanddoclang. The table includes whitespace inside a formatted run, the same inside a formatted run with a hyperlink, and a whitespace-only run. The file also covers:Pre-contract fixtures keep their old version stamp on purpose. Their output is unchanged, so they act as the regression test for the migration.
Release notes
feat!:andBREAKING CHANGE:, which would make python-semantic-release bump to 3.0.docling-coreindependently (docling-serve, docling-eval, docling-mcp, docling-jobkit) need a floor bump in the same release wave.🤖 Generated with Claude Code