Repository navigation
fix(md): keep Markdown content intact around HTML blocks - #4848
avanish-garg wants to merge 2 commits into
Conversation
|
✅ DCO Check Passed Thanks @avanish-garg, all your commits are properly signed off. 🎉 |
Merge Protections🔴 1 of 2 protections blocking · waiting on 👀 reviews
🔴 Require two reviewer for test updatesWaiting for
This rule is failing.When test data is updated, we require two reviewers
Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
AI triage (advisory)Checked commit The PR replaces the whole-document HTML round-trip in CI safety: no concern found. A maintainer must still approve CI. Possible duplicates
1 related pull request(s) in the same code
Reference data changes
Assessment: expected. The PR description explains each reference change: origin mimetype corrected from The |
2053a41 to
b04745d
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
b04745d to
b4984aa
Compare
The Markdown backend exported the whole document to HTML and parsed it again as soon as the file had one HTML block, even a comment. The export does not escape code text, so `<stdio.h>`, `Dict<str>` or `<CONDA_NAME>` were read as tags and dropped, newlines in code collapsed, adjacent paragraphs merged and pictures were lost. Convert each HTML block on its own with the HTML backend and append the result at the position of the block. Markdown content is no longer exported and parsed again. The document origin is now text/markdown. Signed-off-by: Avanish Garg <gargavanish@gmail.com>
The Markdown backend no longer exports the whole document to HTML and parses it again, so nothing relays Markdown pictures through that round trip. Remove `docling/backend/utils/markdown_html.py`, which has no remaining users. The picture tests added with it keep passing unchanged. Signed-off-by: Avanish Garg <gargavanish@gmail.com>
b4984aa to
d3ac58b
Compare
Issue resolved by this Pull Request:
Resolves #4847
Checklist:
Review evidence:
Before: any Markdown HTML block (even
<!-- c -->) routed the whole document throughexport_to_html()andHTMLDocumentBackend, which dropped<CONDA_NAME>,<stdio.h>,Dict<str>, inline<T>andx<y>, collapsednewlines in code, merged adjacent paragraphs and lost pictures (repro in the issue).
After: each HTML block is converted on its own with
HTMLDocumentBackend(newconvert_into(doc),convert()calls it unchanged) and appended at the block's position under the current parent; Markdown-native items are
never exported to HTML again, and the marker machinery and
_html_blocksare removed. With or without aleading/trailing comment the document tree and export are identical, and Markdown without HTML blocks is
unchanged. Per-block conversion is also faster (200 sections with 200 HTML blocks: 0.18 s vs 0.40 s).
Rejected:
doc.add_document(parent=...)(docling-core 2.101.1): it deep-copies items but does not remapcross-references; measured a picture caption pointing at the wrong text and a rich table cell making
validate_tree()False andexport_to_markdown()raise. Escaping code text before the round trip is notlossless and would not fix paragraph merging or lost pictures.
Reference data (regenerated with
--update-groundtruth, each diff reviewed):escaped_characters.md.json/.md: code items keep& < ...literally (as the same input without an HTMLblock does); origin mimetype is
text/markdown(wastext/html); the tree is flat (was nested under headingsonly when an HTML block was present).
signature_stamp_01.md.json: only origin mimetype and binary_hash (the origin was overwritten by the HTML backend).mixed_without_h1.md.yaml: flat tree, origin as above; the2\.list item is an inline group (2,.,strange), which is what the same Markdown gives without a comment; the.mdexport text is unchanged.inline_and_formatting.md.yamlwas regenerated onmainalone too and only reorders groups: not committed.Tests added (
tests/test_backend_markdown.py): content, code, table cell and list numbering identical with noHTML block / leading / trailing / both; adjacent paragraphs stay separate; Markdown pictures survive; HTML block
content (h2, table, details/summary, img, malformed div) still converted; HTML inside a list item; references
(caption, rich cell) stay valid. On
main6 of them fail (plustest_convert_validagainst the new reference data).Commands:
pytest tests/test_backend_markdown.py tests/test_backend_html.py-> 123 passed, 1 skipped;mhtml, text_encoding, epub and topic_labels tests pass;
ruff checkandruff format --checkclean;tyreports139 diagnostics before and after (none new).
Not run (packages missing in the light environment):
test_backend_email.py(mailparser),test_invalid_input.pyand
test_backend_optional_dependencies.py(docling_parse/httpx),test_input_doc.py(pypdfium2); no PDF/MLpipelines.
Behaviour that changes and may deserve a look: a multi-block
<details>split by blank lines no longer nests itsMarkdown body (it is a sibling of the details group) and unbalanced
<div>blocks are no longer paired acrossblocks; documents with HTML blocks use the flat Markdown structure; an HTML-block heading that starts at h2/h3
gets the HTML backend's
header-Nwrapper groups; HTML inside a list item becomes a child of the list group likecode blocks today (#4513 addresses nesting); new public
HTMLDocumentBackend.convert_into. Follow-up worth adocling-core issue:
add_document/insert_documentdo not remap captions or rich-cell references.