Skip to content

fix(md): keep Markdown content intact around HTML blocks - #4848

Open
avanish-garg wants to merge 2 commits into
docling-project:mainfrom
avanish-garg:fix/md-html-block-roundtrip
Open

avanish-garg wants to merge 2 commits into
docling-project:mainfrom
avanish-garg:fix/md-html-block-roundtrip

Conversation

@avanish-garg

Copy link
Copy Markdown

Issue resolved by this Pull Request:
Resolves #4847

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

Review evidence:

Before: any Markdown HTML block (even <!-- c -->) routed the whole document through export_to_html() and
HTMLDocumentBackend, which dropped <CONDA_NAME>, <stdio.h>, Dict<str>, inline <T> and x<y>, collapsed
newlines in code, merged adjacent paragraphs and lost pictures (repro in the issue).

After: each HTML block is converted on its own with HTMLDocumentBackend (new convert_into(doc), convert()
calls it unchanged) and appended at the block's position under the current parent; Markdown-native items are
never exported to HTML again, and the marker machinery and _html_blocks are removed. With or without a
leading/trailing comment the document tree and export are identical, and Markdown without HTML blocks is
unchanged. Per-block conversion is also faster (200 sections with 200 HTML blocks: 0.18 s vs 0.40 s).

Rejected: doc.add_document(parent=...) (docling-core 2.101.1): it deep-copies items but does not remap
cross-references; measured a picture caption pointing at the wrong text and a rich table cell making
validate_tree() False and export_to_markdown() raise. Escaping code text before the round trip is not
lossless and would not fix paragraph merging or lost pictures.

Reference data (regenerated with --update-groundtruth, each diff reviewed):

  • escaped_characters.md.json/.md: code items keep &amp; &lt; ... literally (as the same input without an HTML
    block does); origin mimetype is text/markdown (was text/html); the tree is flat (was nested under headings
    only when an HTML block was present).
  • signature_stamp_01.md.json: only origin mimetype and binary_hash (the origin was overwritten by the HTML backend).
  • mixed_without_h1.md.yaml: flat tree, origin as above; the 2\. list item is an inline group (2, .,
    strange), which is what the same Markdown gives without a comment; the .md export text is unchanged.
  • inline_and_formatting.md.yaml was regenerated on main alone too and only reorders groups: not committed.

Tests added (tests/test_backend_markdown.py): content, code, table cell and list numbering identical with no
HTML block / leading / trailing / both; adjacent paragraphs stay separate; Markdown pictures survive; HTML block
content (h2, table, details/summary, img, malformed div) still converted; HTML inside a list item; references
(caption, rich cell) stay valid. On main 6 of them fail (plus test_convert_valid against the new reference data).
Commands: pytest tests/test_backend_markdown.py tests/test_backend_html.py -> 123 passed, 1 skipped;
mhtml, text_encoding, epub and topic_labels tests pass; ruff check and ruff format --check clean; ty reports
139 diagnostics before and after (none new).
Not run (packages missing in the light environment): test_backend_email.py (mailparser), test_invalid_input.py
and test_backend_optional_dependencies.py (docling_parse/httpx), test_input_doc.py (pypdfium2); no PDF/ML
pipelines.

Behaviour that changes and may deserve a look: a multi-block <details> split by blank lines no longer nests its
Markdown body (it is a sibling of the details group) and unbalanced <div> blocks are no longer paired across
blocks; documents with HTML blocks use the flat Markdown structure; an HTML-block heading that starts at h2/h3
gets the HTML backend's header-N wrapper groups; HTML inside a list item becomes a child of the list group like
code blocks today (#4513 addresses nesting); new public HTMLDocumentBackend.convert_into. Follow-up worth a
docling-core issue: add_document/insert_document do not remap captions or rich-cell references.

@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @avanish-garg, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit —

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) bug Something isn't working html issue related to html backend markdown issue related to markdown backend performance labels Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

AI triage (advisory)

Checked commit d3ac58b144. The analysis read the diff only. It did not run code from this pull request.

The PR replaces the whole-document HTML round-trip in md_backend.py with per-block conversion using a new convert_into() method on HTMLDocumentBackend, removing markdown_html.py and fixing data loss for angle-bracket tokens, collapsed newlines, merged paragraphs, and lost pictures.

CI safety: no concern found. A maintainer must still approve CI.

Possible duplicates

PR State Signal Assessment
#4985 open changes the same code duplicate: PR #4985 patches the same HTML-block conversion path in md_backend.py (fixing the unescape bug) but keeps the round-trip architecture; PR #4848 removes that architecture entirely — only one can be merged.
1 related pull request(s) in the same code
PR State Signal Assessment
#4513 open referenced as #4513 related: PR #4513 fixes list-item child attachment in md_backend.py and includes a test that acknowledges the HTML round-trip flaw; it touches overlapping code but solves a different, narrower problem.

Reference data changes

Change Files
text 3
  • text tests/data/md/groundtruth/escaped_characters.md.json: item attributes changed
  • text tests/data/md/groundtruth/mixed_without_h1.md.yaml: +56 / -26 lines
  • text tests/data/md/groundtruth/signature_stamp_01.md.json: item attributes changed

Assessment: expected. The PR description explains each reference change: origin mimetype corrected from text/html to text/markdown, document tree flattened, and binary hash updated because the HTML backend no longer overwrites the origin — all consistent with the code change.

The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.

@avanish-garg
avanish-garg force-pushed the fix/md-html-block-roundtrip branch from 2053a41 to b04745d Compare October 15, 2026 09:27
@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) and removed ai:ci-safe AI triage found no CI-safety concern (advisory) labels Oct 15, 2026
@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.73684% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
docling/backend/md_backend.py 93.75% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

The Markdown backend exported the whole document to HTML and parsed it
again as soon as the file had one HTML block, even a comment. The export
does not escape code text, so `<stdio.h>`, `Dict<str>` or `<CONDA_NAME>`
were read as tags and dropped, newlines in code collapsed, adjacent
paragraphs merged and pictures were lost.

Convert each HTML block on its own with the HTML backend and append the
result at the position of the block. Markdown content is no longer
exported and parsed again. The document origin is now text/markdown.

Signed-off-by: Avanish Garg <gargavanish@gmail.com>
The Markdown backend no longer exports the whole document to HTML and
parses it again, so nothing relays Markdown pictures through that round
trip. Remove `docling/backend/utils/markdown_html.py`, which has no
remaining users. The picture tests added with it keep passing unchanged.

Signed-off-by: Avanish Garg <gargavanish@gmail.com>
@avanish-garg
avanish-garg force-pushed the fix/md-html-block-roundtrip branch from b4984aa to d3ac58b Compare October 26, 2026 09:49
@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) ai:possible-duplicate AI triage: another PR may fix the same problem labels Oct 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:ci-safe AI triage found no CI-safety concern (advisory) ai:possible-duplicate AI triage: another PR may fix the same problem bug Something isn't working html issue related to html backend markdown issue related to markdown backend performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Markdown backend corrupts code blocks, table cells and structure when the file contains any HTML block

1 participant