Skip to content

fix(md): splice raw HTML blocks back instead of decoding them - #5019

Closed
cau-git wants to merge 2 commits into
mainfrom
cau/md-raw-html-escape
Closed

cau-git wants to merge 2 commits into
mainfrom
cau/md-raw-html-escape

Conversation

@cau-git

@cau-git cau-git commented Oct 26, 2026

Copy link
Copy Markdown
Member

Summary

The Markdown backend converts documents that contain raw HTML blocks by round-tripping through HTML. It puts each raw block into a CodeItem between markers, serializes the document to HTML, and swaps each marked <pre><code> block back for the raw HTML.

Since docling-core 2.104.0 (docling-core#894) the HTML serializer escapes code text. #4771 adapted to this by unescaping each marked block once. That assumes the serializer escaped the block exactly once, which only holds for 2.104.0 and later:

docling-core main this PR
2.104.0 correct correct
2.103.1 (allowed by the current floor) entities in raw HTML decoded twice: &amp; becomes & correct

Change

  • fix(md): the backend stores the raw blocks in a list and puts only an index between the markers. convert() replaces each marked block with the stored HTML, so the result no longer depends on how the serializer treats code text. That includes any future change to its escaping.
  • build: raises the floor to docling-core>=2.104.0. The Markdown groundtruth (escaped_characters.md) already expects literal &amp; inside code spans and code blocks. Only 2.104.0 produces that; older cores decode code text a second time in the round trip. The lock already resolves 2.104.0, so only the specifiers change.

Testing

  • New test_html_block_entities_are_decoded_exactly_once: the entities in a raw HTML block reach the output decoded exactly once.
  • On docling-core 2.104.0, the Markdown and HTML backend tests pass: 181 passed, 1 skipped.
  • On docling-core 2.103.1, the new test passes. The only failure is the escaped_characters golden, which needs 2.104.0; that is why the floor is raised.

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

🤖 Generated with Claude Code

cau-git and others added 2 commits October 26, 2026 09:41
docling-core 2.104.0 escapes CodeItem text in the HTML serializer. Older
versions let the Markdown backend's HTML round trip decode code text a
second time, so `&amp;` inside code became `&`. The Markdown groundtruth
already expects the literal code text, which only 2.104.0 produces, and
the lock already resolves 2.104.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
The Markdown backend puts raw HTML blocks into CodeItem text, serializes
the document to HTML, and strips the <pre><code> wrapper to splice the
HTML back. Since docling-core 2.104.0 escapes code text, convert()
unescapes each block once. That assumes the serializer escaped it exactly
once: with docling-core 2.103.1, which the floor allowed, entities in the
block were decoded twice (`&amp;` became `&`).

Store the raw blocks on the backend and put only an index between the
markers. convert() replaces each marked <pre><code> block with the stored
HTML, so the result no longer depends on how the serializer treats code
text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @cau-git, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@github-actions github-actions Bot added ai:ci-needs-care AI triage: check CI-safety findings before you approve CI ai:possible-duplicate AI triage: another PR may fix the same problem bug Something isn't working markdown issue related to markdown backend labels Oct 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

AI triage (advisory)

Checked commit a561ec188c. The analysis read the diff only. It did not run code from this pull request.

This PR stores raw HTML blocks in a list and inserts numeric index placeholders into code items during Markdown conversion. It restores the stored HTML blocks by index to prevent incorrect entity decoding.

Topic labels added: bug, markdown.

CI safety: needs maintainer care before you approve CI.

  • pyproject.toml: changes build, dependency, or tool configuration
  • uv.lock: changes locked dependencies

Possible duplicates

PR State Signal Assessment
#4985 open changes the same code duplicate: PR #4985 implements the same indexed list mechanism to restore raw HTML blocks in md_backend.py.
1 related pull request(s) in the same code
PR State Signal Assessment
#4771 merged referenced as #4771 related: PR #4771 changed the same HTML block restoration code in md_backend.py with an unescape approach.

The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.

@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
docling/backend/md_backend.py 85.71% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@cau-git cau-git closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:ci-needs-care AI triage: check CI-safety findings before you approve CI ai:possible-duplicate AI triage: another PR may fix the same problem bug Something isn't working markdown issue related to markdown backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant