Skip to content

fix(webvtt): keep emphasis markers clear of span edge whitespace - #4577

Closed
nanhanq1 wants to merge 1 commit into
docling-project:mainfrom
nanhanq1:fix/webvtt-span-edge-whitespace-4352
Closed

nanhanq1 wants to merge 1 commit into
docling-project:mainfrom
nanhanq1:fix/webvtt-span-edge-whitespace-4352

Conversation

@nanhanq1

@nanhanq1 nanhanq1 commented Oct 5, 2026

Copy link
Copy Markdown

Issue resolved by this Pull Request:
Resolves #4352

What changed

WebVTTDocumentBackend now drops the whitespace at the edges of the cue spans that
carry formatting (_detach_span_edge_whitespace()), and no longer emits a span that
holds nothing but whitespace.

Why

Emphasis markers wrap the item text verbatim, so a span ending in a space was emitted
as *steaming bowl of *. A * preceded by whitespace is not a right-flanking
delimiter in CommonMark, so the markup was written out but never rendered.

Before, for the cue
The waiter offered a <i>steaming bowl of <lang es-ES>paella</lang></i> that arrived.:

The waiter offered a  *steaming bowl of * *paella*  that arrived.
<p>The waiter offered a  *steaming bowl of * <em>paella</em>  that arrived.</p>

After:

The waiter offered a  *steaming bowl of* *paella*  that arrived.
<p>The waiter offered a  <em>steaming bowl of</em> <em>paella</em>  that arrived.</p>

Nesting is what makes the space land inside the markers. In
<i><b>unexpected</b> <u><lang it>arcobaleno</lang></u> of flavors</i> the space
between </b> and <u> is itself an italic span, so it contributed a bare * *
pair, and the following span opened with a space (* of flavors*, whose * is not
left-flanking). Both are gone now:

before: The dessert's  ***unexpected*** * * *arcobaleno* * of flavors*  left everyone in awe.
after:  The dessert's  ***unexpected*** *arcobaleno* *of flavors*  left everyone in awe.

The inline-group serializer already inserts a separator between the items it joins,
so dropping the edge whitespace does not lose any word separation.

Scope

  • The residual doubled spaces around an inline group are not addressed here. They
    come from the serializer adding a separator on top of the item's own whitespace,
    which is tracked upstream as docling-core#371 (open since 2025-08). That is a
    docling-core change; the existing "temporary ground truth" comments in
    test_backend_vtt.py therefore still stand.
  • File-independent from feat: honor the InlineGroup whitespace contract in the backends #3891. That PR reworks the InlineGroup whitespace contract
    for the html, md, msword, opendocument and jats backends. It does not touch
    webvtt_backend.py or the webvtt ground truth.

Tests

Two regression tests in tests/test_backend_vtt.py, each verified to fail on main
and pass with this change:

  • test_emphasis_markers_stay_clear_of_span_edge_whitespace
  • test_whitespace_only_span_emits_no_bare_markers

Ground truth for webvtt_example_04 regenerated with DOCLING_GEN_TEST_DATA=1.
Examples 01–03 are byte-identical.

Local run of tests/test_backend_vtt.py: 14 passed. The three tmp_path cases
(test_utf8_bom_does_not_invalidate_the_signature,
test_crlf_and_cr_line_terminators[CRLF|CR]) hang in this Windows sandbox both with
and without the change
— verified by stashing — so they are not covered by the local
run; they touch none of the code paths changed here.

Checks: ruff check and ruff format --check clean; ty reports the same single
pre-existing invalid-argument-type warning as on main; tach check passes.

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

A bold/italic/underline span that ends or begins with a space had that
space emitted inside its emphasis markers, e.g. "*steaming bowl of *".
CommonMark does not treat a space-adjacent "*" as a flanking delimiter,
so the markup was written out but never rendered.

Drop the whitespace at the edges of the spans that carry formatting. The
inline-group serializer already inserts a separator between the items it
joins, so no word separation is lost. A span holding nothing but
whitespace no longer contributes a bare, unrenderable pair of markers.

The remaining doubled spaces around an inline group are tracked upstream
as docling-core#371 and are out of scope here.

Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @nanhanq1, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit —

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@PeterStaar-IBM PeterStaar-IBM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@ceberam ceberam added the xml issue related to supported schema-specific XML formats label Oct 5, 2026

@ceberam ceberam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nanhanq1 for your willingness to improve Docling and for opening this PR.

I have seen some minor issues like an assertion on _create_vtt_stream helper (we should not use assertions in runtime code) and redundant inline comments (the necessary info is already well explained through the docstring).

However, I see a major issue. Despite your message on the description, this PR still tries to solve an issue that should be handled by docling-core. I know that the PR docling-project/docling-core#371 has been opened since long time and the current Markdown serializer may generate invalid text (e.g. **Hello **World instead of **Hello** World), but trying to fix it on the parser is the wrong strategy, since it may propagate the issue to the DoclingDocument. We should still consider docling-project/docling-core#693 and #3891 for this purpose.

For instance, consider the following example:

WEBVTT

04:03:00.000 --> 04:06:00.000
<b>Hello </b>world, <b>Hello</b>world.

Both sentences are valid HTML code, in a valid WebVTT file.
The current version on main will generate 2 TextItem objects for each sentence with the text:

  • Hello , world
  • Hello, world

These items keep the necessary information about spacing. The problem is that docling-core does not apply the proper Markdown serialization to ensure that the emphasis wraps the words, instead of trailing spaces. The problem is not the trailing space, but how the trailing space is processed by the Markdown serialization.

Your PR will create the following TextItem objects in DoclingDocument:

  • Hello, world
  • Hello, world

DoclingDocument stores now exactly the same content for the 2 sentences and it will be unable to distinguish one from the other. The Markdown serializer will output the same (wrong) content and also other serializers (like HTML), since the spacing information is lost.

I would suggest closing this PR. We will fix these issues ASAP through the 2 PRs mentioned above.

@codecov

codecov Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@nanhanq1

nanhanq1 commented Oct 6, 2026

Copy link
Copy Markdown
Author

Thanks @ceberam — I agree with the objection and I'm closing this.

You're right that the edge whitespace is information, not noise. Stripping it in the backend makes <b>Hello </b>world and <b>Hello</b>world collapse to the same TextItem sequence, so DoclingDocument can no longer represent the difference and every downstream serializer loses it — not just Markdown. My change made the md output correct by making the document model less faithful, which is the wrong trade. This belongs in the serializer (docling-project/docling-core#371 / docling-project/docling-core#693), not the parser.

I'll follow docling-project/docling-core#693 and #3891 instead.

@nanhanq1 nanhanq1 closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

xml issue related to supported schema-specific XML formats

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WebVTT inline formatting emits Markdown whose emphasis does not render

3 participants