Repository navigation
Conversation
A bold/italic/underline span that ends or begins with a space had that space emitted inside its emphasis markers, e.g. "*steaming bowl of *". CommonMark does not treat a space-adjacent "*" as a flanking delimiter, so the markup was written out but never rendered. Drop the whitespace at the edges of the spans that carry formatting. The inline-group serializer already inserts a separator between the items it joins, so no word separation is lost. A span holding nothing but whitespace no longer contributes a bare, unrenderable pair of markers. The remaining doubled spaces around an inline group are tracked upstream as docling-core#371 and are out of scope here. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>
|
✅ DCO Check Passed Thanks @nanhanq1, all your commits are properly signed off. 🎉 |
Merge Protections🔴 1 of 2 protections blocking · waiting on 👀 reviews
🔴 Require two reviewer for test updatesWaiting for
This rule is failing.When test data is updated, we require two reviewers
Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
ceberam
left a comment
There was a problem hiding this comment.
Thanks @nanhanq1 for your willingness to improve Docling and for opening this PR.
I have seen some minor issues like an assertion on _create_vtt_stream helper (we should not use assertions in runtime code) and redundant inline comments (the necessary info is already well explained through the docstring).
However, I see a major issue. Despite your message on the description, this PR still tries to solve an issue that should be handled by docling-core. I know that the PR docling-project/docling-core#371 has been opened since long time and the current Markdown serializer may generate invalid text (e.g. **Hello **World instead of **Hello** World), but trying to fix it on the parser is the wrong strategy, since it may propagate the issue to the DoclingDocument. We should still consider docling-project/docling-core#693 and #3891 for this purpose.
For instance, consider the following example:
WEBVTT
04:03:00.000 --> 04:06:00.000
<b>Hello </b>world, <b>Hello</b>world.
Both sentences are valid HTML code, in a valid WebVTT file.
The current version on main will generate 2 TextItem objects for each sentence with the text:
Hello,worldHello,world
These items keep the necessary information about spacing. The problem is that docling-core does not apply the proper Markdown serialization to ensure that the emphasis wraps the words, instead of trailing spaces. The problem is not the trailing space, but how the trailing space is processed by the Markdown serialization.
Your PR will create the following TextItem objects in DoclingDocument:
Hello,worldHello,world
DoclingDocument stores now exactly the same content for the 2 sentences and it will be unable to distinguish one from the other. The Markdown serializer will output the same (wrong) content and also other serializers (like HTML), since the spacing information is lost.
I would suggest closing this PR. We will fix these issues ASAP through the 2 PRs mentioned above.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Thanks @ceberam — I agree with the objection and I'm closing this. You're right that the edge whitespace is information, not noise. Stripping it in the backend makes I'll follow docling-project/docling-core#693 and #3891 instead. |
Issue resolved by this Pull Request:
Resolves #4352
What changed
WebVTTDocumentBackendnow drops the whitespace at the edges of the cue spans thatcarry formatting (
_detach_span_edge_whitespace()), and no longer emits a span thatholds nothing but whitespace.
Why
Emphasis markers wrap the item text verbatim, so a span ending in a space was emitted
as
*steaming bowl of *. A*preceded by whitespace is not a right-flankingdelimiter in CommonMark, so the markup was written out but never rendered.
Before, for the cue
The waiter offered a <i>steaming bowl of <lang es-ES>paella</lang></i> that arrived.:After:
Nesting is what makes the space land inside the markers. In
<i><b>unexpected</b> <u><lang it>arcobaleno</lang></u> of flavors</i>the spacebetween
</b>and<u>is itself an italic span, so it contributed a bare* *pair, and the following span opened with a space (
* of flavors*, whose*is notleft-flanking). Both are gone now:
The inline-group serializer already inserts a separator between the items it joins,
so dropping the edge whitespace does not lose any word separation.
Scope
come from the serializer adding a separator on top of the item's own whitespace,
which is tracked upstream as docling-core#371 (open since 2025-08). That is a
docling-core change; the existing "temporary ground truth" comments in
test_backend_vtt.pytherefore still stand.for the html, md, msword, opendocument and jats backends. It does not touch
webvtt_backend.pyor the webvtt ground truth.Tests
Two regression tests in
tests/test_backend_vtt.py, each verified to fail onmainand pass with this change:
test_emphasis_markers_stay_clear_of_span_edge_whitespacetest_whitespace_only_span_emits_no_bare_markersGround truth for
webvtt_example_04regenerated withDOCLING_GEN_TEST_DATA=1.Examples 01–03 are byte-identical.
Local run of
tests/test_backend_vtt.py: 14 passed. The threetmp_pathcases(
test_utf8_bom_does_not_invalidate_the_signature,test_crlf_and_cr_line_terminators[CRLF|CR]) hang in this Windows sandbox both withand without the change — verified by stashing — so they are not covered by the local
run; they touch none of the code paths changed here.
Checks:
ruff checkandruff format --checkclean;tyreports the same singlepre-existing
invalid-argument-typewarning as onmain;tach checkpasses.Checklist: