Skip to content

fix(webvtt): resolve every HTML character reference in cue text - #926

Open
chrikrah wants to merge 1 commit into
docling-project:mainfrom
chrikrah:fix/webvtt-html-character-references
Open

chrikrah wants to merge 1 commit into
docling-project:mainfrom
chrikrah:fix/webvtt-html-character-references

Conversation

@chrikrah

Copy link
Copy Markdown
Contributor

Fixes #919.

WebVTTFile.parse dropped any cue holding a reference outside the six names of the syntax table, such as ' or é. Cue text and annotations now go through html.unescape, as @ceberam laid out in the issue. An unknown reference or a bare & stays literal text. _escape is untouched.

One addition: 
 is still rejected. It resolves to a line break, and the cue would then be written back across two lines.

$ python -m pytest tests/test_webvtt.py -q      # base b501ec8: 11 passed
12 passed in 0.11s
$ python -m pytest tests -q
1108 passed, 6 skipped, 1 xfailed, 3 warnings in 83.16s      # base b501ec8: 1107 passed

Reverting webvtt.py and keeping the tests fails test_webvtt_cue_html_character_references, and test_vtt_cue_commponents for its two invalid-entity cases, which now assert literal text. ruff 0.16.10 check and format are clean.

$ uv run --no-sync ty check      # ty 0.0.33, base b501ec8: exit 0, 469 diagnostics
Found 469 diagnostics      # exit 0, no errors

Signed off for DCO. A cue holding 
 is still dropped with a warning, as before, and keeping it needs _escape to write line breaks back. I'd leave that until a real file has one. Fine with you?

The cue text and annotation validators accepted only the six named
references of the WebVTT syntax table and dropped the whole cue on
anything else, such as &docling-project#39; or é. The tokenizer (WebVTT 6.4)
hands both states to the HTML character reference algorithm, so read
with html.unescape and keep a reference that does not resolve as
literal text. Serialization still writes only the escapes it needs.

A reference that resolves to a line terminator is still rejected,
because the cue would be written back across two lines.

Refs docling-project#919

Signed-off-by: Christopher Krah <48338417+chrikrah@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @chrikrah, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit —

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WebVTT cue with a numeric or HTML named character reference is dropped

1 participant