Repository navigation
Conversation
… attributes in HTML `BaseMeta.language` is metadata about the text, but the Markdown meta serializer had no branch for `LanguageMetaField` and fell through to `str(field_val)`, so a document with language metadata gained paragraphs such as `confidence=0.99 created_by='langdetect' code=<HumanLanguageLabel.DE: 'de'>` in Markdown and plain text. `EntitiesMetaField` leaked its repr the same way, and HTML had no way to express the language at all: `HTMLParams.html_lang` was never emitted and elements carried no `lang` attribute. Markdown / plain text: `LanguageMetaField` and `EntitiesMetaField` are skipped unless `mark_meta=True`, where they render as `[Language] de` and `[Entities] IBM, Zurich`. Any other structured field without a renderer is skipped instead of printing its pydantic repr; custom scalar fields keep rendering as before. HTML: the root element is `<html lang="...">`. `html_lang` now defaults to None, which resolves to the document language (`doc.body.meta.language`), else the character-weighted majority of the text items' `meta.language`, else "en". Paragraphs, headings and list items whose `meta.language` differs from the page language (compared by primary subtag) get a `lang` attribute (WCAG 3.1.1 and 3.1.2). The language row in the meta details block is unchanged. Ground-truth HTML files are regenerated for the root element only. Closes docling-project#745 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
Contributor
|
✅ DCO Check Passed Thanks @wittjeff, all your commits are properly signed off. 🎉 |
Contributor
Merge Protections🔴 1 of 2 protections blocking · waiting on 👀 reviews
🔴 Require two reviewer for test updatesWaiting for
This rule is failing.When test data is updated, we require two reviewers
Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #745. Related: #868 (full BCP 47 tag on
LanguageMetaField), docling-project/docling#4741 and docling-project/docling#4781 (PDF/Langattached tobody.meta, opt-in until this lands).Problem
BaseMeta.languageis metadata about the text, but once set it is treated as content:MarkdownMetaSerializer._serialize_meta_fieldhas noLanguageMetaFieldbranch and falls through tostr(field_val)), so the document gains a paragraph such asconfidence=0.99 created_by='langdetect' code=<HumanLanguageLabel.DE: 'de'>.EntitiesMetaFieldleaks the same way.langattribute on the elements that carry the text.HTMLParams.html_langis declared but never emitted; the root is the hard-coded<html>.Change
Markdown / plain text
LanguageMetaFieldandEntitiesMetaFieldare skipped unlessmark_meta=True, where they render as[Language] deand[Entities] IBM, Zurich.BaseModel) without a renderer is skipped instead of printing its repr. Custom scalar fields keep rendering as before.HTML
<html lang="...">.html_lang(params andexport_to_html) now defaults toNone, resolved byresolve_document_language():doc.body.meta.language, else the character-weighted majority of the text items'meta.language, else"en". The resolution happens once inHTMLDocSerializer.model_post_init, so every element serializer compares against the same page language.<p>,<h1>..<h6>and<li>getlang="<code>"when the item'smeta.languagediffers from the page language, compared by primary subtag (en-USvsenis the same language). This is WCAG 3.1.1 (page) and 3.1.2 (parts).languagerow in the<details class="docling-meta">block is unchanged: existing tests pin it, and it does no harm.Ground truth
All
.gt.html/.html.gtfiles are regenerated; the only change in each is<html>-><html lang="en">. (The regeneration also rewrote the visualization PNGs through local font drift; those are reverted and not part of this PR.)Tests
tests/test_metadata.py: Markdown and plain text exclude language and entities by default and list them withmark_meta=True; a structured custom field without a renderer is skipped while a scalar one is kept; HTMLlangattributes with the majority default, an explicithtml_lang="en-US", and a body-level language;<html lang="en">without any language metadata. Full suite and pre-commit (ruff, mypy, pytest, docs) pass.🤖 Generated with Claude Code