Skip to content

fix(serializer): keep meta.language out of text exports and emit lang attributes in HTML - #869

Open
wittjeff wants to merge 1 commit into
docling-project:mainfrom
wittjeff:fix/meta-language-serialization
Open

wittjeff wants to merge 1 commit into
docling-project:mainfrom
wittjeff:fix/meta-language-serialization

Conversation

@wittjeff

Copy link
Copy Markdown
Contributor

Closes #745. Related: #868 (full BCP 47 tag on LanguageMetaField), docling-project/docling#4741 and docling-project/docling#4781 (PDF /Lang attached to body.meta, opt-in until this lands).

Problem

BaseMeta.language is metadata about the text, but once set it is treated as content:

  1. Markdown and plain text print the field's pydantic repr (MarkdownMetaSerializer._serialize_meta_field has no LanguageMetaField branch and falls through to str(field_val)), so the document gains a paragraph such as confidence=0.99 created_by='langdetect' code=<HumanLanguageLabel.DE: 'de'>. EntitiesMetaField leaks the same way.
  2. HTML has no lang attribute on the elements that carry the text.
  3. HTMLParams.html_lang is declared but never emitted; the root is the hard-coded <html>.

Change

Markdown / plain text

  • LanguageMetaField and EntitiesMetaField are skipped unless mark_meta=True, where they render as [Language] de and [Entities] IBM, Zurich.
  • Any other structured field (a BaseModel) without a renderer is skipped instead of printing its repr. Custom scalar fields keep rendering as before.

HTML

  • Root element is <html lang="...">. html_lang (params and export_to_html) now defaults to None, resolved by resolve_document_language(): doc.body.meta.language, else the character-weighted majority of the text items' meta.language, else "en". The resolution happens once in HTMLDocSerializer.model_post_init, so every element serializer compares against the same page language.
  • <p>, <h1>..<h6> and <li> get lang="<code>" when the item's meta.language differs from the page language, compared by primary subtag (en-US vs en is the same language). This is WCAG 3.1.1 (page) and 3.1.2 (parts).
  • The language row in the <details class="docling-meta"> block is unchanged: existing tests pin it, and it does no harm.

Ground truth

All .gt.html / .html.gt files are regenerated; the only change in each is <html> -> <html lang="en">. (The regeneration also rewrote the visualization PNGs through local font drift; those are reverted and not part of this PR.)

Tests

tests/test_metadata.py: Markdown and plain text exclude language and entities by default and list them with mark_meta=True; a structured custom field without a renderer is skipped while a scalar one is kept; HTML lang attributes with the majority default, an explicit html_lang="en-US", and a body-level language; <html lang="en"> without any language metadata. Full suite and pre-commit (ruff, mypy, pytest, docs) pass.

🤖 Generated with Claude Code

… attributes in HTML

`BaseMeta.language` is metadata about the text, but the Markdown meta
serializer had no branch for `LanguageMetaField` and fell through to
`str(field_val)`, so a document with language metadata gained paragraphs
such as `confidence=0.99 created_by='langdetect' code=<HumanLanguageLabel.DE: 'de'>`
in Markdown and plain text. `EntitiesMetaField` leaked its repr the same
way, and HTML had no way to express the language at all: `HTMLParams.html_lang`
was never emitted and elements carried no `lang` attribute.

Markdown / plain text: `LanguageMetaField` and `EntitiesMetaField` are
skipped unless `mark_meta=True`, where they render as `[Language] de` and
`[Entities] IBM, Zurich`. Any other structured field without a renderer is
skipped instead of printing its pydantic repr; custom scalar fields keep
rendering as before.

HTML: the root element is `<html lang="...">`. `html_lang` now defaults to
None, which resolves to the document language (`doc.body.meta.language`),
else the character-weighted majority of the text items' `meta.language`,
else "en". Paragraphs, headings and list items whose `meta.language`
differs from the page language (compared by primary subtag) get a `lang`
attribute (WCAG 3.1.1 and 3.1.2). The language row in the meta details
block is unchanged.

Ground-truth HTML files are regenerated for the root element only.

Closes docling-project#745

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit —

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant