fix(latex): convert accent and letter macros in body text by pandasuwu · Pull Request #4517 · docling-project/docling

pandasuwu · 2026-10-03T02:19:02Z

Accent macros in running text lost their accent, \~n lost its letter, and letter macros such as \o, \ss and \L produced nothing (#4514). The inline macro path (_process_macro_node_inline) had no accent handling: \'e fell through to the generic branch, which kept only the argument, and \~n matched MACROS_INLINE_VERBATIM and became a space. The LatexNodes2Text conversion in _process_macro was never reached, because that method is only called for MACROS_STRUCTURAL macros.

Input:

caf\'e, Espa\~na, J{\"u}rgen, {\c{C}}aglar, {\o}, \ss{}, \aa.

Before:

cafe, Espa a, Jurgen, Caglar, , , .

After:

café, España, Jürgen, Çaglar, ø, ß, å.

The fix adds _char_macro_to_text to TextHelperMixin. It converts an accent macro that carries an argument, or one of the standard text-mode letter macros (new MACROS_LETTERS in constants.py), with pylatexenc's latex2text. Both the inline path and _macro_node_to_text (headings, captions, table cells, macro arguments) use it. It leaves alone:

names the document redefines with \newcommand/\renewcommand,
accents parsed without an argument, which keep the current handling,
\~{} and \^{}, which keep their literal ~ and ^ (e.g. http://x.org/\~{}user); latex2text renders these as an empty string.

The now-unreachable MACROS_ACCENTS branch in _process_macro is removed.

Tests. test_latex_accent_macro said accents "are converted to Unicode characters" but only asserted "caf" in md. It now checks the exact text for accents, \~n, letter macros, \~{} and a heading, and fails on main.

Ground truth. Regenerated with DOCLING_GEN_TEST_DATA=1. Only text changes, all corrections:

1706.03762: Caglar Gulcehre → Çaglar Gülçehre, Jurgen → Jürgen, ukasz Kaiser → Łukasz Kaiser (\L was dropped)
arXiv-2501.01300v2: Universitat Munchen → Universität München, Strae → Straße, Fakultat fur → Fakultät für

pytest tests/test_latex: 123 passed. prek run --all-files: passed.

Issue resolved by this Pull Request:
Resolves #4514

Checklist:

Documentation has been updated, if necessary.
Examples have been added, if necessary.
Tests have been added, if necessary.

Accent macros in running text lost their accent (caf\'e -> cafe), \~n lost its letter (Espa\~na -> Espa a) and letter macros such as \o, \ss and \L produced nothing. The inline macro path had no accent handling, and the latex2text conversion in _process_macro was never reached. Convert accents that carry an argument and the standard letter macros with pylatexenc's latex2text in both the inline path and _macro_node_to_text, unless the document redefines the name. \~{} and \^{} keep their literal ~ and ^. Resolves docling-project#4514 Signed-off-by: pandasuwu <suhani.m024@gmail.com>

github-actions · 2026-10-03T02:19:11Z

✅ DCO Check Passed

Thanks @pandasuwu, all your commits are properly signed off. 🎉

mergify · 2026-10-03T02:19:39Z

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

When test data is updated, we require two reviewers

#approved-reviews-by >= 2

codecov · 2026-10-03T03:47:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

geoHeil

Please fix the loss of custom macro text inside accent arguments. I confirmed the case below on d184531. All 123 LaTeX tests pass, but they do not cover this case.

latex2text does not know the document's own macros, so an accent on one dropped the letter: with \newcommand{\vowel}{e}, caf\'{\vowel} became "caf". When the accent's argument uses a document macro, expand it with the backend first and then apply the accent, giving "café". Signed-off-by: pandasuwu <suhani.m024@gmail.com>

geoHeil

The direct custom-macro case now passes. One chained macro still loses text; see below. All 124 LaTeX tests pass on 4058344.

The argument of an accent was expanded only one level, so a chain such as \newcommand{\letter}{e} \newcommand{\vowel}{\letter} left \letter, which latex2text then dropped: caf\'{\vowel} became "caf". Expand the argument level by level until no document macros remain, then apply the accent. An argument that never resolves is kept without the accent. Signed-off-by: pandasuwu <suhani.m024@gmail.com>

A macro defined in terms of itself must stop expanding and keep its text, as on main, instead of looping or dropping it. Signed-off-by: pandasuwu <suhani.m024@gmail.com>

An accent with no parsed argument (pylatexenc gives \t none) converts the same without the early return, and tolerant parsing never raises LatexWalkerParseError, so neither branch was reachable or needed. Signed-off-by: pandasuwu <suhani.m024@gmail.com>

geoHeil

LGTM. Both earlier text-loss cases are fixed on d7ba852. All 124 LaTeX tests pass. I also checked macro chains, macros with arguments, redefined letters, and conversion in headings, captions, and table cells. No remaining blocking findings.

pandasuwu · 2026-10-16T06:04:37Z

Hi @PeterStaar-IBM @ceberam, a gentle ping on this one. @geoHeil has approved it after two rounds of review, and the remaining Mergify protection needs a second approval because the PR updates tests (the ground-truth changes are text corrections only, e.g. Jurgen → Jürgen, Strae → Straße). Would one of you have time to take a look? Happy to make any changes. Thanks!

Signed-off-by: pandasuwu <suhani.m024@gmail.com>

github-actions · 2026-10-16T06:15:41Z

AI triage (advisory)

Checked commit e939160ba8. The analysis read the diff only. It did not run code from this pull request.

The PR fixes accent and letter macros in the LaTeX backend body text, so that macros such as \'e, \~n, \o, and \ss now produce the correct Unicode characters instead of dropping the accent or producing nothing.

Topic labels added: bug, language support.

CI safety: no concern found. A maintainer must still approve CI.

2 related pull request(s) in the same code

PR	State	Signal	Assessment
#4817	open	changes the same code	related: PR #4817 also edits `constants.py`, `macros.py`, and `test_macros.py` in the LaTeX backend, but it fixes equation environment preprocessing, `\hfuzz`/`\vfuzz` removal, and bibliography parsing — not accent or letter macros.
#4544	open	changes the same code	related: PR #4544 edits `utils/text.py` only to fix `_extract_verbatim_content` for the `minted` environment; it does not touch accent or letter macro handling.

Reference data changes

Change	Files
text	5

text tests/data/latex/groundtruth/1706.03762_main.tex.itxt: +3 / -3 lines
text tests/data/latex/groundtruth/1706.03762_main.tex.json: 6 text item(s) changed
text tests/data/latex/groundtruth/1706.03762_main.tex.md: +6 / -6 lines
text tests/data/latex/groundtruth/arXiv-2501.01300v2_main.tex.json: 2 text item(s) changed
text tests/data/latex/groundtruth/arXiv-2501.01300v2_main.tex.md: +4 / -4 lines

Assessment: expected. The PR description lists the exact author names that are corrected (e.g. Jurgen → Jürgen, ukasz → Łukasz), and all five changed reference files show only text corrections that match the accent and letter macro fix.

_{The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.}

github-actions

AI first review (advisory): no blocker found

This review covers the full diff at e939160ba8. It read the code and did not run it.

The PR correctly fixes accent and letter macros in body text, headings, and table cells. Two rounds of reviewer feedback were resolved: custom-macro arguments in accents are now expanded iteratively, with a cycle guard. The removed MACROS_ACCENTS branch in _process_macro was unreachable, so its deletion is safe. Check that bare \~ (no argument, previously mapped to a space) now silently produces ~ instead — the test covers \~{} and \~n but not the bare form; confirm that this behavior change is acceptable.

_{A maintainer makes the merge decision. This review never approves the pull request.}

PeterStaar-IBM

lgtm!

geoHeil requested changes Oct 3, 2026

View reviewed changes

Comment thread docling/backend/latex/utils/text.py Outdated

geoHeil requested changes Oct 3, 2026

View reviewed changes

Comment thread docling/backend/latex/utils/text.py Outdated

pandasuwu added 3 commits October 3, 2026 23:50

test(latex): cover an accent on a macro that never resolves

f420b07

A macro defined in terms of itself must stop expanding and keep its text, as on main, instead of looping or dropping it. Signed-off-by: pandasuwu <suhani.m024@gmail.com>

geoHeil previously approved these changes Oct 3, 2026

View reviewed changes

Merge branch 'main' into fix/latex-accents

e939160

Signed-off-by: pandasuwu <suhani.m024@gmail.com>

pandasuwu dismissed geoHeil’s stale review via e939160 October 16, 2026 06:13

github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) bug Something isn't working language support labels Oct 16, 2026

github-actions Bot reviewed Oct 16, 2026

View reviewed changes

github-actions Bot added the ai:review-lgtm AI first review found no blocker or major issue (advisory) label Oct 16, 2026

This was referenced Oct 16, 2026

fix(latex): backend updates for better parsing of arxiv papers #4817

Merged

fix(latex): number the items of enumerate lists #4518

Open

PeterStaar-IBM approved these changes Oct 19, 2026

View reviewed changes

cau-git approved these changes Oct 19, 2026

View reviewed changes

PeterStaar-IBM merged commit b845ce6 into docling-project:main Oct 19, 2026
33 checks passed

Navigation Menu

Repository navigation

fix(latex): convert accent and letter macros in body text - #4517

fix(latex): convert accent and letter macros in body text#4517
PeterStaar-IBM merged 6 commits into
docling-project:mainfrom
pandasuwu:fix/latex-accents

pandasuwu commented Oct 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 3, 2026

Uh oh!

mergify Bot commented Oct 3, 2026 •

edited

Loading

🟢 Enforce conventional commit

🟢 Require two reviewer for test updates

Uh oh!

codecov Bot commented Oct 3, 2026 •

edited

Loading

Uh oh!

geoHeil left a comment

Uh oh!

Uh oh!

geoHeil left a comment

Uh oh!

Uh oh!

geoHeil left a comment

Uh oh!

pandasuwu commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

PeterStaar-IBM left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pandasuwu commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 3, 2026

Uh oh!

mergify Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🟢 Enforce conventional commit

🟢 Require two reviewer for test updates

Uh oh!

codecov Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

geoHeil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

geoHeil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

geoHeil left a comment

Choose a reason for hiding this comment

Uh oh!

pandasuwu commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026

AI triage (advisory)

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

AI first review (advisory): no blocker found

Uh oh!

PeterStaar-IBM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pandasuwu commented Oct 3, 2026 •

edited

Loading

mergify Bot commented Oct 3, 2026 •

edited

Loading

codecov Bot commented Oct 3, 2026 •

edited

Loading