[wittjeff]

Summary

The cell text sanitizer folded the whole dash block U+2010..U+2015 and the minus sign U+2212 to - before any text left the parser. The fold runs outside the do_sanitization flag and the exported orig field is a copy of the folded text, so nothing downstream could recover the character. That changes meaning: pages 3–5 and 2024–2026 (ranges) came out as 3-5 and 2024-2026, Product — Nutshell lost its em dash, and a minus sign became a hyphen.

Fixes case 1 of #362 (cases 2 and 3 are separate code paths, see the triage comment there).

Cause

text_constants::replacements in src/parse/pdf_sanitators/constants.h, same table as #335 / #349.

Change

Remove the U+2012, U+2013, U+2014, U+2015 and U+2212 entries. U+2010 (hyphen) and U+2011 (non-breaking hyphen) keep folding to -: docling's PageAssembleModel.sanitize_text joins hyphenated line breaks by looking for an ASCII hyphen alone, and docling folds quotes itself but not dashes, so the dashes now reach the DoclingDocument unchanged. Six lines plus a comment.

Test

tests/test_unit_dash_sanitization.py, self-contained via tests/pdf_builder.py:

• core-14 Helvetica with /WinAnsiEncoding and no embedded program, with 0x96 (endash) and 0x97 (emdash), so the glyph decode is not in question: the issue's own Nutshell — pages 3–5 case;
• an ASCII-hyphen control and an orig == text check;
• a non-embedded Identity-H font whose /ToUnicode covers U+2010..U+2015 and U+2212, checking that the two hyphen variants still fold and the five others are kept.

Against main the three dash tests fail with ['Nutshell - pages 3-5']; with the change all five pass, and tests/test_unit_quote_sanitization.py is unaffected.

Regression set

Line-cell text of all 940 pages in the regression set (447 documents, pinned HF_DATASET_REVISION), this branch against main built from the same tree:

	pages
unchanged	709
differ only in dash characters	228 (113 documents)
differ otherwise	3 (the same three pages of 3763025099401294496-2.pdf that already differ on main)

The 228 pages carry U+2013 (en dash, most common), U+2014, U+2212 and U+2015 where the ground truth has -; nothing else changes. The checked-in parser ground truth therefore needs a regeneration for those pages, as #361 did for its change. I did not touch the dataset.

[github-actions]

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[morten-lagabote]

Tested on a real document, Innst. 240 S (2013–2014) of the Norwegian parliament (5 pages), with a build of 0983d9c on macOS arm64. Result: this PR keeps all 16 en dashes of the document, each in the same word as in the pdfium text layer, and changes nothing else.

	En dashes (U+2013)
pdfium text layer (reference)	16
7.22.2, word cells / line cells	0 / 0
this PR, word cells / line cells	16 / 16

For example, (2013-2014) on page 1 becomes (2013–2014), and the running header Innst. 240 S - 2013-2014 becomes Innst. 240 S – 2013–2014. All other word and line cells are the same as in 7.22.2 (3615 words, 486 lines, same text and boxes). The 189 lines that end with a hyphen still end with -, so dehyphenation in Docling still sees an ASCII hyphen. The document has no em dashes or minus signs, so those are not tested here.