Repository navigation
Extend sanitize_text with known exceptions #1127
Description
With the new reading order module we are currently missing some text normalization which were applied by the glm component. For example in https://github.com/DS4SD/deepsearch-glm/blob/main/src/andromeda/tooling/structs/normalizers/text_normalizer.h#L52.
Currently some text normalization is done in https://github.com/DS4SD/docling/blob/main/docling/models/page_assemble_model.py#L55, however
- it is not complete (see complete glm tooling linked above)
- it does not preserve the original text in the orig field
Some extra notes:
- while porting the normalization we should look at issues like this Disabling char normalization does not work #928.
- the text normalization should be in the
textfield, leaving theorigcontent with the original text.
Activity
cau-git commentedon Sep 1, 2026
Hi @dolfim-ibm, this issue hasn't had any activity since it was opened. Do you still consider it relevant? If we don't hear back within a week, we'll close it. Feel free to reopen if you see a need.
morten-lagabote commentedon Oct 4, 2026
This is still relevant for us, for the second point (
origdoes not preserve the original text). We convert legal texts and compare the result with the source. #4434 made the HTML backend keep the source text inorig; in the PDF pipelineorigis still the sanitized text.On
main(58f1f8d),TextElementonly hastext(sanitize_text(textlines)), and the reading-order model passes it on withoutorig, so docling-core setsorig = text._merge_elementshas the TODOs "we don't have theorigfield of the merged element". In a Supreme Court judgment (HR-2013-2663-A), the only items withorig != textare the 76 list items whose marker is stripped fromtext, and the hyphen that is dropped at a line break (#4151) is dropped inorigtoo:§ 14-12becomes§ 1412in both.If you agree with the direction, I can prepare a PR:
TextElementgets anorig: the lines of the cluster joined with a space, before dehyphenation and character normalization.textstays as it is;origon, also in_merge_elements;ListItemMarkerProcessorkeeps matching the marker onorig, but strips it fromtext(todaytextis derived fromorig).One open question:
prov.charspanis computed from the length oftext, while the DocLang serializer slicesorigwith it (item.orig[prov_.charspan[0] : prov_.charspan[1]]). Should the charspans indexorig? #4217 / #4222 touch the same code, so I would wait for #4222 either way.morten-lagabote commentedon Oct 12, 2026
An update on my comment above. For PDF, some characters are already replaced in docling-parse, before Docling gets the text. Its cell sanitizer folds the quote and dash characters to ASCII, and the
origof each cell is a copy of the folded text (see the triage in docling-project/docling-parse#362).Two examples with docling-parse 7.22.2:
”(U+201D) in its text layer. The char, word and line cells have 6 ×", intextand inorig.(2013–2014)in its text layer. The line cell has(2013-2014).So the change in my comment above can keep the text before dehyphenation in
orig, but not these characters. They need a change in docling-parse first; docling-project/docling-parse#388 stops the fold for the dashes.