Repository navigation
Dehyphenation merges line-final hyphens that belong to the word, losing them irrecoverably (regression in 2.118.0) #4151
Description
Bug
When a hyphenated word is broken across a line, Docling joins the halves and deletes the hyphen without checking whether the hyphen is part of the word. n-tridecanal becomes ntridecanal.
Mid-line hyphens are handled correctly, so the logic is already position-aware. The gap is that a line-final hyphen can also be a real hyphen.
The loss is unrecoverable: once the hyphen is gone, nothing downstream can tell whether the page printed ntridecanal or n-tridecanal. The opposite failure — leaving n- tridecanal unjoined, which 2.107.0 does — is wrong but repairable by anyone later.
Steps to reproduce
Open-access paper, CC-BY: 10.1371/journal.pone.0260149
from docling.document_converter import DocumentConverter
doc = DocumentConverter().convert("pone.0260149.pdf").document
print(doc.export_to_markdown())The same word, three times in one document
| where | how the PDF prints it | docling 2.124.0 |
|---|---|---|
p1, line ends …undecanal or n-, next line begins tridecanal |
broken at the line end | undecanal or ntridecanal ❌ |
p9, line ends …n-dodecanal (77) and n-, next line begins tridecanal (84) |
broken at the line end | and ntridecanal (84) ❌ |
| p11, mid-line | n-octanal (10) and n-tridecanal (84) could be |
and n-tridecanal (84) ✅ |
Docling's own output already contains the correct form. The third row is preserved because the hyphen is not at a line break. The first two are destroyed because it is.
Line position alone cannot resolve these: the hyphen is line-final and part of the word. But the document says so plainly elsewhere:
- The p1 line reads
…n-octanal, n-nonanal, n-decanal, n-dodecanal, undecanal or n-— four instances of the same construction, unbroken, on the very line that then breaks atn-. - The p9 line reads
…n-undecanal (52), n-dodecanal (77) and n-, likewise. n-tridecanalitself appears unbroken on p11, and Docling extracts it correctly there.
A dehyphenation pass that checked the forms already present in the document before deleting a hyphen would get all three right.
Why this matters beyond the character count
The losses fall on chemical nomenclature, where the hyphen is part of the name: n-decanal, 4,5-dimethylthiazole, 6-methyl-5-hepten-2-one. ndecanal is not a variant spelling but a different string, and it breaks exact search and entity linking against any chemical vocabulary. The same applies to gene and compound identifiers.
For comparison, PyMuPDF preserves the hyphen in all three positions.
Docling version
| version | p1 (line-final) | p9 (line-final) |
|---|---|---|
| 2.107.0 | n- tridecanal |
ntridecanal |
| 2.117.0 | n- tridecanal |
ntridecanal |
| 2.118.0 | ntridecanal |
ntridecanal |
| 2.121.0 / 2.122.0 / 2.124.0 | ntridecanal |
ntridecanal |
The p9 case merges in every version tested. The p1 case changed at 2.118.0, whose release notes carry "Dehyphenate hard continuations".
Tested docling 2.107.0, 2.117.0, 2.118.0, 2.121.0, 2.122.0, 2.124.0. PyPdfiumDocumentBackend. Identical results under torch 2.13.0+cpu and 2.14.0.
Python version
Python 3.12.13, Linux
Activity
fwwwrrr commentedon Sep 3, 2026
Worth adding, because the issue as written can be read as "revert the dehyphenation or
keep it", and that is a false choice. There is a rule that keeps every prose gain and
still fixes these cases.
When a word is broken at a line end, check whether the hyphenated or the joined form
occurs elsewhere in the same document, and prefer the attested one. If neither occurs,
do what you do now.
Measured over the two papers in this report, 179 words broken at a line end:
ana-lyzed→analyzed,gam-biae→gambiae,mos-quito→mosquiton-tridecanal,n-octanal,i-butyric,blood-feedingThe rule changes the answer on 2.2% of line breaks, and the four it changes are
exactly the ones being lost. No prose dehyphenation is given up.
It also handles compounds with a real internal hyphen:
blood-feed-|ingjoins toblood-feeding,non-pre-|ferredtonon-preferred,human-spe-|cifictohuman-specific— the second hyphen closes, the first survives, because that is whatthe document attests.
The lookup needs the document's own token frequencies, which are available by the time
dehyphenation runs. No dictionary and no language model.
wittjeff commentedon Sep 3, 2026
I reproduced this end-to-end on 2.124.0 with the PDF you cite, and traced it to two separate join sites. Some findings that may help scope a fix.
The document loses eight tokens, not two.
ntridecanal×2n-tridecanalnoctanaln-octanalibutyrici-butyricbloodfeedingblood-feeding6-methyl5-hepten-2-one6-methyl-5-hepten-2-one6-methyl-5-hepten2-one6-methyl-5-hepten-2-onePlasmodiumassociatedPlasmodium-associatedOne line of the export carries the destroyed and the intact form side by side:
Two join sites, and the version table maps onto them exactly
I instrumented both and attributed every one of the eight:
PageAssembleModel.sanitize_text, which joins lines within one cluster. This is the older path, which is why the p9 case merges in every version tested.ReadingOrderModel._merge_elements, which joins across clusters. That is the path fix(reading-order): dehyphenate hard continuations #3888 added, which is why p1 alone changed at 2.118.0.So a patch to either site alone fixes at most half of this. The two also disagree with each other today:
_merge_elementsrequires the continuation to be lowercase,sanitize_textrequires onlyisalnum()on both sides.Both still merge on current
main(deb45fb9). Thehyphen_attached_to_wordguard added since 2.124.0 does not change these cases —n-has an alnum character before the hyphen, so it still closes up.A note on how this PDF marks the break
Worth recording because it shapes what a fix can rely on: in this document the line-break hyphen is U+0002, not
-. It accounts for 103 of the 104 line-final hyphens and never appears mid-line, andpage_assemble_model.py#L201rewrites it to-before any of the above runs.It is not a usable signal on its own — the typesetter emits the same
\x02forn-|tridecanaland forana-|lyzed, so it only restates the line position we already know. But it does yield a clean 103-case corpus for measuring candidate rules, which is what the numbers below come from.Your proposed rule, measured on those 103 breaks
i-butyric,n-tridecanal,n-octanal,blood-feeding— all correctNo false flips, and no prose dehyphenation given up — matching your measurement. It recovers 5 of the 8 losses.
Two document-independent guards close most of the rest
Measured on the same 103 breaks, each also with no false positives:
6-methylcases. No syllable break puts a digit after the hyphen.i-,n-×2, plus one spurious cross-column join it correctly declines).\lefthyphenminis ≥2 in every language TeX ships; no typesetter leaves one letter alone at a line end.Together with the attested-form rule this recovers 7 of the 8. Only
Plasmodium-associatedresists — unattested, and structurally indistinguishable fromWagen-|ingen.One structural catch for the attested-form rule
sanitize_textruns inside the threaded per-page streaming stage, whilereading_order_modelruns later, over the whole document. The site responsible for 7 of the 8 losses therefore has no document-wide vocabulary available, and by the time one exists the line breaks have already been erased.Implementing the lookup means deferring the join decision out of page assembly — leaving a marker for reading-order to resolve, or re-deriving line breaks from
TextElement.cluster.cells, which does survive into that stage. The two guards above carry no such constraint and could land independently.Happy to open a PR for the two document-independent guards with regression tests over this PDF's breaks, if that split is useful to the maintainers.
morten-lagabote commentedon Oct 4, 2026
Data points from Norwegian legal and government PDFs, where a line-final hyphen often belongs to a provision number or a compound. All on
main(51fe9ff), default options:mainarbeidsmiljøloven § 1412 eller arbeidsmiljøloven § 14-13§ 14-12§ 115,§ 118,§ 217 bokstav c,og 1110§ 11-5,§ 11-8,§ 21-7,11-10EØSrettslige,helseog sosialtjenesterEØS-rettslige,helse- og sosialtjenesterEØSterskelverdien,byggeog anleggskontraktEØS-terskelverdien,bygge- og anleggskontraktThe first two rows are fixed by the digit guard of both #4211 (348219e) and #4387 (7c768b7). For a provision number, the hyphen is part of the reference. In three of the documents the intact form is also attested elsewhere (
§ 14-12,helse- og,bygge- og), as in the original report.Two classes remain with both PRs:
EØS-rettslige): an abbreviation compound, not a syllable break. Like the digit guard, it needs no vocabulary;helse- og), which needs the hyphen and the space. The same happens in English intests/data/pdf/groundtruth/2203.01017v2.md, line 13:complex columnand row-headers(column- and row-headers).I can open a follow-up for the first one once #4211 or #4387 is merged. The second needs the conjunctions of each language, so it may be out of scope.