[stanleys12]

Issue resolved by this Pull Request:
Resolves #4151

Relation to #4211

#4211 was opened first and proposes the same two guards for this issue. Pointing out the overlap so reviewers can pick one. Differences visible in its current diff:

• It also changes the soft-hyphen path in _merge_elements and keeps a hyphen after any token that is not purely alphanumeric. This PR only adds the two rules below and leaves the soft-hyphen and detached-hyphen paths alone (fix(reading-order): keep a detached hyphen when merging elements #4180 is open against the latter).
• Its regenerated reference data also touches elsevier-00, redp5110_sampled and table_mislabeled_as_picture, including table structure changes. Here the reference files are main plus the hyphen deltas only.

Description

Since 2.118.0 both sites that close up a word split across a line break drop the hyphen
whenever an alphanumeric character stands on either side of it. A hyphen that is genuinely
part of the word is destroyed with no way to recover it downstream:

output	should be
noctanal, ntridecanal, ibutyric	n-octanal, n-tridecanal, i-butyric
pages 1513715145	pages 15137-15145
xcoordinate	x-coordinate

Two shapes of line-final hyphen cannot have been produced by hyphenation, independent of
the document:

1. A hyphen after a single-character token. No typesetter breaks a word leaving one letter
   behind, so n- / octanal is a chemical prefix, not a split word.
2. A hyphen before a continuation that opens on a digit. 6-methyl- / 5-hepten-2-one is
   a locant, and COVID- / 19 is not a hyphenation point either.

Both are now kept, at both join sites. This deliberately leaves out the attested-form
lookup suggested in the issue comments: that needs the whole document available at the
point of the decision, which neither site has.

Changes

• PageAssembleModel.sanitize_text: two more conditions on the existing join test, one per
  rule above.
• ReadingOrderModel._merge_elements: the same two rules, so the two join sites agree, and
  the kept hyphen is closed up against the lstripped continuation, so 6-methyl- + 5-hepten-2-one
  gives 6-methyl-5-hepten-2-one.
• Reference data: 2203.01017v2, 2206.01062 and 2305.03393v1 regenerated, and a one-line change to main's newspaper-00 reference.

Reference data

Every difference in the reference data restores a hyphen. The only other changes are the
provenance charspans that grow by the one recovered character:

• pages 1162-1167, pages 65-72, pages 15137-15145, pp. 545-561, pp. 1868-1873
• x-coordinate
• newspaper-00: Anna Hug, nft becomes Anna Hug, n-ft (single-character token before the hyphen), charspan 218 -> 219

The data was generated with DOCLING_GEN_TEST_DATA=1 uv run pytest and then reduced to
the hyphen differences, because a full local regeneration also picks up layout-model
differences between this machine and CI (element splitting and bounding boxes in
2203.01017v2 and 2305.03393v1) that have nothing to do with this change. Against main
the committed files are therefore main plus the hyphen deltas only.

Testing

• tests/test_page_assemble_model.py::TestSanitizeTextHyphenation::test_attached_hyphen_belonging_to_word_is_kept
  and the new rows in tests/test_readingorder_hyphenated_merges.py fail on main and
  pass with this change (6 failures before, all green after).
• uv run pytest tests/test_page_assemble_model.py tests/test_readingorder_hyphenated_merges.py: 52 passed.
• Converting tests/data/pdf/sources/newspaper-00.pdf on this branch produces n-ft, matching the updated reference line.
• uv run pytest tests/test_e2e_conversion.py: 2206.01062.pdf passes all three checks
  locally. 2203.01017v2.pdf and 2305.03393v1.pdf fail identically on main and on this
  branch in this environment, from the layout differences noted above, so CI is the place to
  confirm them.
• make check: ruff and ty pass. The max-lines check fails only on files this PR does not touch.

Checklist:

• Documentation has been updated, if necessary. (not needed: behaviour fix, no API change)
• Examples have been added, if necessary. (not needed)
• Tests have been added, if necessary.

🤖 Generated with Claude Code

[github-actions]

✅ DCO Check Passed

Thanks @stanleys12, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

	Protection	Waiting on
🔴	Require two reviewer for test updates	👀 reviews
🟢	Enforce conventional commit	—

🔴 Require two reviewer for test updates

Waiting for

• #approved-reviews-by >= 2

This rule is failing.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

---

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[dolfim-ibm]

/ai review

[github-actions]

AI triage (advisory)

Checked commit 8b680edd27. The analysis read the diff only. It did not run code from this pull request.

Warning

The AI analysis did not finish. Only the deterministic checks are shown.

CI safety: not assessed.

Possibly related pull requests

PR	State	Signal	Assessment
#4211	open	also references issue #4151; referenced as #4211	not assessed
#4180	open	referenced as #4180	not assessed

Reference data changes

Change	Files
text	8

• text tests/data/pdf/groundtruth/2203.01017v2.json: 3 text item(s) changed
• text tests/data/pdf/groundtruth/2203.01017v2.md: +3 / -3 lines
• text tests/data/pdf/groundtruth/2206.01062.json: 1 text item(s) changed
• text tests/data/pdf/groundtruth/2206.01062.md: +1 / -1 lines
• text tests/data/pdf/groundtruth/2305.03393v1.json: 2 text item(s) changed
• text tests/data/pdf/groundtruth/2305.03393v1.md: +2 / -2 lines
• text tests/data/pdf/groundtruth/newspaper-00.json: 1 text item(s) changed
• text tests/data/pdf/groundtruth/newspaper-00.md: +1 / -1 lines

_{The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.}

[github-actions]

AI first review (advisory): no blocker found

This review covers the full diff at 03c6d1df07. It read the code and did not run it.

I checked the line-final hyphen retention logic in docling/models/stages/page_assemble/page_assemble_model.py and docling/models/stages/reading_order/readingorder_model.py, and verified the test cases and reference data updates.

_{A maintainer makes the merge decision. This review never approves the pull request.}