[kevin9327]

Part B of #4672 (the docling-side change; part A in docling-parse is independent).

Problem

PageAssembleModel.sanitize_text joins every line break of a cluster with " ". Chinese and Japanese put no spaces between words, so every wrapped line of a horizontal CJK paragraph gets a spurious space in the middle of a word or sentence (地方公共団 体, 准确识别版面结构并保留表格、公式和阅读顺序， 从而…).

Change

• A break is closed up when the last character of the previous line and the first character of the next line are both Han, kana, or CJK / full-width punctuation (_CJK_NO_SPACE_RE; covers Extensions A–H, compatibility ideographs, half-width katakana, vertical punctuation forms).
• Hangul is excluded, and a cluster that contains any Hangul keeps every space: Korean separates words with spaces, and Hanja inside Korean text (대한민국 憲法 第一條) are space-separated words too. This also leaves the Korean ground truth (normal_4pages) untouched.
• Mixed breaks (docling は / PDF, 令和元年 / 10月) still get a space; hyphenation handling is unchanged.
• The Hangul scan only runs once a CJK/CJK break is found, and a single code-point comparison skips the regex for Latin text.

Verification

• tests/test_page_assemble_model.py: 17 new cases (Japanese, Chinese, full-width punctuation, half-width katakana, Ext-B ideograph, multi-line, mixed Latin/CJK, digits, Korean, Korean+Hanja, hyphenation, empty line). 6 of them fail on main; all 56 pass with this change.

• End to end, standard PDF pipeline (no OCR) on a horizontal PDF rendered with Noto Serif CJK, counting spaces between two CJK characters in the output:

  paragraph	main	this PR
  Japanese (4 wrapped lines)	4	0
  Chinese (3 wrapped lines)	3	0
  Korean	unchanged	unchanged

• Cost of sanitize_text on an 8-line cluster (best of 5 × 20k): Latin 2.75 → ~3.1–3.5 µs, Japanese 2.5 → 7.0 µs (that path now does the real work). Negligible next to the layout model per page.

• ruff format/ruff check clean; mypy reports only the pre-existing FigureElement error on main.

Vertical text (part A) and vertical reading order remain out of scope.

[github-actions]

✅ DCO Check Passed

Thanks @kevin9327, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[github-actions]

AI triage (advisory)

Checked commit 130bb27ffc. The analysis read the diff only. It did not run code from this pull request.

The PR changes PageAssembleModel.sanitize_text to join wrapped lines without a space when both sides of the break are Han, kana, or CJK/full-width punctuation. Clusters that contain Hangul keep every space. The PR adds tests for this.

CI safety: no concern found. A maintainer must still approve CI.

1 related pull request(s) in the same code

PR	State	Signal	Assessment
#4387	open	changes the same code	related: Both PRs change sanitize_text in page_assemble_model.py. This PR adds CJK line joining. Candidate 4387 changes which line-final hyphens are kept. The fixes are different, but the edits are close and may need a merge.

_{The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.}

[github-actions]

AI first review (advisory): no blocker found

This review covers the full diff at 1a902ea5c8. It read the code and did not run it.

The CJK join logic in sanitize_text is correct for the cases I traced, and the tests cover it. The main problem is scope: the PR also reformats Markdown code blocks in four usage-skill files, and none of that is mentioned in the description. Ask the author to remove those changes.

_{A maintainer makes the merge decision. This review never approves the pull request.}

[github-actions]

suggestion: Unrelated reformatting of four usage-skill files

The PR changes only code-block formatting in extraction.md, python-sdk.md, rag.md and service-client.md. It re-wraps lines, changes comment spacing and adds blank lines. None of this relates to CJK line joining, and the description does not mention it. These files are shipped in the wheel. They also overlap with open PRs #4218, #4496 and #4752, so the churn will cause merge conflicts for those PRs. Revert these four files and keep the PR to page_assemble_model.py and its test. If a formatter hook caused the changes, open a separate PR for them.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[github-actions]

AI first review (advisory): no blocker found

This review covers the changes since 1a902ea5c8 at 130bb27ffc. It read the code and did not run it.

The new commit only reverts the Markdown formatting changes in the four usage-skill files. The PR is now back to page_assemble_model.py and its test. I found no new defect in the incremental changes.

Earlier findings

• fixed (suggestion): Unrelated reformatting of four usage-skill files

_{A maintainer makes the merge decision. This review never approves the pull request.}