[dobi02]

Summary

• fall back to an empty native text layer when text decoding fails during explicitly requested full-page OCR
• preserve the existing exception behavior for all other OCR modes
• make table structure processing reuse the OCR-updated parsed page instead of reparsing the malformed backend text layer

Testing

• uv run pytest tests/test_page_preprocessing_model.py tests/test_threaded_pipeline.py -q (6 passed)
• uv run prek run --files docling/models/stages/page_preprocessing/page_preprocessing_model.py docling/models/stages/table_structure/table_structure_model.py docling/pipeline/standard_pdf_pipeline.py docling/pipeline/legacy_standard_pdf_pipeline.py tests/test_page_preprocessing_model.py
• reproduced against the PDF attached to Full-page OCR is blocked by UnicodeDecodeError in native PDF text parsing #3887 with RapidOCR on page 1: conversion status success, one page produced, and 190 characters of OCR output

Fixes #3887

[github-actions]

✅ DCO Check Passed

Thanks @dobi02, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[cau-git]

@dobi02 can you please explain in more detail what you consider the desired behavior? I don't fully see it from this proposal. It appears to be working around a UnicodeDecodeError raised by docling-parse but how does it relate to full-page OCR?

[dolfim-ibm]

@dobi02 kind reminder, did you see the comment of @cau-git above?

[dobi02]

@cau-git @dolfim-ibm
Thanks for the reminder, and sorry for the delayed response.

The connection to full-page OCR is the pipeline stage ordering and which text cells are authoritative:

1. Page preprocessing currently calls page._backend.get_segmented_page() before the OCR stage. This decodes the native PDF text layer.

2. In OcrMode.FULL_PAGE, the OCR post-processing logic intentionally discards the native text cells and uses only the OCR-generated cells as the final page text.

3. For the PDF in Full-page OCR is blocked by UnicodeDecodeError in native PDF text parsing #3887, page rasterization succeeds, but decoding the native text layer raises UnicodeDecodeError. Therefore the pipeline fails during preprocessing and never reaches the full-page OCR stage, even though that OCR mode does not need the failed native text for its final output.

This PR keeps the fallback deliberately narrow: it is enabled only when do_ocr is true and ocr_options.mode == OcrMode.FULL_PAGE. In every other OCR mode, the original exception is still raised because those modes may depend on native text cells.

The empty SegmentedPdfPage is only used as the page geometry/container required by the OCR stage. Full-page OCR then populates its textline cells with the OCR result.

The table-structure change is related for the same reason: after OCR, page.parsed_page is the pipeline’s current, OCR-updated page state. Calling page._backend.get_segmented_page() again would both retry the failing native decode and ignore the OCR-updated cells.

I agree that docling-parse could also be made more tolerant of individual malformed text cells. I see that as a complementary upstream fix; the Docling-side fallback is where we can apply the mode-specific policy that native text failure is recoverable only for explicitly requested full-page OCR.

I reproduced this with the PDF attached to #3887: the page image renders successfully, and after the fallback RapidOCR completes the conversion and produces OCR text.

[cau-git]

@dobi02 this should have been fixed in the meantime on main and a remaining fix will come with #5022 . Therefore I will close this PR. Thanks.