Repository navigation
Conversation
Formula, code, picture, and chart enrichment kept every page backend until the conversion ended, so memory grew with the page count. Cache the page images those stages crop, then unload the backend. Signed-off-by: Yun. <fangyun1008@gmail.com>
|
✅ DCO Check Passed Thanks @Yun-0000, all your commits are properly signed off. 🎉 |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
AI triage (advisory)Checked commit The PR makes the standard PDF pipeline cache the page images that enrichment stages need. It then unloads the page backend, so memory no longer grows with page count. Topic labels added: CI safety: no concern found. A maintainer must still approve CI. 1 related pull request(s) in the same code
The |
|
Tested on the PDFs from docling-project/docling-parse#424 with this PR's two files on top of docling 2.142.1 (docling-serve 1.38.1, H100 MIG 1g.12gb, OCR + tables + formula/code enrichment + picture classification). Peak / RSS after the conversion:
Conversion time and Markdown output are unchanged. On these PDFs the peak comes mostly from pages still in the parser read-ahead and the stage queues, each holding the shared images from docling-parse#424. So this PR alone lowers the peak by 4–19% and the memory left after the conversion by 48–70%. Together with docling-parse#425 both drop to 5–6 GB. |
Checklist:
Resolves #1886.
Formula, code, picture, and chart enrichment set
keep_backend, so every page backend stayed alive until the conversion ended and memory grew with the page count. Those stages only crop a page image. The pipeline now caches the scales they use and unloads the page backend. A caller that setskeep_backenditself still keeps the live backend.Page.get_image()can crop a cached page image after the backend is gone.Review evidence:
Before:
tests/test_pdf_enrichment_backend_release.pyonmain: 2 failed (test_cached_page_image_is_cropped_without_a_backend,test_enrichment_releases_the_page_backend_after_caching_its_scale), 1 passed (test_explicit_keep_backend_is_left_alive).After: the same file passes, and a page whose enrichment scale is cached no longer keeps its backend. The crop of that cached image stays available after
_integrate_results.Passed:
uv run pytest tests/test_pdf_enrichment_backend_release.pyonmain: 2 failed, 1 passed.uv run pytest tests/test_pdf_enrichment_backend_release.py tests/test_pdf_streaming_foundation.py tests/test_extraction.py::test_standard_pipeline_integrate_preserves_failed_page_category: 11 passed.make validate: passed.Failed:
Not run:
make test. No reference files were regenerated.