Skip to content

fix(xlsx): do not create empty cells while finding tables - #5024

Open
chdre wants to merge 2 commits into
docling-project:mainfrom
chdre:fix/xlsx-no-empty-cell-materialization
Open

chdre wants to merge 2 commits into
docling-project:mainfrom
chdre:fix/xlsx-no-empty-cell-materialization

Conversation

@chdre

@chdre chdre commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Table detection in MsExcelDocumentBackend called sheet.iter_rows(...) and sheet.cell(...) over the whole data bounds. In openpyxl both create a Cell for every position they visit and keep it in sheet._cells. A sheet with a few values in a large rectangle therefore grew by one cell object per empty position, about 200 MB per million cells, and that memory was not released.

This change reads sheet._cells directly in _find_data_tables and _find_table_bounds:

  • Existing cells are visited in the same row-major order.
  • A missing cell counts as having no value and no text.
  • A missing merge anchor is still recognised, through a new _MergedCellIndex.contains_at.

The output does not change. _cells was already read directly in _find_true_data_bounds and _value_bounds.

Checklist

  • Documentation has been updated, if necessary. (not needed, internal change)
  • Examples have been added, if necessary. (not needed)
  • Tests have been added, if necessary.

Review evidence

Measured with DocumentConverter().convert() on generated workbooks, median of 3, tracemalloc (Python allocations only), Windows, Intel Core Ultra 9 285HX.

Workbook Cells in sheet after table detection Peak memory Wall time Markdown
Sparse: values in A1 and AD33334 (before) 1,000,020 194.3 MB 4.1 s identical
Sparse (after) 2 1.1 MB 0.5 s identical
Dense, 5000 rows x 30 columns (before) 150,000 388.3 MB 12.4 s identical
Dense (after) 150,000 388.3 MB 11.7 s identical

Dense sheets are unchanged, as expected.

New tests in tests/test_backend_msexcel.py:

  • test_find_data_tables_does_not_materialize_empty_cells: the sheet's cell count is unchanged by table detection, and peak memory stays under 50 MiB. It fails on main.
  • test_sparse_merged_anchor_and_comment_only_cell_tables: output equality for merges and comments in a sparse sheet.
  • test_missing_merge_anchor_inside_table_keeps_span: a merge whose anchor cell is missing keeps its span. It fails without contains_at.

Old and new _find_data_tables output was also compared on all workbooks in tests/data and on 300 random sheets (values, merges, comments, gap tolerance 0 to 3), with 0 differences.

Commands run:

  • uv run pytest tests/test_backend_msexcel.py tests/test_backend_msexcel_charts.py: 43 passed, 2 skipped.
  • uv run prek run --files ...: all hooks passed.

Not run:

  • The full test suite.
  • Ground truth was not regenerated. test_e2e_excel_conversions passes unchanged.

Not included:

  • Lighter intermediate objects for ExcelCell and TableCell.
  • Processing a sheet in row windows.
  • The chart code that still calls sheet.cell(); it is already capped by MAX_CHART_TABLE_CELLS.

Table detection used sheet.iter_rows and sheet.cell, which make openpyxl create a Cell for every empty position in the data bounds. Read sheet._cells directly instead, so memory is proportional to the filled cells. Output is unchanged.

Signed-off-by: Christer Dreierstad <christerdr@outlook.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @chdre, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) bug Something isn't working performance xlsx issue related to xlsx backend labels Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

AI triage (advisory)

Checked commit 9840b009e2. The analysis read the diff only. It did not run code from this pull request.

The PR changes xlsx table detection in MsExcelDocumentBackend so it reads only existing cells from sheet._cells. This stops openpyxl from creating empty cells for sparse sheets. It also adds _MergedCellIndex.contains_at and new tests.

CI safety: no concern found. A maintainer must still approve CI.

1 related pull request(s) in the same code
PR State Signal Assessment
#5015 open changes the same code related: Both PRs change msexcel_backend.py and its tests. This PR fixes memory use in table detection. The other PR reads images placed in cells, which is a different change.

The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI first review (advisory): no blocker found

This review covers the full diff at 7dbc2abc65. It read the code and did not run it.

I checked table detection in MsExcelDocumentBackend for sparse sheets, coordinate traversal order, and merged-cell fallback handling. The change avoids creating empty Cell objects in sheet._cells and keeps table extraction results intact.

A maintainer makes the merge decision. This review never approves the pull request.

@github-actions github-actions Bot added the ai:review-lgtm AI first review found no blocker or major issue (advisory) label Oct 26, 2026
@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
docling/backend/msexcel_backend.py 83.33% 3 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@github-actions github-actions Bot mentioned this pull request Oct 27, 2026
1 of 3 tasks
Signed-off-by: Christer Dreierstad <christerdr@outlook.com>
@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) and removed ai:ci-safe AI triage found no CI-safety concern (advisory) ai:review-lgtm AI first review found no blocker or major issue (advisory) labels Oct 27, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI first review (advisory): no blocker found

This review covers the changes since 7dbc2abc65 at 9840b009e2. It read the code and did not run it.

The incremental change adds one test (test_threaded_comment_fallback_without_xml) for the threaded-comment XML-absent fallback path. The test logic is correct: cells with values are already in _cells after load, comments are injected in-memory, and _parse_threaded_comments correctly returns empty because no XML part exists in the saved file. The earlier review found no blockers; the original change has no new behavioral code in this diff. The human reviewer should confirm that Codecov's 5 uncovered lines do not include critical branches from this PR's new code paths.

A maintainer makes the merge decision. This review never approves the pull request.

@github-actions github-actions Bot added the ai:review-lgtm AI first review found no blocker or major issue (advisory) label Oct 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:ci-safe AI triage found no CI-safety concern (advisory) ai:review-lgtm AI first review found no blocker or major issue (advisory) bug Something isn't working performance xlsx issue related to xlsx backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant