Skip to content

fix(pdf): preserve nested lists and Word circle markers - #5021

Open
its-amann wants to merge 2 commits into
docling-project:mainfrom
its-amann:fix/pdf-nested-lists-5020
Open

its-amann wants to merge 2 commits into
docling-project:mainfrom
its-amann:fix/pdf-nested-lists-5020

Conversation

@its-amann

@its-amann its-amann commented Oct 26, 2026 •

Copy link
Copy Markdown

Group PDF list items by horizontal indentation, attach nested groups to their parent item, and return to the outer list on an outdent. Lists continue across page headers, footers and footnotes. A new column or body paragraph starts a separate list.

Keep quoted numbered siblings at the same level when an opening quote hangs left of the marker. Use the text-line height and matching marker format to distinguish this from a real sublist, including when the parent spans several lines.

Recognise Word's ASCII o circle on items already detected as list items. Store it in marker, remove it from text, and preserve orig. Ordinary prose is unchanged.

Issue resolved by this Pull Request:
Resolves #5020

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary. No example changes are needed.
  • Tests have been added, if necessary.

Validation:

  • 90 focused reading-order, marker, form and layout tests passed; 1 external-dataset test skipped.
  • Checked the public KOFA PDF (pages 2–3), HR-2026-2227-A (pages 3–4), and Finanstilsynet recovery guidelines (pages 3–4 and 17), using the standard OCR/table pipeline. The quoted pairs remain siblings, the page-4 sublist continues after page-3 footnotes, and the KOFA file retains all nine nested children. Text and provenance match the previous PR head.
  • make validate, wheel/sdist builds and whitespace checks pass. The five focused type warnings match the previous PR head.
  • No reference outputs or dependencies changed. The full repository suite was not run.

Related export behavior:
Default Markdown still needs a companion docling-core change: its AUTO mode prints the stored alphabetic o marker again. That patch is implemented and tested separately; it preserves the document fields and ALWAYS behavior. Publication is pending, and the link will be added here. The SymbolMT font issue remains tracked in docling-project/docling-parse#416.

Thanks @morten-lagabote for testing the additional PDFs and identifying the quoted-list and footnote cases. @cau-git and @PeterStaar-IBM, the updated reading-order changes are ready for review.

Signed-off-by: its-amann <amanagnihotri902@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @its-amann, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) bug Something isn't working pdf PDF issue (except docling-parse) reading_order labels Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

AI triage (advisory)

Checked commit ad4fc36fcb. The analysis read the diff only. It did not run code from this pull request.

This change adds nested list detection in PDF reading order based on horizontal indentation. It also recognizes Word circle bullet markers on list items and extracts them into ListItem.marker.

CI safety: no concern found. A maintainer must still approve CI.

The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI first review (advisory): no blocker found

This review covers the full diff at e256508ce6. It read the code and did not run it.

The list-nesting and o-marker changes are logically correct. The list_stack is not cleared when _handle_text_element resets current_list to None for a non-list, non-form element (e.g. a section header between nested list items), but the current_list is None guard at line 758 of readingorder_model.py clears the stale stack before it is ever used, so no incorrect output results. The reviewer should confirm that the column-detection condition (rel.t >= previous_list_element.t - indent_tolerance) uses bottom-left-origin coordinates consistently with how rel.t is populated in _assembled_to_readingorder_elements.

A maintainer makes the merge decision. This review never approves the pull request.

@github-actions github-actions Bot added the ai:review-lgtm AI first review found no blocker or major issue (advisory) label Oct 26, 2026
@morten-lagabote

Copy link
Copy Markdown
Contributor

Tested e256508 with docling-core 2.104.0 on five public Norwegian PDFs, against the base of this PR (821526e). The text is the same in all five. Only the list structure changes.

  • KOFA 2018/246: the nine "o" items are now a nested list in "Følgende inngår i vurderingen:", with marker="o".
  • Finanstilsynet's guidelines for recovery plans: the "o" items on pages 3 and 17 are nested in the same way. The sub-list of page 3 continues with one more "o" item at the top of page 4, and that item starts a new top-level list. Page headers and footers do not end the list, but the footnotes of page 3 do: they come between the two items in the reading order.

One case nests items that are siblings: a quoted list whose first item starts with «. On page 3 of HR-2026-2227-A, the quoted judgment «1. Anken forkastes. / 2. A og NITO … has two items. The first box starts at x 113.4 pt because of the «, and the second at x 118.9 pt. The difference is 5.5 pt, more than indent_tolerance (5 pt), so "2." becomes a sub-item of "1.". Page 4 has the same with «(1) … / (2) …, a quoted section of an act. Norwegian quotes start with «, so any quoted list in a Norwegian document can have this layout.

In Markdown, the marker is still printed (- o Drifts- og vedlikeholdskostnader), because the AUTO marker mode of the Markdown serializer prints a marker that contains a letter.

@github-actions github-actions Bot removed the ai:ci-safe AI triage found no CI-safety concern (advisory) label Oct 26, 2026
Signed-off-by: its-amann <amanagnihotri902@gmail.com>
@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) and removed ai:review-lgtm AI first review found no blocker or major issue (advisory) labels Oct 26, 2026
@its-amann

Copy link
Copy Markdown
Author

Thanks @morten-lagabote for testing the extra PDFs and giving the exact coordinates. I reproduced the three findings.

Pushed ad4fc36 with the reading-order fixes:

  • The quoted «1. / 2. and «(1) / (2) pairs stay siblings. The adjustment uses the text-line height and matching marker format, so it preserves real sublists, including a multiline quoted parent.
  • Footnotes stay in the document and no longer interrupt the sublist between pages 3 and 4. Normal body paragraphs still end the list.

Rechecked KOFA pages 2–3, HR-2026-2227-A pages 3–4, and the recovery guidelines pages 3–4 and 17. Text and provenance match the previous PR head. The four page-range checks pass, as do 90 focused tests (1 dataset test skipped), make validate, and both package builds.

You are also right about Markdown AUTO. I have a tested companion patch for docling-core that omits only unordered o markers in AUTO, keeps ALWAYS unchanged, and preserves marker and orig. I will link that PR here once it is published.

@cau-git, the reading-order update is ready for another look when you have time. The full upstream CI still needs maintainer approval.

@morten-lagabote

Copy link
Copy Markdown
Contributor

Reran on ad4fc36 with the same five PDFs. The two reading-order cases are fixed:

  • HR-2026-2227-A: «1. / 2. and «(1) / (2) are siblings.
  • Finanstilsynet's guidelines: the "o" item at the top of page 4 is in the sub-list of page 3.

The text of all five PDFs is the same as on the base. The only change in order is in the guidelines: in three places, list items that continue on the next page now come before the footnotes of the previous page, not after them. KOFA 2018/246 is the same as with e256508.

@its-amann

Copy link
Copy Markdown
Author

Thanks @morten-lagabote for rerunning all five PDFs and confirming both fixes.

I checked the footnote order too. Keeping the continued list in one group is intentional; the footnote reordering is a consequence of that grouping. The next-page items are added to the existing list group, while the footnotes remain separate body items. Tree traversal therefore visits the continued items before those footnotes. The footnote text, labels and page coordinates are unchanged, but their position in the traversal changes.

I should have called out that distinction in the earlier validation summary: unchanged text and provenance do not mean unchanged traversal order.

@cau-git, is keeping the logical list contiguous the preferred behavior here, or should the previous page's footnotes remain interleaved before its continuation? Preserving both needs explicit handling of that boundary, so I don't want to treat the changed order as automatically correct.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:ci-safe AI triage found no CI-safety concern (advisory) bug Something isn't working pdf PDF issue (except docling-parse) reading_order

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PDF: nested bullet lists are flat, and the "o" marker of the second level stays in the text

2 participants