Repository navigation
fix(markdown): keep GFM task-list checkboxes and paragraph [x] markers - #4935
Siddharth-DWT wants to merge 2 commits into
Conversation
marko's GFM paragraph strips a leading "[x] " / "[ ] " from every paragraph and records it in `checked`, which the Markdown backend never read. Task-list items lost their state, and any paragraph starting with the marker lost that text. Only a list item's first paragraph now treats the marker as a task marker; the backend turns those items into checkbox_selected / checkbox_unselected items in the list, as the HTML backend does. Elsewhere the marker stays text. Fixes docling-project#4930 Signed-off-by: Siddharth-DWT <mernhash@gmail.com>
|
✅ DCO Check Passed Thanks @Siddharth-DWT, all your commits are properly signed off. 🎉 |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
AI triage (advisory)Checked commit The PR changes CI safety: no concern found. A maintainer must still approve CI. 3 related pull request(s) in the same code
The |
There was a problem hiding this comment.
AI first review (advisory): no blocker found
This review covers the full diff at 9134a0886e. It read the code and did not run it.
The fix is correct for the main cases: bullet task lists, nested task lists, and [x] paragraphs outside list items. The main risk is ordered task lists. They now export with the wrong marker, and the PR does not test them. Check that the author accepts the change for ordered lists.
A maintainer makes the merge decision. This review never approves the pull request.
| start = list_start_by_ref.get(parent_ref, 1) | ||
| count = list_item_counter_by_ref.get(parent_ref, 0) | ||
| marker = f"{start + count}." | ||
| checked: bool | None = getattr(child, "checked", None) |
There was a problem hiding this comment.
suggestion: Ordered task-list items lose their number on export
Trigger: 1. [x] a\n2. [ ] b. The code makes every checked item a checkbox_* text item through doc.add_text, and it drops enumerated and marker. The docling_core text serializer always writes - [x] or - [ ] for these labels (markdown.py:345-348). Before the PR, the Markdown export kept 1. a and 2. b. After the PR, ordered task items export as bullets. In a mixed list, a plain item can still be numbered by its position in the list. The PR table and the tests cover bullet lists only. Add an ordered-list case to the tests. Either accept the new export and say so in the PR, or keep list_item for enumerated lists.
There was a problem hiding this comment.
Addressed in 36e2adb: task items are no longer converted to checkbox items, so ordered lists keep their numbers (test_ordered_task_list_items_keep_their_numbers).
| @classmethod | ||
| def parse(cls, source): | ||
| state = source.state | ||
| opens_list_item = isinstance(state, marko.block.ListItem) and not getattr( |
There was a problem hiding this comment.
suggestion: "First paragraph" check is wrong when a non-paragraph block comes first, and it probes attributes
The flag _docling_has_paragraph is set only when a paragraph is parsed. In - # Title\n\n [x] text, the heading comes first, so the paragraph counts as the "first" one. checked is set and the marker is stripped. The list-item branch in _iterate_elements needs children[0] to be a Paragraph, so it is skipped. The [x] text is then lost, which is the defect this PR fixes. GFM allows the marker only when the paragraph is the first block of the item. AGENTS.md also asks to avoid getattr, setattr, and hasattr probing. The code uses them on a marko ListItem and on _ParsedLines. Prefer a typed approach, for example a field on a small _GfmParagraph subclass, or a check that the item has no earlier block.
There was a problem hiding this comment.
Addressed in 36e2adb: the marker is never stripped, so the heading-first case keeps it (covered in test_task_markers_outside_list_items_are_kept), and the attribute probing is gone.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
geoHeil
left a comment
There was a problem hiding this comment.
Reviewed 9134a0886efd6abd33c7123e0e4e2614c3fa3a0e. Changes are still needed despite green CI and full patch coverage.
The doc.add_text() checkbox path discards ordered-list markers. 7. [x] seven\n8. [ ] eight\n9. nine exports as - [x] seven\n- [ ] eight\n9. nine. It also changes - [x] done\n- plain into HTML with <ul><p>done</p><li>plain</li></ul>. Preserve the list-item contract in both exports.
The earlier first-block finding also remains open: - # header\n\n [x] ordinary second block still loses [x]. That loss exists on base, so it is an incomplete fix rather than a new regression. See the open ordered-list and first-block discussions: #4935 (comment) and #4935 (comment).
All 73 Markdown and image tests pass locally. The current tests do not detect these cases.
The new export defect is in the checkbox representation.
Local make check stops on file-length violations that already exist on base. The other read-only checks pass. I made no source or reference-data changes.
Review feedback: turning task items into checkbox text items dropped ordered list numbers on export, broke the HTML list structure, still lost the marker when the item did not start with a paragraph, and probed attributes. Base _GfmParagraph on marko's plain Paragraph instead of GFM's, so the "[x] " / "[ ] " marker is never stripped. Task items stay list items with the marker in their text, which keeps numbering and both exports intact, and no paragraph loses its leading marker. Signed-off-by: Siddharth-DWT <mernhash@gmail.com>
|
Thanks for the careful review @geoHeil. You're right on all points, and the checkbox-item approach was the root of them, so I've replaced it in 36e2adb.
Tests cover all of these (8 cases; they fail on |
There was a problem hiding this comment.
AI first review (advisory): no blocker found
This review covers the changes since 9134a0886e at 36e2adb928. It read the code and did not run it.
The updated changes fix the earlier findings by subclassing marko.block.Paragraph in docling/backend/md_backend.py:66. This subclass keeps task-list checkbox markers and ordered-list numbers inside the element text. I checked the subclassing of marko.block.Paragraph and the added tests for nested and ordered lists in tests/test_backend_markdown.py:1323. The code behaves correctly.
Earlier findings
- fixed (suggestion): Ordered task-list items lose their number on export
- fixed (suggestion): "First paragraph" check is wrong when a non-paragraph block comes first, and it probes attributes
A maintainer makes the merge decision. This review never approves the pull request.
marko's GFM
Paragraphstrips a leading[x]/[ ]from every paragraph and keeps it only inchecked, which the Markdown backend never read. Task-list items lost their marker, and any paragraph starting with one lost that text._GfmParagraphis now based on marko's plainParagraph(withoverride = True, and the existing table-awarebreak_paragraph), so the marker is never stripped. Task items staylist_items with the marker in their text.Following review, this replaces the first version, which turned task items into
checkbox_*text items: that dropped ordered-list numbers on export, broke the HTML list structure, still lost the marker when an item did not start with a paragraph, and relied on attribute probing. Keeping the marker as text has none of those problems. Representing task items as structured checkboxes inside a list would need list-item checkbox support in docling-core's serializers, so it is better as a separate change.Issue resolved by this Pull Request:
Resolves #4930
Checklist:
Review evidence:
- [x] done item/- [ ] open item- [x] done item, HTML<li>[x] done item</li>7. [x] seven/8. [ ] eight/9. nine7. seven/8. eight/9. nine7. [x] seven/8. [ ] eight/9. nine- [x] parentwith nested- [ ] child- [x] parent/- [ ] child[x] I accept the terms,> [x] quoted, a second paragraph[x] ...in an item,- # Titlefollowed by[x] ...Tests (
tests/test_backend_markdown.py):test_task_list_items_keep_their_markers(labels, Markdown and HTML export),test_ordered_task_list_items_keep_their_numbers,test_nested_task_list_items_keep_their_markers, andtest_task_markers_outside_list_items_are_kept(5 cases, including the heading-first item). All 8 fail onmainand pass with the change.Checks run:
pytest tests/test_backend_markdown.py tests/test_backend_markdown_images.py: 75 passed, includingtest_convert_validagainst the reference data. No reference data changes: none oftests/data/md/sourcescontains task markers.ruff check/ruff format --check(0.15.12): clean.ty check(0.0.33) onmd_backend.py: same 4 diagnostics as onmain.make validate.Written with AI assistance; I reviewed the change and ran the checks myself.