Repository navigation
fix: retain DOCX table content during knowledge ingestion - #14616
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Already-ingested DOCX files never re-extract: mtime-gated scans skip unchanged files, so existing libraries silently keep their table-less text after upgrade. WatchNo migration story for prior ingests. The PR's stated harm — "the knowledge base looks like it ingested the file while missing most of what the user cared about, and nobody is told" — persists for every DOCX ingested before this fix: [DESIGN-REVIEWED] 793ded3 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/knowledge/readers.py:217 -- removing False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the content-control widening: Not justified as shipped
What this change shipsInventory (3 items) — 2 justifiedIntent: get the table text inside a Word file into the knowledge base instead of silently dropping it — a FIX.
Provenance is checkable: [FIRST-PRINCIPLES-REVIEWED] 793ded3 |
Walk the physical document body, descending into table containers and content-control (w:sdt/w:customXml) wrappers, so table-cell and template content is ingested instead of dropped. Revision-wrapped paragraphs stay excluded and paragraph_count stays the top-level body count. Co-authored-by: Leon <leon.lai.yc@gmail.com>
c8d63a3 to
793ded3
Compare
|
self-added: yes
|
|
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
bolichen97
left a comment
There was a problem hiding this comment.
Read the body-order walk; containers set covers tbl/tr/tc + sdt/sdtContent/customXml, w:ins stays excluded as before, paragraph_count unchanged. test/test_knowledge_docx_tables.py 5 passed locally. LGTM.
Problem / Motivation
Goal: A Word file whose facts live in tables gets those facts into the knowledge base, not dropped.
Today
_read_docxreads onlydoc.paragraphs, which is the top-level body text. Any text inside a table cell is skipped. A price list, a spec sheet, or any document that puts its content in a grid ingests as near-empty, so searches over it find nothing.Why it matters
Tables are where a lot of documents keep their real content. Silently dropping every table cell means the knowledge base looks like it ingested the file while missing most of what the user cared about, and nobody is told.
Not a goal
What changed (motivation → approach → change)
A DOCX file's tables were invisible to ingestion because
_read_docxinsrc/kiro_crew/knowledge/readers.pyiterateddoc.paragraphs, which never descends into table cells.The fix rewrites that method to walk the document's physical body in order instead. It steps through each body element, descending into
w:tbl/w:tr/w:tccontainers (so nested tables come along) and intow:sdt/w:sdtContent/w:customXmlcontent-control wrappers (so template fields Word wraps around rows and paragraphs come along too), collecting everyw:pparagraph it meets. Merged cells are visited once each because it walks the physical cells, and revision-wrapped paragraphs stay excluded exactly as python-docx's own paragraph collection excludes them.Heading Nstyles still become#-prefixed markdown, andparagraph_countstill reports the top-level body count.docs/system-specs/modules/knowledge.mdupdates the_read_docxreader line to describe the new table-aware walk.Backwards compatibility
Compatible: a document with no tables walks the same body paragraphs in the same order and produces the same output;
paragraph_countis unchanged. The only behavior change is that table-cell text, previously dropped, is now included.Tests
test/test_knowledge_docx_tables.py(five cases): a table-only document keeps its cell text; body and table paragraphs stay in document order with nested tables included; a merged cell's text is not repeated; a content-control (w:sdt) wrapped paragraph is retained; revision-wrapped paragraphs remain excluded.Manual verification
N/A — unit coverage exercises the table walk, ordering, merged-cell dedupe, and revision exclusion directly.
Screenshots / video
N/A — no user-visible frontend surface is touched (backend reader plus a spec doc line).
Related Issues
no linked issue: this supersedes fork PR #9401 and closes no tracked issue.
Supersedes #9401 (author @leonlaiyc); re-applied onto current
main.Pattern harvest
Not generalizable: this is a one-off completeness gap in one reader (iterating the paragraph proxy instead of the physical body); no other reader shares the pattern.
Checklist