Skip to content

fix: retain DOCX table content during knowledge ingestion - #14616

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/knowledge-docx-table-content-r9
Sep 28, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/knowledge-docx-table-content-r9

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Word file whose facts live in tables gets those facts into the knowledge base, not dropped.

Today _read_docx reads only doc.paragraphs, which is the top-level body text. Any text inside a table cell is skipped. A price list, a spec sheet, or any document that puts its content in a grid ingests as near-empty, so searches over it find nothing.

Why it matters

Tables are where a lot of documents keep their real content. Silently dropping every table cell means the knowledge base looks like it ingested the file while missing most of what the user cared about, and nobody is told.

Not a goal

  • No new file format, config, or ingestion option.
  • No change to how PDF, HTML, or plain-text readers work.
  • Cell styling, column layout, and table borders are not preserved — only the text.

What changed (motivation → approach → change)

A DOCX file's tables were invisible to ingestion because _read_docx in src/kiro_crew/knowledge/readers.py iterated doc.paragraphs, which never descends into table cells.

The fix rewrites that method to walk the document's physical body in order instead. It steps through each body element, descending into w:tbl / w:tr / w:tc containers (so nested tables come along) and into w:sdt / w:sdtContent / w:customXml content-control wrappers (so template fields Word wraps around rows and paragraphs come along too), collecting every w:p paragraph it meets. Merged cells are visited once each because it walks the physical cells, and revision-wrapped paragraphs stay excluded exactly as python-docx's own paragraph collection excludes them. Heading N styles still become #-prefixed markdown, and paragraph_count still reports the top-level body count.

docs/system-specs/modules/knowledge.md updates the _read_docx reader line to describe the new table-aware walk.

Backwards compatibility

Compatible: a document with no tables walks the same body paragraphs in the same order and produces the same output; paragraph_count is unchanged. The only behavior change is that table-cell text, previously dropped, is now included.

Tests

test/test_knowledge_docx_tables.py (five cases): a table-only document keeps its cell text; body and table paragraphs stay in document order with nested tables included; a merged cell's text is not repeated; a content-control (w:sdt) wrapped paragraph is retained; revision-wrapped paragraphs remain excluded.

Manual verification

N/A — unit coverage exercises the table walk, ordering, merged-cell dedupe, and revision exclusion directly.

Screenshots / video

N/A — no user-visible frontend surface is touched (backend reader plus a spec doc line).

Related Issues

no linked issue: this supersedes fork PR #9401 and closes no tracked issue.

Supersedes #9401 (author @leonlaiyc); re-applied onto current main.

Pattern harvest

Not generalizable: this is a one-off completeness gap in one reader (iterating the paragraph proxy instead of the physical body); no other reader shares the pattern.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested a review from a team as a code owner September 28, 2026 04:11
@iamwhatever iamwhatever added the drive-to-green PR claimed by drive-to-green pipeline label Sep 28, 2026
@iamwhatever iamwhatever self-assigned this Sep 28, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 28, 2026
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 793ded367177fcc7a28904602711b358421526e6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Already-ingested DOCX files never re-extract: mtime-gated scans skip unchanged files, so existing libraries silently keep their table-less text after upgrade.

Watch

No migration story for prior ingests. The PR's stated harm — "the knowledge base looks like it ingested the file while missing most of what the user cared about, and nobody is told" — persists for every DOCX ingested before this fix: folder_watcher.py skips files where mtime <= state.get("mtime", 0) and local_folder.detect_changes returns False, so the new walk only runs on files that change on disk. The description's "Backwards compatibility: … table-cell text, previously dropped, is now included" is true only for future extractions.
Clears when: a reader-version stamp in per-file state (or equivalent one-shot invalidation for .docx) ships as a follow-up, or the release notes/changelog explicitly tell users to re-sync DOCX sources.

[DESIGN-REVIEWED] 793ded3

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 793ded367177fcc7a28904602711b358421526e6 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/knowledge/readers.py:217 -- removing "w:customXml" leaves the documented traversal allowlist unpinned -> Fix: exercise a custom-XML wrapper in the new reader tests.
[GPT-REVIEWED] 793ded3

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 793ded367177fcc7a28904602711b358421526e6: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 793ded367177fcc7a28904602711b358421526e6 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the content-control widening: w:sdt-wrapped paragraphs outside any table — text python-docx itself excludes — now ingest, beyond the stated table goal.

Not justified as shipped

  • Item 2 — rides along: declared ("so template fields Word wraps around rows and paragraphs come along too"); required for sdt-wrapped table rows, but its top-level effect exceeds the Goal's "facts live in tables".

What this change ships

Inventory (3 items) — 2 justified

Intent: get the table text inside a Word file into the knowledge base instead of silently dropping it — a FIX.

  1. DOCX table-cell text (nested tables included, each merged cell once) now reaches the knowledge base — justified
  2. Paragraphs wrapped in content controls (w:sdt/w:customXml), including ones outside any table, now ingest — rides along: declared, but widens behavior beyond tables
  3. The knowledge.md reader line now describes the table-aware walk — justified

Provenance is checkable: test_table_only_document_retains_cell_text fails on base, where _read_docx reads only doc.paragraphs. No new config key, flag, or exported symbol (lens 5: nothing to count). Sibling check (lens 6): the only other proxy-iterating reader is _read_pptx (slide.shapes drops table GraphicFrames), but .pptx is deliberately excluded from FileReader.SUPPORTED and no supported path reaches it — 0 reachable siblings. The knowledge.md edit is mandated by the same-commit spec invariant in AGENTS.md.

[FIRST-PRINCIPLES-REVIEWED] 793ded3

Walk the physical document body, descending into table containers and
content-control (w:sdt/w:customXml) wrappers, so table-cell and template
content is ingested instead of dropped. Revision-wrapped paragraphs stay
excluded and paragraph_count stays the top-level body count.

Co-authored-by: Leon <leon.lai.yc@gmail.com>
@iamwhatever
iamwhatever force-pushed the fix/knowledge-docx-table-content-r9 branch from c8d63a3 to 793ded3 Compare September 28, 2026 04:21
@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: DOCX body walk descends w:tbl/w:tr/w:tc plus w:sdt/w:sdtContent/w:customXml content-control wrappers

  • span=80ccb364638c readers.py:213 — content-control-wrapped rows skipped

fixed. The walk's descend set now includes w:sdt, w:sdtContent, and w:customXml, so
content-control-wrapped paragraphs, rows, and cells that Word emits from templates are ingested
instead of dropped. Revision wrappers (w:ins/w:del) stay excluded. New regression test
test_content_control_wrapped_paragraphs_are_retained fails without the fix and passes with it.
Fixed in 793ded3.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 28, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=a33eff640af6 readers.py:217 — w:customXml allowlist entry unpinned by tests

rebutted (advisory). The descent-into-wrapper branch (element.tag in containers -> descend
into the subtree) is already exercised by test_content_control_wrapped_paragraphs_are_retained
via the w:sdt/w:sdtContent path; w:customXml is the identical branch with a different tag
literal in the same set, so the traversal behaviour is covered. Adding a dedicated w:customXml
case is a cheap coverage nicety, not a correctness gap, and does not warrant spending a CI round
on an already-passing head; noting it as a follow-up rather than widening this PR.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 793ded367177fcc7a28904602711b358421526e6 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 793ded3

Verdict parsed from the review's SHA-scoped output markers for commit 793ded367177fcc7a28904602711b358421526e6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 793ded367177fcc7a28904602711b358421526e6: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@chenmingwei23
chenmingwei23 merged commit 2bdda95 into main Sep 28, 2026
99 of 105 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/knowledge-docx-table-content-r9 branch September 28, 2026 05:37

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the body-order walk; containers set covers tbl/tr/tc + sdt/sdtContent/customXml, w:ins stays excluded as before, paragraph_count unchanged. test/test_knowledge_docx_tables.py 5 passed locally. LGTM.

@dwu96 dwu96 removed the drive-to-green PR claimed by drive-to-green pipeline label Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants