Skip to content

fix: retain DOCX table content during knowledge ingestion - #9401

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/knowledge-docx-table-content
Closed

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/knowledge-docx-table-content

Conversation

@leonlaiyc

Copy link
Copy Markdown
Contributor

Problem / Motivation

Knowledge ingestion reads only Document.paragraphs, which excludes every table cell. A DOCX price list containing only a table is extracted as an empty string; a report with paragraphs around a table silently loses the table's values before hashing, chunking and indexing.

Why it matters

The Knowledge library cannot retrieve facts that exist only in document tables, despite accepting DOCX files as a supported format.

What changed (motivation → approach → change)

Walk physical body/table/row/cell nodes in document order and pass their paragraphs through the existing text/heading conversion. This retains nested table content and avoids repeating merged-cell proxies. Keep revision wrappers excluded and preserve the existing top-level paragraph_count metadata. Update the reader specification.

Tests

  • Red-before on main c02cdd67cd811e24b136efcc531da1bbdfb47082: three real DOCX fixtures fail because table-only/merged content is empty and interleaved/nested text is missing.
  • Green-after: 19 passed, covering the new real-file regression suite and existing TestFileReader* / TestDocxContentType tests in test/test_knowledge.py, using Python 3.13.5 and two xdist workers without coverage.
  • Added revision-wrapper coverage to retain the existing exclusion behavior. All fixtures stay under tmp_path.
  • Scoped isort, flake8, mypy and Black for the new test pass; the existing reader is on the repository's Black baseline. Both local review contracts report no findings (secondary review uses a substitute model; Opus unavailable).

Manual verification

N/A — real DOCX archives are written and reopened through FileReader.read, the production extraction boundary. Tests assert extracted values, ordering and metadata without network or host-state changes.

Related Issues

No linked issue. Fresh main, open PRs, both contributor handoffs and local claims/worktrees have no equivalent fix or owner of knowledge/readers.py. Other open changes to the shared Knowledge specification affect unrelated sections; their patches do not touch the DOCX reader entry.

Pattern harvest

Rule candidate: review-prompt

Pattern: Document paragraph collections may exclude table content; verify extraction with table-only documents and interleaved body/table content.

Checklist

  • One Conventional Commits commit
  • Relevant existing tests pass and real-file regression tests added
  • Self-review and local contract reviews completed
  • Reader specification updated
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@leonlaiyc
leonlaiyc requested review from a team and krishdhasmana September 8, 2026 07:06
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of af6c70dec0455307ff5e8378ad0ea92277cb21f6 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Real ingestion gap, fixed at the root (document-order XML walk), proportionate, spec updated in the same commit, pinned by real-file regression tests.

[DESIGN-REVIEWED] af6c70d

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of af6c70dec0455307ff5e8378ad0ea92277cb21f6 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All checks complete. Here is the review.

First-Principles-Verdict: PASS

The walker bypasses python-docx's public API for oxml internals (doc.element.body, qn) — confirm behavior on the python-docx>=1,<2 floor (setup.cfg:67), not just 1.x-latest.

What this change ships

Intent: make facts that live only in DOCX tables retrievable from the Knowledge library — a FIX, with provenance (three added tests that fail on base: doc.paragraphs excludes table cells, verified by reading readers.py:186).

Inventory (6 items)
  1. Table-cell text, including nested tables, now appears in extracted DOCX knowledge — justified
  2. Table text appears interleaved in body order, not appended — justified
  3. Merged cells contribute their text once, not once per grid position — justified
  4. Revision-wrapped (w:ins) paragraphs stay excluded, now pinned by a test — justified
  5. paragraph_count metadata keeps its meaning (top-level body paragraphs only) — justified
  6. Reader spec line in docs/system-specs/modules/knowledge.md updated in the same commit — justified

No deleted pins: the existing DOCX tests (test/test_knowledge.py:2553-2580) assert only content_type and dispatch, and none pinned table exclusion. The analogous miss in _read_pptx (readers.py:162-168, text frames only, 1 sibling counted) sits behind the recorded decision keeping .pptx out of SUPPORTED (readers.py:83), so it is not an unfixed sibling of a reachable harm. paragraph_count has zero consumers outside its defining site and spec, but it predates this PR and is preserved, not added.

[FIRST-PRINCIPLES-REVIEWED] af6c70d

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed af6c70dec0455307ff5e8378ad0ea92277cb21f6 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] af6c70d

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed af6c70dec0455307ff5e8378ad0ea92277cb21f6 via the fork AI-review pipeline; updated in place on each push.

Review details

I have enough to decide. Let me verify my reasoning on the candidate's core claim about the observable outcome.

The candidate concerns empty table-cell paragraphs producing blank lines and an "inflated" line_count. Checking the bar:

  • (a) input — a docx with empty/merged/nested table cells: real. ✓
  • (b) path — empty w:p elements yield para.text == '', appended unconditionally, joined with \n, then line_count = text.count('\n') + 1: accurate. ✓
  • (c) observable wrong outcome — this is where it fails. The pre-change code already appended empty body paragraphs (lines.append(text) with no emptiness guard), so blank lines in extracted text and newline-counted line_count are the reader's existing, consistent behavior, not something this diff makes wrong. line_count counts newlines in the text actually returned — it is not "inflated relative to actual content" in any consumer-observable sense; the candidate itself admits it "could not confirm whether downstream chunking/indexing cares." That is a "might," and the extracted-text blank lines are exactly what the author documented and pinned in tests as intended. No wrong runtime behavior is demonstrated.

The candidate resolves to a cosmetic, unconfirmed-consequence tradeoff — its own confidence is "low." It does not clear Step 1. No security, crash, corruption, or data-loss angle exists, and nothing else in the diff grounds a new finding at 80+ (the DFS walk preserves order, skips section/table-property elements safely, and excludes revision wrappers consistently).

No findings.

[OPUS-REVIEWED] af6c70d

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 8, 2026
@NicholasRBowers NicholasRBowers added the needs-pr-triage PR scanner: awaiting automated triage label Sep 15, 2026
@chenmingwei23 chenmingwei23 added drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 15, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor

Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days with a failing readiness check. I have assessed the blockers and they appear resolvable, so I will push fixes directly to this branch as a co-author.

Assessment: The board is green except one lane. mergeable=MERGEABLE (no conflict); all backend test shards (Linux + Windows) pass; all 5 AI review lanes pass with zero blocking findings. The sole failing lane, Backend Lint & Type Check (3.12), was CANCELLED mid-run on the 2026-09-08 merge ref (concurrency cancellation, not a real lint/type error), and PR Readiness is red only because it aggregates that cancelled lane. Fix plan: rebase onto current main to cut a fresh merge ref so the lane re-runs to completion; fix mechanically only if the lane surfaces something real on the current base. No design change.

If you would prefer I do not touch this PR, add the pr-no-autofix label.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 23, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks for this fix, @leonlaiyc — retaining DOCX table content during knowledge ingestion is a real gap worth closing.

To land it against current main (this PR had drifted into a conflict), your commit has been re-applied onto a fresh branch in an in-repo replacement PR, with your authorship preserved and a Co-authored-by trailer:

The replacement carries your _read_docx body walk and tests, plus one follow-up: the walk now also descends into content-control wrappers (w:sdt / w:sdtContent / w:customXml) that Word emits from templates, so rows and paragraphs inside those wrappers are ingested too. It is green (all checks pass, both AI review lanes clear).

Closing this one in favor of the replacement. Full credit for the fix is yours.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 28, 2026
@mbriones98 mbriones98 removed the drive-to-green PR claimed by drive-to-green pipeline label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants