feat(rag): hybrid docs retrieval and readable ingest path

[SanthoshToorpu]

Summary

• Index Kubeflow docs as hybrid Milvus rows (dense + BM25 + release_date) and route queries in MCP instead of dense-only search.
• Keep ingest as importable modules in a slim image; kubeflow-pipeline.py is the DAG and milvus_store.py writes the collection (v=4 is only the schema stamp).
• Split MCP into router / search / citations, park incremental ingest under legacy/, and record the eval case in docs/RAG_V4_ARCHITECTURE.md.

Test plan

• pytest tests/test_mcp_server.py tests/test_kubeflow_pipeline_v4.py tests/test_canonical_rag_ingest.py tests/test_hugo_ingest.py
• python docs-agent-mcp/pipelines/kubeflow-pipeline.py compiles
• Chatbot Sources panel still renders citations after MCP deploy
• Do not run incremental pipeline against production kubeflow_docs
• Cluster ingest run needs the docs-rag-ingest image built separately (not part of MCP CD)

Made with Cursor

[tarekabouzeid]

I think this logic could be simplified, have you considered using tool to convert to MD. Can you check markdownify and frontmatter?

[SanthoshToorpu]

Measured this on kubeflow/website default branch (master @ 8ee3017), content/en/docs/**/*.md only (220 files; 210 TOML +++, 10 YAML ---). 26 files have HTML <table, 11 have rowspan/colspan. Parse + chunk only (parse_frontmatter → build_milvus_records); no TEI/Milvus.

	Before (custom +++/--- + rowspan grid)	After (python-frontmatter + markdownify + html-table-rescuer tables)
Files / exceptions	220 / 0	220 / 0
title / weight present	220 / 219	220 / 219
Chunks / median content_text	2276 / 327	2180 / 351
Exact-match (path, chunk_index)	—	92.05% of shared keys; all 26 mismatches are the HTML-table pages

No fail cases: <YOUR_HF_TOKEN> kept, fences intact, no empty titles, no table flattened off pipes. Hard slice (Katib config, release 26.03 date/component tables, style-guide): rowspan labels still repeat (AutoML Working Group ×2, Notebooks Working Group ×9).

Table bake-off on the 26 HTML-table files:

• custom grid: expands rowspan (good for RAG) but no GFM --- row
• markdownify: 26/26 pipe tables; drops spanned cells
• html-to-markdown 1.16: 26/26; empty cells for rowspan (not a win)
• html2text: 25/26; flattens one table
• html-table-rescuer + REPEAT_VALUE: 26/26; repeats rowspan labels like the old grid

Winner: python-frontmatter for YAML/TOML, html-table-rescuer for process_html_table, markdownify still for leftover HTML in clean_hugo_markdown. Still custom (no lib covers Hugo): fence/code/GFM stash, <YOUR_HF_TOKEN> stash, {{% alert %}} / shortcodes, fa-check icons.

That swap is in the worktree (hugo_ingest.py + ingest-image deps). Not committed/pushed until we get a go-ahead.

[SanthoshToorpu]

Lol asked cursor to comment on this but yeah it seems to work lemme test a few more cases

[abhishekKokadwar]

Took a careful pass over this since it touches the ingest path I've been working in. Three findings, all reproduced against b92f87c. The first one ships broken and CI won't catch it.

1. incremental-pipeline.py no longer imports — and the compile step skips it

Moving utils.py to utils/utils.py turns utils into a directory. The three pipelines that got the _UTILS_DIR shim are fine:

_UTILS_DIR = Path(__file__).resolve().parent / "utils"
if str(_UTILS_DIR) not in sys.path:
    sys.path.insert(0, str(_UTILS_DIR))

docs-agent-mcp/pipelines/incremental-pipeline.py (restored to the live folder in b92f87c) did not get it, and still does from utils import DEFAULT_EMBEDDING_BATCH_SIZE, DOCS_COLLECTION at line 11:

$ python incremental-pipeline.py
ImportError: cannot import name 'DEFAULT_EMBEDDING_BATCH_SIZE' from 'utils' (unknown location)

Since there's no utils/__init__.py, utils resolves to the namespace package rather than utils.py.

Why CI stays green. The Compile docs RAG pipeline step runs exactly the three files that work:

python kubeflow-pipeline.py
python issues-pipeline.py
python code-pipeline.py

and compileall in tests.yml only byte-compiles — it never executes the import, so it passes on the broken file:

$ python -m compileall -q incremental-pipeline.py   # exit 0
$ python incremental-pipeline.py                    # ImportError

Prepending the same shim the siblings use makes it compile. Adding python incremental-pipeline.py to that CI step would stop this recurring — I have that one-line addition in #250 if it's useful, happy to drop it there and let this PR own it instead.

2. Two divergent copies of incremental-pipeline.py

The file now exists at both docs-agent-mcp/pipelines/ and legacy/pipelines/, and they are not the same file. Beyond the import line, the legacy/ copy is an older revision that reinstates the CUDA base image #224 (ac49afb) removed:

base_image="docker.io/pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime",
packages_to_install=["sentence-transformers==3.3.1", "transformers==4.44.2", ...]
device = 'cuda' if torch.cuda.is_available() else 'cpu'

main has base_image="python:3.11-slim" there. It also drops the version pins that #224 added. If the legacy/ copy is meant as the archived one, it's archiving a state older than main; if the live one is canonical, the legacy/ copy looks like it can just go.

3. Heads-up on an overlap with #250

#250 adds embedding_dim plus the store_milvus schema guard to store_milvus_incremental, and touches the same oke-cicd.yaml compile step. Given this PR restructures the same files, I'm happy to rebase #250 on top once this lands rather than the other way round — this is much the larger change and shouldn't have to wait on mine.

Nothing here is about the architecture, which reads well — the router/search/citations split in particular. Just the import and the duplicate file, which look like they slipped in during the folder move.

[SanthoshToorpu]

Hello @abhishekKokadwar thanks for the review. fixed the import in incremental PL. gotta refactor that one though

[tarekabouzeid]

Thank you @SanthoshToorpu
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tarekabouzeid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [tarekabouzeid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment