test(pipelines): cover the hugo_ingest functions the v4 ingest path uses

[abhishekKokadwar]

Part of #259. First of the three test modules #249's test plan names but that were never committed.

Scope

Only the functions production actually calls. canonical_rag_ingest imports parse_frontmatter and process_html_table from utils/hugo_ingest.py. The third function, clean_hugo_markdown, has no caller, so tests for it would count as coverage without exercising anything (the same problem clean_content has in test_pipeline_utils.py).

File	Change
tests/test_hugo_ingest.py	New, 18 tests
requirements-test.txt	beautifulsoup4, python-frontmatter, markdownify, html-table-rescuer, toml, pinned to the versions in pipelines/requirements.txt and Dockerfile.pipeline

Where the expectations come from

These tests assert intended behaviour, not whatever the code happens to return today. Every assertion is traceable to one of these sources:

• Docstrings and inline comments. For example, "YAML (---) or TOML (+++) frontmatter", "expand rowspan/colspan", and the explicit try/except fallback.
• How the output is consumed. parse_canonical_document reads title, description and int(weight or 0). It groups table rows as consecutive lines starting with | (GFM_TABLE_ROW_RE), so the tests group rows the same way.
• docs/RAG_V4_ARCHITECTURE.md, which names Hugo frontmatter and release tables among the inputs v4 was built to handle.

Where the intent was ambiguous I asserted only the unambiguous part. For colspan, the config sets a strategy only for rowspan. So the test checks that every row keeps the same width, not whether the spanned value is repeated.

What is covered

parse_frontmatter: YAML and TOML produce identical metadata; a byte-order mark doesn't hide the frontmatter; missing keys stay absent so callers fall back to defaults; empty and None input pass through; malformed YAML and TOML fall back to ({}, content) instead of aborting the ingest run.

process_html_table: a table becomes a GFM table with header, separator and rows; a rowspan value repeats into every spanned row (release tables); colspan keeps row width consistent; an escaped pipe inside a cell doesn't add a column; text around and between tables stays off the table rows; the markdownify fallback still yields a table; input without a table comes back unchanged.

Do the tests catch regressions?

I broke hugo_ingest.py in ten ways and re-ran the suite (the source was restored afterwards):

Mutation	Result
TOML frontmatter parsed as YAML	3 failed
byte-order mark no longer stripped	1 failed
malformed frontmatter raises instead of falling back	2 failed
frontmatter left in the body	4 failed
empty input coerced to ""	1 failed
no-table early return removed	1 failed
rowspan strategy REPEAT_VALUE → EMPTY	1 failed
markdownify fallback emits raw HTML	1 failed
converted table not placed on its own lines	1 failed
meta or {} → meta	18 passed

The one survivor changes no behaviour: python-frontmatter returns a dict even for empty or null frontmatter, so or {} is defensive, and nothing a test could observe depends on it.

Checking this caught a weak test of my own. The first "no-table input is unchanged" case used input that BeautifulSoup round-trips unchanged anyway, so it couldn't tell whether the early return existed. It now uses Katib & Trainer <br> v1.9, which BeautifulSoup would rewrite as Katib &amp; Trainer <br/> v1.9.

Verification

• Runs, doesn't skip, in PR Safety conditions. In a fresh Python 3.11 venv with only requirements-test.txt installed (no KFP), all 18 tests pass with 0 skipped. Without the added dependencies, they would all skip.
• The new dependencies don't change any other test. In that same fresh venv, the rest of the suite gives identical results with and without them.
• The pins resolve together. pip check reports no broken requirements alongside fastmcp and the rest.
• Lint and format. pyflakes and pycodestyle (E4/E7/E9) are clean. I couldn't run ruff locally because of a Windows Application Control policy, so I used black -l 120 as a proxy for ruff format. I also avoided implicit string concatenation that fits on one line, since that's where the two formatters disagree. The file is pure ASCII: the byte-order-mark test writes that character as a Python escape sequence rather than embedding it invisibly in the source. CI is the authoritative check.

Interaction with #258

pytest on main currently aborts at collection because of three other test modules, which #258 fixes. This PR doesn't depend on #258. Its tests pass on their own, as above. But the pytest job will stay red until #258 lands.

Found while writing this: a latent bug for the next PR

It isn't in hugo_ingest, so it's out of scope here, but it's worth recording. In canonical_rag_ingest._stash_for_html, the same table cell can be stored in two different ways, depending on unrelated content elsewhere on the page:

Page content	Stored content_text
the table is the only HTML	HPO & NAS, <YOUR_TOKEN>
the table plus any other HTML tag	HPO & NAS

get_text(), which decodes the entities, only runs if a raw < and > survive process_html_table. A page whose only HTML is a table has none left. This also contradicts hugo_ingest's own comment about preserving placeholder tokens like <YOUR_HF_TOKEN>.

It's latent today. I ran all 220 pages under kubeflow/website/content/en/docs through parse_canonical_document, and none of the 124 table blocks is affected. I checked that the scan can detect the bug by feeding it a synthetic page, which it catches. On real pages it doesn't trigger, because every page with a table also contains other HTML. I'll cover it with a test in the canonical_rag_ingest PR.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tarekabouzeid for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.