DOCS-2996: Add all-versions sitemap and full-coverage link check - #2911
Merged
ctauchen merged 2 commits intoAug 11, 2026
Conversation
Add a second sitemap plugin instance that emits sitemap-full.xml with no version ignorePatterns, so it lists every built version rather than latest only. The preset sitemap.xml is unchanged: it stays latest only for search engines and remains the only sitemap referenced in robots.txt. Add a FULL_COVERAGE env switch to the crawler so it seeds from sitemap-full.xml when set, and turn it on in the weekly link check workflow. The weekly job is non-blocking and files an issue on failure, so it is the safe place to surface the full-version backlog. PR CI is unchanged and still crawls latest only, so this does not gate any pull request yet.
✅ Deploy Preview for calico-docs-preview-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview succeeded!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands link-check crawl coverage to include all built documentation versions by introducing an additional “full” sitemap and a FULL_COVERAGE toggle, while keeping PR CI behavior unchanged (latest/SEO-limited coverage).
Changes:
- Add a second Docusaurus sitemap plugin instance that emits
sitemap-full.xmlwithout version ignore patterns. - Add
FULL_COVERAGEenv switching in the crawler to seed fromsitemap-full.xmlwhen enabled. - Enable
FULL_COVERAGEin the weekly link-check workflow (non-blocking) to surface the full-version backlog.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| docusaurus.config.js | Adds a second sitemap plugin instance to generate an all-versions sitemap (sitemap-full.xml). |
| .github/workflows/link-check-weekly.yml | Enables full-coverage crawling for the weekly (non-blocking) link check job via FULL_COVERAGE=true. |
| tests/crawler.test.js | Makes the crawler seed from sitemap-full.xml when FULL_COVERAGE=true and logs the coverage mode. |
Suppressed comments (1)
tests/crawler.test.js:618
- The log message prints “latest only” for the default crawl, but the default sitemap in this repo is “SEO-limited” (via ignorePatterns) rather than strictly “latest-only”. Updating the wording makes log output accurate and easier to interpret when debugging coverage.
console.log(`Crawling the docs (${DOCS}) and executing tests.`);
console.log(`Coverage: ${fullCoverage ? 'FULL (all versions)' : 'latest only'} via ${SITEMAP}.`);
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+25
to
+28
| // FULL_COVERAGE seeds the crawl from the all-versions sitemap (sitemap-full.xml) instead of the | ||
| // latest-only sitemap.xml. Used by the weekly job; leave off for PR CI until the backlog is fixed. | ||
| const fullCoverage = process.env.FULL_COVERAGE === 'true'; | ||
| const SITEMAP = fullCoverage ? 'sitemap-full.xml' : 'sitemap.xml'; |
Comment on lines
+414
to
+417
| // Second sitemap instance for the internal link checker only. The preset sitemap above | ||
| // stays latest-only for SEO (sitemap.xml, the one advertised in robots.txt). This one has | ||
| // no version ignorePatterns, so it lists every built version at sitemap-full.xml. The | ||
| // crawler seeds from it when FULL_COVERAGE=true; it is not referenced in robots.txt. |
Comment on lines
+47
to
+50
| # Crawl every built version (via sitemap-full.xml), not just latest. This job is | ||
| # non-blocking and files an issue on failure, so it is the safe place to surface the | ||
| # full-version backlog. PR CI stays latest-only until the backlog is cleared. | ||
| FULL_COVERAGE: 'true' |
The second sitemap instance references @docusaurus/plugin-sitemap by name, but it was only present transitively via preset-classic and could not be resolved as a standalone plugin. Add it as a direct dependency, pinned to 3.10.1 to match the rest of the Docusaurus packages, since Docusaurus requires all of its packages to be on the same version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The build link checker only verifies pages in the latest version of each product, because the crawler seeds from sitemap.xml, which is deliberately latest only. Older and preview versions are built and served but never link-checked, so a dead link on an older page passes CI while the same link on latest fails. This was the root cause behind the split behavior seen in DOCS-2995.
This is phase one: produce full-version coverage and start surfacing the backlog, without changing what gates pull requests.
Changes:
What this does not do: it does not gate any pull request on older versions. Enforcement policy for PR CI is a later decision, to be made only after the backlog surfaced by the weekly job is triaged and fixed. See DOCS-2996 for the full plan and the enforcement options.
Verification: once the deploy preview builds, sitemap-full.xml lists numbered versions in addition to latest, while sitemap.xml is unchanged. The tigera deploy preview link check runs without FULL_COVERAGE, so its behavior is unchanged.
https://tigera.atlassian.net/browse/DOCS-2996