Repository navigation
Keep superseded versions out of llms.txt - #80
Merged
Merged
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
ahellegit
approved these changes
Oct 14, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Coding agents and search engines currently see 437 superseded pages (v0.15.x–v0.17.x) in docs.cyborg.co/llms.txt alongside the 173 current ones, so agents learn old APIs; after this only the v0.18.x (
versions/latest/) pages are listed.What changed
scripts/docs.py:desired_tagnow returnsnoindex: truefor every non-latest version.noindex: trueis what already keepsversions/next/out of llms.txt; archives got acanonical:link back tolatest/instead, which does not exclude them because Mintlify builds llms.txt from indexed pages only. The module docstring and theCONTRIBUTING.mdlayout table now say archives arenoindex; theSITEconstant only fed the canonical URL and is removed.python3 scripts/docs.py seorewrote 445 archive frontmatters (142 in v0.15.x, 144 in v0.16.x, 159 in v0.17.x):canonical:removed,noindex: trueadded. The other 50 archive pages had nolatest/twin and were alreadynoindex.versions/latest/andversions/next/are untouched..github/workflows/llms-txt.yml: a weekday 09:00 UTC check (alsoworkflow_dispatch) that fetches the live llms.txt, prints the latest/archive entry counts, and fails if anyversions/v0.*entry is present or the latest quickstart is missing.Trade-off
Archive pages leave Google (intended, core#2020) and llms.txt. Whether Mintlify's site search still returns them is unverified — reviewers can check on the preview by searching a v0.17-only term. The version dropdown still reaches them.
Verification
python3 scripts/docs.py seo— updated 445 pages total (142 / 144 / 159)python3 scripts/docs.py seo --check— all pages already correctpython3 scripts/docs.py check— nav parity, internal links and redirects all OKpython3 scripts/docs.py sync-next --check— next/ is in sync with latest/grep -rL 'noindex: true' versions/v0.*/ --include='*.mdx'— empty (all 495 archive pages tagged);git diff --stat -- versions/latest— emptylatest entries: 173, archive entries: 437. After merge, run thellms.txt has no archive pagesworkflow viaworkflow_dispatch~15 min after the deploy and expectlatest entries: 173, archive entries: 0.Closes cyborginc/cyborgdb-core#2711
Refs cyborginc/cyborgdb-core#2020, cyborginc/cyborgdb-core#2051