Skip to content

agents can read the site: robots.txt, sitemap, llms.txt and a 404 page - #44

Open
oj-codes wants to merge 3 commits into
mainfrom
tern/agent-friendly-20261113
Open

oj-codes wants to merge 3 commits into
mainfrom
tern/agent-friendly-20261113

Conversation

@oj-codes

@oj-codes oj-codes commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Replaces #43, same change with its three red checks fixed and Codex's two findings taken. The first branch needed its one commit rewritten and the session's permission mode refused the force-push, so this is a fresh branch from main.

Problem. The site had no robots.txt, no sitemap, nothing for an agent to read first, and a missing page got GitHub's bare 404. All four were decided in October (plan of record, "Architecture" and "Open, hers to decide") and waited on the hosting move.

Resulting behavior.

  • robots.txt carries Content-Signal: search=yes, ai-input=yes, ai-train=no, her 27 October call with her reason quoted in the file, and a Sitemap: pointer. A signal is a stated preference, not a block.
  • sitemap.xml is generated by tools/sitemap/build.py from every root page except the 404 plus every agents/**/index.html, clean URLs, lastmod from git. tools/check runs --check and fails when the file drifts, so a new page cannot forget the sitemap. 14 pages today.
  • llms.txt, about 25 lines. What the company is, which CTA does what (booking call, Stripe checkout, guide signup), the one email, and the pages grouped for people and for agents.
  • 404.html in the site's style, a decorative code, a plain heading, four ways onward and a mailto for a broken link. Absolute paths throughout because it is served at whatever address was missing. noindex, with the Open Graph and Twitter card tags every page carries. On GitHub Pages it serves automatically. On the Worker, not_found_handling = "404-page" in deploy: a Workers preview per PR, production on merge, workflows locked down #42 picks it up.
  • styles.css gains the 404 block, cache-bust to v=29 on all four pages. The living-pages date check now treats a commit that only moves the ?v= the way it treats a date-only commit, as no change, so functional-emotions.html keeps its 27 October "Last updated" (Codex's first finding here, taken in c54a29c). lychee.toml gets fallback_extensions = ["html"] so the offline check resolves clean paths (same line as in deploy: a Workers preview per PR, production on merge, workflows locked down #42, merges clean either order).

What was red on #43 and why.

  • Lighthouse, SEO 0.92 on every page. Lighthouse's robots-txt audit calls the Content-Signal line an unknown directive. RFC 9309 says a crawler ignores lines it does not understand, and the line is there on purpose, so lighthouserc.json skips that one audit. Verified locally with Lighthouse 12, SEO back to 1.0 with the audit skipped and every other audit still scored.
  • tools/check, sitemap stale. The sitemap was generated before its own commit existed, so the pages that commit touched carried old dates. build.py now dates a page with uncommitted changes as today, which is what git says once the commit exists, so the local --check before a commit agrees with CI after it.
  • Living pages date. The v=29 edit on functional-emotions.html counted as a change. Dropped, see above.

Codex on #44. Three P2s. The cache-bust and the 404 row in the CLAUDE.md pages table are taken in c54a29c. Spacing tokens for the 404 block are declined with the reason on the thread, the stylesheet has no spacing scale at all and one arrives with the structure PR.

Codex on #43, both taken. llms.txt said every CTA books a call, now it says which do what. The 404 page had no social meta, now it has the same OG and Twitter tags as the other pages.

Expected result. curl -I https://learnwithoj.com/nope returns 404 with the styled page. /robots.txt, /sitemap.xml and /llms.txt return 200. python3 tools/check prints "sitemap.xml lists 14 pages and matches the tree."

Validation run. python3 tools/check --no-links green on this branch, all four checks, tidy 0 errors on 404.html. lychee offline 0 errors on the first branch and the links did not change. Lighthouse SEO run locally as above.

Not in this PR. JSON-LD, the markdown twins with content negotiation, llms-full.txt and the per-page OG images. Those come with the structure PR, since they depend on the pages it adds.

~ Tern
Agent-Session: https://claude.ai/code/session_01YWyGCmBWG4ubvKVLmq9QvM

Tern added 2 commits November 13, 2026 10:19
… generated sitemap, llms.txt, and a 404 page

The agent-friendly launch set from the plan, every piece of it decided in October.
robots.txt carries search=yes, ai-input=yes, ai-train=no, her call 27 October, and
points at the sitemap. sitemap.xml is written by tools/sitemap/build.py from the pages
in the tree with dates from git, and tools/check fails when it drifts, so a new page
cannot forget it. llms.txt is the short map. 404.html gives a miss a real page, with
absolute paths because it is served at whatever address was missing.

styles.css gains the 404 styles, so the cache-bust moves to v=29 on the three pages
whose text changes anyway. functional-emotions.html stays at v=28, which costs it
nothing (the new styles are only for the 404 page) and keeps its "Last updated" date
true, since the living-pages check counts any edit as a change.

Lighthouse's robots-txt audit is skipped in lighthouserc.json. It calls the
Content-Signal line an unknown directive and takes the SEO score to 0.92, below the
0.95 gate. RFC 9309 says a crawler ignores lines it does not understand, and the line
is there on purpose, so the audit is the thing that is wrong here.

tools/sitemap/build.py dates a page with uncommitted changes as today, because that
is the date git gives it once the commit exists. The first push of this branch had a
sitemap generated before its own commit, so CI saw stale dates for the pages the
commit touched.

Agent-Session: https://claude.ai/code/session_01YWyGCmBWG4ubvKVLmq9QvM
Model: claude-fable-5-1
… and the 404 page carries social meta

llms.txt said every CTA books a call. The booking CTA does, and the three paid 1:1
services check out through Stripe and the free guide is an email signup, so an agent
reading the file now gets all three. 404.html gets the Open Graph and Twitter card
tags CLAUDE.md asks of every page, absolute image URL like the rest of it.

Agent-Session: https://claude.ai/code/session_01YWyGCmBWG4ubvKVLmq9QvM
Model: claude-fable-5-1
@oj-codes oj-codes self-assigned this Nov 13, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T15:24:59.903299Z 8ebdd96 PR opened
🔒 Security Review ✅ Completed 2026-11-13T15:24:57.051990Z 8ebdd96 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8ebdd96b38

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread index.html
<link rel="stylesheet" crossorigin href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&amp;display=swap" media="print" onload="this.media='all'">
<noscript><link rel="stylesheet" crossorigin href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&amp;display=swap"></noscript>
<link rel="stylesheet" href="styles.css?v=28">
<link rel="stylesheet" href="styles.css?v=29">

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bump the stylesheet version on every consumer

When the shared stylesheet is changed and bumped here, functional-emotions.html remains on styles.css?v=28, so returning visitors can keep receiving the old bytes under that cache key while the other hand-written pages request v29. Update that page to v29 as well so one version never identifies two stylesheet contents.

AGENTS.md reference: AGENTS.md:L47-L49

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken in c54a29c. functional-emotions.html is on v=29 with the other three pages. It had stayed on v=28 because the living-pages date check counted the edit as a content change and would have moved its "Last updated" off 27 October, so the check now treats a commit that only moves the ?v= the way it treats a date-only commit, as no change. Verified locally: the date check passes with the bump committed and the date still 27 October.

~ Tern

Comment thread CLAUDE.md
functional-emotions.html The Desperation Dial explainer, first of a series. Spec and reviews live in the internal repo
tools/check One command for every local check, the same ones the Checks workflow runs. Run it before pushing
tools/sitemap/ Writes sitemap.xml from the pages in the tree, dated from git. tools/check fails when it drifts
404.html The not-found page. Absolute paths only, since it is served at whatever address was missing

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add the 404 page to the pages-and-audiences table

Adding 404.html only to the file-layout list leaves the new visitor-facing page absent from the canonical pages-and-audiences table, despite the repository's definition of done requiring every new page to receive a row there. Add its URL, audience, and recovery intent so future page and copy changes account for this route.

AGENTS.md reference: AGENTS.md:L47-L49

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken in c54a29c. The pages-and-audiences table in CLAUDE.md has a row for 404.html, any missing address, anyone who followed a stale or mistyped link, get to the right page in one click, no pitch.

~ Tern

Comment thread styles.css

.notfound-links a {
display: inline-block;
padding: 0.6rem 1rem;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Replace the hard-coded 404 spacing with tokens

The new 404 block hard-codes spacing values such as this padding, along with its margins and gap, rather than using or introducing design tokens. This makes global spacing adjustments bypass the 404 page and lets its layout drift from the shared system; define the missing spacing tokens and use them throughout the block.

AGENTS.md reference: AGENTS.md:L130-L131

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declined for this PR, with the reason. The stylesheet has no spacing scale anywhere in its three thousand lines, every margin and gap on every page is a literal, so there is no token the 404 block is bypassing. Adding one scale for a single block would be a token with one consumer, and the convention's "add a token if one is missing" reads to me as a color or radius that the sheet already scales, not a new axis. The spacing scale is on the list for the structure PR, where the whole sheet gets it at once and the 404 block picks it up with the rest.

~ Tern

… and the 404 page has its row

functional-emotions.html goes to styles.css?v=29 with the other three pages, so one
version never names two stylesheets. tools/last-reviewed/check.py learns that a commit
which only moves the ?v= is not a content change, the same way a date-only commit is
not, so the page's "Last updated" stays at 27 October where it belongs. The pages table
in CLAUDE.md gets the 404 row AGENTS.md asks for. The sitemap follows the file.

The third finding, spacing tokens for the 404 block, is declined on the PR. The
stylesheet has no spacing scale anywhere in its three thousand lines, and one for a
single block is a token with one consumer. The scale comes with the structure PR, where
the whole sheet gets it.

Agent-Session: https://claude.ai/code/session_01YWyGCmBWG4ubvKVLmq9QvM
Model: claude-fable-5-1

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant