Skip to content

agents can read the site: robots.txt, sitemap, llms.txt and a 404 page - #43

Closed
oj-codes wants to merge 1 commit into
mainfrom
tern/agent-friendly-20261112
Closed

oj-codes wants to merge 1 commit into
mainfrom
tern/agent-friendly-20261112

Conversation

@oj-codes

Copy link
Copy Markdown
Owner

Problem. The site had no robots.txt, no sitemap, nothing for an agent to read first, and a missing page got GitHub's bare 404. All four were decided in October (plan of record, "Architecture" and "Open, hers to decide") and waited on the hosting move.

Resulting behavior.

  • robots.txt carries Content-Signal: search=yes, ai-input=yes, ai-train=no, her 27 October call with her reason quoted in the file, and a Sitemap: pointer. A signal is a stated preference, not a block.
  • sitemap.xml is generated by tools/sitemap/build.py from every root page except the 404 plus every agents/**/index.html, clean URLs, lastmod from git. tools/check runs --check and fails when the file drifts, so a new page cannot forget the sitemap. 14 pages today.
  • llms.txt, about 25 lines. What the company is, the one CTA, the one email, and the pages grouped for people and for agents.
  • 404.html in the site's style, a decorative code, a plain heading, four ways onward and a mailto for a broken link. Absolute paths throughout because it is served at whatever address was missing. noindex. On GitHub Pages it serves automatically. On the Worker, not_found_handling = "404-page" in deploy: a Workers preview per PR, production on merge, workflows locked down #42 picks it up.
  • styles.css gains the 404 block, cache-bust to v=29 on every page that loads it. lychee.toml gets fallback_extensions = ["html"] so the offline check resolves clean paths (same line as in deploy: a Workers preview per PR, production on merge, workflows locked down #42, merges clean either order).

Expected result. curl -I https://learnwithoj.com/nope returns 404 with the styled page. /robots.txt, /sitemap.xml and /llms.txt return 200. python3 tools/check prints "sitemap.xml lists 14 pages and matches the tree."

Validation run. tools/check --no-links --voice green, tidy 0 errors on 404.html. lychee offline 0 errors across all pages including the new one. Voice linter on 404.html: the only flags left are the shared footer and the repeated brand name.

Not in this PR. JSON-LD, the markdown twins with content negotiation, llms-full.txt and the per-page OG images. Those come with the structure PR, since they depend on the pages it adds.

~ Tern
Agent-Session: https://claude.ai/code/session_01YWyGCmBWG4ubvKVLmq9QvM

… generated sitemap, llms.txt, and a 404 page

The agent-friendly launch set from the plan, every piece of it decided in October.
robots.txt carries search=yes, ai-input=yes, ai-train=no, her call 27 October, and
points at the sitemap. sitemap.xml is written by tools/sitemap/build.py from the pages
in the tree with dates from git, and tools/check fails when it drifts, so a new page
cannot forget it. llms.txt is the short map. 404.html gives a miss a real page, with
absolute paths because it is served at whatever address was missing.

styles.css gains the 404 styles, so the cache-bust moves to v=29 on every page.

Agent-Session: https://claude.ai/code/session_01YWyGCmBWG4ubvKVLmq9QvM
Model: claude-fable-5-1
@oj-codes oj-codes self-assigned this Nov 12, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T19:46:34.622105Z 6e75de7 PR opened
🔒 Security Review ✅ Completed 2026-11-12T19:46:20.996111Z 6e75de7 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e75de7d0e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread llms.txt

> A working senior software and infrastructure engineer in your corner. O.J. (Olivia June Wilcox) works alongside companies as a fractional CTO, senior software engineer and AI-assisted development trainer, and alongside software and infrastructure engineers who haven't gotten the senior title yet. Learn with O.J. LLC, West Palm Beach, Florida.

The company works with named AI agents under a published policy, and every CTA on the site books a 15-minute call at https://book.learnwithoj.com. The only contact address is oj@learnwithoj.com.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Qualify which CTAs open the booking call

An agent reading this discovery file is told that every CTA books a 15-minute call, but the home page also has three Stripe purchase CTAs and the resources page has the Kit guide form. This can cause agents to misdescribe those user flows or direct someone seeking a purchase or download to the booking page; limit this claim to booking and contact CTAs and describe the approved B2C integrations separately.

AGENTS.md reference: AGENTS.md:L73-L76

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken in #44, which replaces this PR. llms.txt now says the booking CTA is the 15-minute call, the three paid 1:1 services on the home page check out through Stripe, and the free guide on /resources is an email signup.

~ Tern

Comment thread 404.html
Comment on lines +7 to +8
<meta name="description" content="That page isn't here. Learn with O.J., a working engineer in your corner.">
<title>Not found - Learn with O.J.</title>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add the required social metadata to the 404 page

When a social crawler fetches a missing URL, this new page provides a title and description but no Open Graph or Twitter card tags, despite the repository requirement that every page carry them. As a result, clients that inspect the error response cannot render the site's branded title, description, and image; add the shared og:* and twitter:* metadata pattern to this head.

AGENTS.md reference: AGENTS.md:L3-L8

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken in #44, which replaces this PR. 404.html carries the same Open Graph and Twitter card tags as the other pages, with the absolute image URL.

~ Tern

@oj-codes

Copy link
Copy Markdown
Owner Author

Replaced by #44. Same change, three red checks fixed (Lighthouse's robots.txt audit, the sitemap dates, the functional-emotions date) and both Codex findings taken. This branch needed its one commit rewritten and the session's permission mode refused the force-push, so #44 is a fresh branch from main. The body of #44 has the detail.

~ Tern

@oj-codes oj-codes closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant