From c698b9bdd548ee6467bf7d9c69535f93d93435b8 Mon Sep 17 00:00:00 2001 From: Tern Date: Fri, 13 Nov 2026 10:19:44 -0500 Subject: [PATCH 1/3] agents can read the site: robots.txt with her Content Signals line, a generated sitemap, llms.txt, and a 404 page The agent-friendly launch set from the plan, every piece of it decided in October. robots.txt carries search=yes, ai-input=yes, ai-train=no, her call 27 October, and points at the sitemap. sitemap.xml is written by tools/sitemap/build.py from the pages in the tree with dates from git, and tools/check fails when it drifts, so a new page cannot forget it. llms.txt is the short map. 404.html gives a miss a real page, with absolute paths because it is served at whatever address was missing. styles.css gains the 404 styles, so the cache-bust moves to v=29 on the three pages whose text changes anyway. functional-emotions.html stays at v=28, which costs it nothing (the new styles are only for the 404 page) and keeps its "Last updated" date true, since the living-pages check counts any edit as a change. Lighthouse's robots-txt audit is skipped in lighthouserc.json. It calls the Content-Signal line an unknown directive and takes the SEO score to 0.92, below the 0.95 gate. RFC 9309 says a crawler ignores lines it does not understand, and the line is there on purpose, so the audit is the thing that is wrong here. tools/sitemap/build.py dates a page with uncommitted changes as today, because that is the date git gives it once the commit exists. The first push of this branch had a sitemap generated before its own commit, so CI saw stale dates for the pages the commit touched. Agent-Session: https://claude.ai/code/session_01YWyGCmBWG4ubvKVLmq9QvM Model: claude-fable-5-1 --- 404.html | 80 ++++++++++++++++++++++++++++++++++++++++ CHANGELOG.md | 1 + CLAUDE.md | 4 ++ index.html | 2 +- lighthouserc.json | 3 ++ llms.txt | 19 ++++++++++ lychee.toml | 4 ++ resources.html | 2 +- robots.txt | 16 ++++++++ showcase.html | 2 +- sitemap.xml | 59 ++++++++++++++++++++++++++++++ styles.css | 72 ++++++++++++++++++++++++++++++++++++ tools/check | 11 +++++- tools/sitemap/build.py | 83 ++++++++++++++++++++++++++++++++++++++++++ 14 files changed, 353 insertions(+), 5 deletions(-) create mode 100644 404.html create mode 100644 llms.txt create mode 100644 robots.txt create mode 100644 sitemap.xml create mode 100644 tools/sitemap/build.py diff --git a/404.html b/404.html new file mode 100644 index 0000000..f23090c --- /dev/null +++ b/404.html @@ -0,0 +1,80 @@ + + + + + + + + Not found - Learn with O.J. + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+
+
+ +

That page isn't here

+

Nothing lives at this address. It may have moved when the site did, or the link was typed from memory, which is how most of us type links.

+ +

If a link on this site brought you here, tell me which one and I'll fix it. For once, it wasn't DNS.

+
+
+
+ + + + diff --git a/CHANGELOG.md b/CHANGELOG.md index 57d8c1b..5dcc40f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ The format is inspired by [Keep a Changelog](https://keepachangelog.com), and th ## [Unreleased] ### Added +- The agent-friendly launch set. `robots.txt` carries the Content Signals line she chose on 27 October (`search=yes, ai-input=yes, ai-train=no`), `sitemap.xml` is generated from the tree by `tools/sitemap/build.py` and checked by `tools/check`, `llms.txt` maps the site for agents, and `404.html` gives a miss a real page in the site's style - `SECURITY.md` with private vulnerability reporting, current site boundaries and security review guidance ### Changed - `AGENTS.md` clarifies generated updates, decorative image alt text, approved purchase integrations and the evidence expected in reviews and PR descriptions diff --git a/CLAUDE.md b/CLAUDE.md index 50af3db..933f4db 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -65,6 +65,10 @@ showcase.html B2B referral page (post-event landing) resources.html Free resources page functional-emotions.html The Desperation Dial explainer, first of a series. Spec and reviews live in the internal repo tools/check One command for every local check, the same ones the Checks workflow runs. Run it before pushing +tools/sitemap/ Writes sitemap.xml from the pages in the tree, dated from git. tools/check fails when it drifts +404.html The not-found page. Absolute paths only, since it is served at whatever address was missing +robots.txt Content Signals line (search=yes, ai-input=yes, ai-train=no, her call 27 Oct 2026) and the sitemap pointer +llms.txt The short agent-readable map of the site tools/last-reviewed/ Derives the "Last updated" date on living pages from git, run by the Checks workflow on PRs tools/policy-pii/ PII and name gate over the generated agents/ pages. Allowlist only, copied from the internal repo. No blocklist here, this repo is public functions/ Cloudflare Pages Functions. /api/health today, the Stripe webhook and signed downloads later diff --git a/index.html b/index.html index 8b539ab..6454024 100644 --- a/index.html +++ b/index.html @@ -51,7 +51,7 @@ - +