From e2e16ad54c14a3394487fc3a1b11682b98db83be Mon Sep 17 00:00:00 2001 From: e271828- Date: Tue, 6 Oct 2026 14:47:57 -0400 Subject: [PATCH 1/2] Prefer use of distinct subdomains for different crawl types --- draft-illyes-webbotauth-cbcp.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/draft-illyes-webbotauth-cbcp.md b/draft-illyes-webbotauth-cbcp.md index 555627c..ebe5d7b 100644 --- a/draft-illyes-webbotauth-cbcp.md +++ b/draft-illyes-webbotauth-cbcp.md @@ -133,6 +133,14 @@ in the case-insensitive `User-Agent` header, such as clearly identify both the crawler owner and its purpose as much as reasonably possible. +Crawler operators SHOULD use separate domains or subdomains to identify +crawlers with different purposes, and separate keys when cryptographic +authentication is used. For example, a search indexing crawler and an AI +training crawler could use `https://search.example.com` and +`https://training.example.com`, respectively, with a separate key for each. +This allows website operators to distinguish each kind of crawler and +reliably apply crawling policies independently. + ## Do not interfere with the normal operation of a site From 1ae2ff70e605f3d79da65bd6b7e40716172770e1 Mon Sep 17 00:00:00 2001 From: e271828- Date: Mon, 12 Oct 2026 11:03:16 -0400 Subject: [PATCH 2/2] Identify crawler purposes with distinct Signature-Agent URLs --- draft-illyes-webbotauth-cbcp.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/draft-illyes-webbotauth-cbcp.md b/draft-illyes-webbotauth-cbcp.md index ebe5d7b..a13c42c 100644 --- a/draft-illyes-webbotauth-cbcp.md +++ b/draft-illyes-webbotauth-cbcp.md @@ -37,6 +37,7 @@ normative: HTTP-CACHING: rfc9111 HTTP-LINK: rfc8288 HTTP-SEMANTICS: rfc9110 + HTTPSIG-PROTOCOL: I-D.ietf-webbotauth-httpsig-protocol JAFAR: I-D.draft-illyes-aipref-jafar REP: rfc9309 @@ -133,13 +134,11 @@ in the case-insensitive `User-Agent` header, such as clearly identify both the crawler owner and its purpose as much as reasonably possible. -Crawler operators SHOULD use separate domains or subdomains to identify -crawlers with different purposes, and separate keys when cryptographic -authentication is used. For example, a search indexing crawler and an AI -training crawler could use `https://search.example.com` and -`https://training.example.com`, respectively, with a separate key for each. -This allows website operators to distinguish each kind of crawler and -reliably apply crawling policies independently. +Crawler operators that use cryptographic authentication SHOULD use a distinct +`Signature-Agent` URL, and therefore a distinct key directory, for each crawler +purpose, as described in {{Section 6.5 of HTTPSIG-PROTOCOL}}. This is the +cryptographic counterpart of the purpose identification in the `User-Agent` +header. ## Do not interfere with the normal operation of a site