From 066465bba12ecd7d2cc71a20ce6924cbdbb560ac Mon Sep 17 00:00:00 2001 From: Chris Needham Date: Fri, 7 Aug 2026 19:29:51 +0100 Subject: [PATCH] Strengthen X-robots-tag requirement --- draft-illyes-webbotauth-cbcp.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft-illyes-webbotauth-cbcp.md b/draft-illyes-webbotauth-cbcp.md index 0b98217..7b8be04 100644 --- a/draft-illyes-webbotauth-cbcp.md +++ b/draft-illyes-webbotauth-cbcp.md @@ -103,8 +103,8 @@ All well behaved-crawlers MUST support the REP as defined in {{Section 2.2.1 of REP}} to allow site owners to opt out from crawling. Especially if the website chooses not to use a robots.txt file as defined -by the REP, crawlers further need to respect the `X-robots-tag` in the HTTP -header. +by the REP, crawlers further MUST respect the `X-robots-tag` in HTTP +response headers from the website. ## Crawlers MUST be easily identifiable through their user agent string