From e240ea9305d1b7b0a2541ef9e5b4d5b285bd6cb6 Mon Sep 17 00:00:00 2001 From: frmoretto <47558125+frmoretto@users.noreply.github.com> Date: Sat, 26 Sep 2026 13:14:29 +0200 Subject: [PATCH] Cover 429 in back-out logic and in robots.txt fetches --- draft-illyes-webbotauth-cbcp.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/draft-illyes-webbotauth-cbcp.md b/draft-illyes-webbotauth-cbcp.md index 4f5f442..ccc56c0 100644 --- a/draft-illyes-webbotauth-cbcp.md +++ b/draft-illyes-webbotauth-cbcp.md @@ -37,6 +37,7 @@ normative: HTTP-CACHING: rfc9111 HTTP-LINK: rfc8288 HTTP-SEMANTICS: rfc9110 + HTTP-429: rfc6585 JAFAR: I-D.draft-illyes-aipref-jafar REP: rfc9309 @@ -106,6 +107,13 @@ Especially if the website chooses not to use a robots.txt file as defined by the REP, crawlers further MUST respect the `X-robots-tag` in HTTP response headers from the website. +{{Section 2.3.1.3 of REP}} treats a robots.txt response in the 400-499 range as +unavailable, in which case the crawler MAY access any resource on the server. A +429 (Too Many Requests) response ({{Section 4 of HTTP-429}}) is a request to +slow down, not a statement about the file. Crawlers SHOULD NOT treat a 429 +response to a robots.txt fetch as unavailable, and SHOULD instead handle it as +unreachable ({{Section 2.3.1.4 of REP}}) until a later fetch succeeds. + ## Crawlers MUST be easily identifiable through their user agent string @@ -133,8 +141,10 @@ Depending on a site's setup (computing resources and software efficiency) and its size, crawling may slow down the site or even take it offline altogether. Crawler operators MUST ensure that their crawlers are equipped with back-out logic that relies on at least the standard signals defined by -{{Section 15.6 of HTTP-SEMANTICS}}, preferably also additional heuristics such -as a change in the relative response time of the server. +{{Section 15.6 of HTTP-SEMANTICS}} and the 429 (Too Many Requests) status code +defined in {{Section 4 of HTTP-429}}, honoring the `Retry-After` header field +({{Section 10.2.3 of HTTP-SEMANTICS}}) when present, preferably also additional +heuristics such as a change in the relative response time of the server. Therefore, crawlers SHOULD log already visited URLs, the number of requests sent to each resource, and the respective HTTP status codes in the responses,