Repository navigation
Conversation
|
Common Crawl has always treated 429 as unreachable. |
|
Does "honoring" 429 Retry-After mean we have to wait 86400 even when that's obviously bogus? It's a similar situation as "honoring" crawl-delay, which all of the major crawlers second-guess. |
|
@wumpus fair point, and it matches what was reported on the list this week: many sites send 86400 when a few minutes would do. "Honoring" here should mean not coming back sooner than the server asked, not waiting for any value it sends. I'd make that explicit and let operators cap it: A published cap keeps the behaviour predictable for site owners without making it unbounded, which is more than crawl-delay gets today. The same reading covers the other end: a Happy to push that change if it works for you. |
|
+1 to Greg's comment/s. Hosts frequently emit persistent backoff signals -- specifically the What do you think about specifying that clients may evaluate these origin-provided backoff signals against historical telemetry for that specific host? And perhaps include an informative pseudocode heuristic showing how a client can discount signals when an origin sends unreasonable signals/values. Somthing like: |
|
Thanks Gary. Yes, checking against the host's history works well for the long end: a host that sends the same multi-day value on every response while latency is normal is not telling the client anything about load. Two guardrails I'd keep in the text, so that discounting stays predictable for site owners:
One small fix in the pseudocode comment: 86400 is seconds (one day), not hours. If that works for you, I'll fold your paragraph and the pseudocode into this PR with those two points and the cap wording from my earlier comment. |
|
sounds good, let's do it if Greg is ok with it too. re 86400: sorry, that was an idiotic attempt at humor on my end. I do think that the value in the illustration should be ridiculous though. maybe 606024*30 ? Basically anything that intuitively suggests on first sight that the value is in fact wrong. |
|
@wumpus does this direction work for you? Unless you'd rather keep it narrower, I'll fold Gary's paragraph and pseudocode into this PR, with the two guardrails above (discounting never goes below the client's own back-off; the operator publicly documents its cap and that it applies such a heuristic), the cap wording from my first comment, and 606024*30 as the deliberately implausible value in the example. |
Fixes #19.
Two changes, both about 429.
Why. Section 2.3 bases back-out on Section 15.6 of RFC 9110, which covers 5xx only; 429 and Retry-After don't appear in the draft. And RFC 9309 §2.3.1.3 treats any 4xx on /robots.txt as "unavailable", after which "the crawler MAY access any resources on the server". Read literally, a server answering 429 on /robots.txt has granted access to everything, the opposite of what it meant.
This reading is already shipped, and practice already diverges from it:
urllib.robotparsermaps 401/403 to disallow-all and every other 4xx, 429 included, to allow-all, citing §2.3.1.3 in the code (Lib/urllib/robotparser.py).A real case. In a GET-only census run on 19 September 2026 (data and method: https://doi.org/10.5281/zenodo.22893539), one host answered 429 on both /robots.txt and its home page. The instrument treated that as a refusal and stopped, leaving 12 of 14 planned requests unmade. A crawler following RFC 9309 to the letter could have concluded the opposite.
What the text does. Adds 429 (with Retry-After) to the back-out signals in "Crawlers MUST not interfere", and recommends that a 429 on /robots.txt be handled as unreachable (§2.3.1.4) rather than unavailable (§2.3.1.3). It's a SHOULD, so it narrows how the REP's MAY is used without contradicting RFC 9309. Adds RFC 6585 as a normative reference.
Happy to adjust the wording or move the robots.txt part elsewhere if you'd rather keep the REP section untouched.