Skip to content

Cover 429 in back-out logic and in robots.txt fetches - #22

Open
frmoretto wants to merge 1 commit into
garyillyes:mainfrom
frmoretto:429-backoff-robots
Open

frmoretto wants to merge 1 commit into
garyillyes:mainfrom
frmoretto:429-backoff-robots

Conversation

@frmoretto

Copy link
Copy Markdown

Fixes #19.

Two changes, both about 429.

Why. Section 2.3 bases back-out on Section 15.6 of RFC 9110, which covers 5xx only; 429 and Retry-After don't appear in the draft. And RFC 9309 §2.3.1.3 treats any 4xx on /robots.txt as "unavailable", after which "the crawler MAY access any resources on the server". Read literally, a server answering 429 on /robots.txt has granted access to everything, the opposite of what it meant.

This reading is already shipped, and practice already diverges from it:

  • CPython's standard-library urllib.robotparser maps 401/403 to disallow-all and every other 4xx, 429 included, to allow-all, citing §2.3.1.3 in the code (Lib/urllib/robotparser.py).
  • Google's documentation carves it out: "Google's crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist" (robots.txt spec).

A real case. In a GET-only census run on 19 September 2026 (data and method: https://doi.org/10.5281/zenodo.22893539), one host answered 429 on both /robots.txt and its home page. The instrument treated that as a refusal and stopped, leaving 12 of 14 planned requests unmade. A crawler following RFC 9309 to the letter could have concluded the opposite.

What the text does. Adds 429 (with Retry-After) to the back-out signals in "Crawlers MUST not interfere", and recommends that a 429 on /robots.txt be handled as unreachable (§2.3.1.4) rather than unavailable (§2.3.1.3). It's a SHOULD, so it narrows how the REP's MAY is used without contradicting RFC 9309. Adds RFC 6585 as a normative reference.

Happy to adjust the wording or move the robots.txt part elsewhere if you'd rather keep the REP section untouched.

@wumpus

wumpus commented Oct 6, 2026

Copy link
Copy Markdown

Common Crawl has always treated 429 as unreachable.

@wumpus

wumpus commented Oct 10, 2026

Copy link
Copy Markdown

Does "honoring" 429 Retry-After mean we have to wait 86400 even when that's obviously bogus?

It's a similar situation as "honoring" crawl-delay, which all of the major crawlers second-guess.

@frmoretto

Copy link
Copy Markdown
Author

@wumpus fair point, and it matches what was reported on the list this week: many sites send 86400 when a few minutes would do.

"Honoring" here should mean not coming back sooner than the server asked, not waiting for any value it sends.

I'd make that explicit and let operators cap it:
"...honoring the Retry-After header field ({{Section 10.2.3 of HTTP-SEMANTICS}}) when present, up to a maximum that the crawler operator documents publicly..."

A published cap keeps the behaviour predictable for site owners without making it unbounded, which is more than crawl-delay gets today.

The same reading covers the other end: a Retry-After: 0 doesn't override the crawler's own back-off.

Happy to push that change if it works for you.

@garyillyes

Copy link
Copy Markdown
Owner

+1 to Greg's comment/s.

Hosts frequently emit persistent backoff signals -- specifically the retry-after HTTP response header field or non-standard crawl-delay / retry-after rules in robots.txt as Greg suggested. The latter one is crazy unreliable, I haven't looked at retry-after.

What do you think about specifying that clients may evaluate these origin-provided backoff signals against historical telemetry for that specific host? And perhaps include an informative pseudocode heuristic showing how a client can discount signals when an origin sends unreasonable signals/values. Somthing like:

Origins and reverse proxies sometimes emit explicit backoff signals -- such as the `retry-after` HTTP field (Section 10.2.3 of [HTTP-SEMANTICS]) or non-standard `retry-after` rules in `robots.txt` ([REP]). Clients MAY evaluate the credibility of such advisory signals against historical observations for that specific host. A client MAY discount or ignore backoff directives from a host when historical telemetry indicates persistent misconfiguration (for example, emitting an identical, multi-day `retry-after` duration on every response regardless of request rate or server latency).

The following pseudocode illustrates a non-normative heuristic for evaluating whether to honor an origin's `retry-after` signal based on historical host behavior:

~~~pseudocode
function evaluate_backoff_signal(host, proposed_delay, current_response):
    # Hard ceiling: reject absurdly long static delays (e.g., > 86400 hours)
    if proposed_delay > MAX_REASONABLE_DELAY:
        record_unreliable_signal(host, proposed_delay)
        return DEFAULT_BACKOFF

    history = get_host_telemetry(host)

    # Check if the host emits the exact same delay value on every response
    # while other load indicators (TTFB, 5xx rate) remain completely nominal
    if history.signal_variance(proposed_delay) == 0 and history.is_latency_nominal():
        host.trust_score = max(0.0, host.trust_score - TRUST_PENALTY)
    else if current_response.status in [429, 503]:
        # Signal correlates with actual overload status; restore trust
        host.trust_score = min(1.0, host.trust_score + TRUST_RECOVERY)

    if host.trust_score < TRUST_THRESHOLD:
        # Signal is historically unreliable; fall back to congestion control
        return calculate_dynamic_backoff(history.baseline_latency)

    return proposed_delay * host.trust_score
~~~

@frmoretto

Copy link
Copy Markdown
Author

Thanks Gary. Yes, checking against the host's history works well for the long end: a host that sends the same multi-day value on every response while latency is normal is not telling the client anything about load.

Two guardrails I'd keep in the text, so that discounting stays predictable for site owners:

  1. Discounting only shortens an unreasonable delay down to the client's own back-off, never below it.
    In the pseudocode the last line would become something like return max(proposed_delay * host.trust_score, calculate_dynamic_backoff(history.baseline_latency)), so a low-trust signal falls back to congestion control, not to an early retry.
  2. The client documents publicly the maximum it honours and that it applies this kind of heuristic, as in my earlier suggestion. Otherwise a site owner can't tell a discounted Retry-After from one that was ignored.

One small fix in the pseudocode comment: 86400 is seconds (one day), not hours.

If that works for you, I'll fold your paragraph and the pseudocode into this PR with those two points and the cap wording from my earlier comment.

@garyillyes

Copy link
Copy Markdown
Owner

sounds good, let's do it if Greg is ok with it too.

re 86400: sorry, that was an idiotic attempt at humor on my end. I do think that the value in the illustration should be ridiculous though. maybe 606024*30 ? Basically anything that intuitively suggests on first sight that the value is in fact wrong.

@frmoretto

Copy link
Copy Markdown
Author

@wumpus does this direction work for you? Unless you'd rather keep it narrower, I'll fold Gary's paragraph and pseudocode into this PR, with the two guardrails above (discounting never goes below the client's own back-off; the operator publicly documents its cap and that it applies such a heuristic), the cap wording from my first comment, and 606024*30 as the deliberately implausible value in the example.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add HTTP 429 in the rate limiting section

3 participants