Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions draft-illyes-webbotauth-cbcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,8 @@ All well behaved-crawlers MUST support the REP as defined in
{{Section 2.2.1 of REP}} to allow site owners to opt out from crawling.

Especially if the website chooses not to use a robots.txt file as defined
by the REP, crawlers further need to respect the `X-robots-tag` in the HTTP
header.
by the REP, crawlers further MUST respect the `X-robots-tag` in HTTP
response headers from the website.


## Crawlers MUST be easily identifiable through their user agent string
Expand Down
Loading