Skip to content

Explain what counts as a crawler, and stop flagging QQDownload - #671

Open
JayBizzle wants to merge 1 commit into
masterfrom
what-counts-as-a-crawler
Open

JayBizzle wants to merge 1 commit into
masterfrom
what-counts-as-a-crawler

Conversation

@JayBizzle

@JayBizzle JayBizzle commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Issues like #430 and #361 ask why apps such as Word or Outlook show up as crawlers. This adds a README section to answer that: CrawlerDetect flags anything that isn't a person viewing the page in a browser. That covers:

  • link previews (WhatsApp, Slack);
  • Office apps visiting a link before opening it, and mail clients loading images;
  • feed readers, podcast apps and download tools.

Those are flagged on purpose, since counting them would inflate page views and click counts. The section points to getMatches() for anyone who wants to let some of them through.

It also says a browser a real person is using should never be flagged. QQDownload broke that rule. It's an Internet Explorer add-on that puts its name in the browser's User-Agent, for example:

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)

So every match was a person browsing. This removes the pattern and moves its three UAs from crawlers.txt to devices.txt. No other pattern picks them up.

Behaviour change: IE user agents carrying a QQDownload token are no longer reported as crawlers.

Add a README section saying what CrawlerDetect flags: anything that
isn't a person viewing the page in a browser, including link previews,
Office and email clients, feed readers and download tools. Those are
flagged on purpose, because they'd otherwise inflate views and clicks.

QQDownload doesn't fit that. It's an Internet Explorer add-on that puts
its name in the browser's User-Agent, so every match was a person
browsing. Remove the pattern and move its three UAs to devices.txt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant