From 59b0757bc3f8d03a7cff0357df87e2402583ca84 Mon Sep 17 00:00:00 2001 From: Mark Beech Date: Mon, 16 Nov 2026 22:28:08 +0000 Subject: [PATCH] Explain what counts as a crawler, and stop flagging QQDownload Add a README section saying what CrawlerDetect flags: anything that isn't a person viewing the page in a browser, including link previews, Office and email clients, feed readers and download tools. Those are flagged on purpose, because they'd otherwise inflate views and clicks. QQDownload doesn't fit that. It's an Internet Explorer add-on that puts its name in the browser's User-Agent, so every match was a person browsing. Remove the pattern and move its three UAs to devices.txt. --- README.md | 12 ++++++++++++ src/Fixtures/Crawlers.php | 1 - tests/data/user_agent/crawlers.txt | 3 --- tests/data/user_agent/devices.txt | 3 +++ 4 files changed, 15 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 946df4df..86eeb925 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,18 @@ if ($CrawlerDetect->isCrawler()) { Prefer this over `isCrawler($request->getHeaderLine('User-Agent'))`. Some crawlers — Googlebot in particular — send a genuine browser `User-Agent` and identify themselves in another header such as `From` or `Sec-CH-UA`. Passing the full set lets CrawlerDetect check all of them; passing a single string can only ever check one. +## What counts as a crawler + +CrawlerDetect flags any request that isn't a person viewing the page in a browser. That covers search engines, SEO tools and scrapers, and also some apps that fetch pages automatically on someone's behalf: + +- **Link previews.** WhatsApp, Slack and similar apps fetch a page to build the preview card when a link is shared. +- **Office and email clients.** Word, Excel and PowerPoint visit a link themselves before opening it in the browser, and Outlook and mail apps load images when an email is opened. +- **Feed readers, podcast apps and download tools.** These fetch on a schedule or download files rather than show a page to a reader. + +These are flagged on purpose. They aren't someone reading the page, and counting them would inflate page views and click counts. If you want to treat any of them as real visitors, check `getMatches()` and allow the ones you care about. + +A browser that a real person is using should never be flagged, even if an add-on or toolbar puts its name in the `User-Agent`. If you find one that is, please open an issue. + ## Contributing If you find a bot, spider or crawler that CrawlerDetect fails to detect, please open a pull request that: diff --git a/src/Fixtures/Crawlers.php b/src/Fixtures/Crawlers.php index 97cb4ea3..405950a5 100644 --- a/src/Fixtures/Crawlers.php +++ b/src/Fixtures/Crawlers.php @@ -1046,7 +1046,6 @@ class Crawlers extends AbstractProvider 'Python-urllib', 'Python-xmlrpc', 'Qirina Hurdler', - 'QQDownload', 'QrafterPro', 'Qseero', 'Qualidator', diff --git a/tests/data/user_agent/crawlers.txt b/tests/data/user_agent/crawlers.txt index 211fe7f4..306515c2 100644 --- a/tests/data/user_agent/crawlers.txt +++ b/tests/data/user_agent/crawlers.txt @@ -3071,9 +3071,6 @@ Mozilla/5.0 (Grabify.link - https://grabify.link/expander) biNu image cacher (info@binu-inc.com) Mozilla/5.0 (compatible; Google-Youtube-Links) ping.blo.gs/2.0 -Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727) -Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322) -Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322) Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector Mozilla/5.0 (X11; Linux x86_64; rv:50.1.0) Gecko/20100101 Firefox/50.1.0 Ghost Inspector diff --git a/tests/data/user_agent/devices.txt b/tests/data/user_agent/devices.txt index a0d96071..cc31e2aa 100644 --- a/tests/data/user_agent/devices.txt +++ b/tests/data/user_agent/devices.txt @@ -168537,3 +168537,6 @@ Mozilla/5.0 (Linux; Android 13; octopus Build/R152-16765.49.0) AppleWebKit/537.3 Mozilla/5.0 (Linux; x86_64) LibWeb+LibJS/1.0 Ladybird/1.0 Mozilla/5.0 (Linux; x86_64) Ladybird/1.0 Mozilla/5.0 (macOS; AArch64) Ladybird/1.0 Chrome/146.0.0.0 AppleWebKit/537.36 Safari/537.36 +Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727) +Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322) +Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)