diff --git a/README.md b/README.md index 946df4d..86eeb92 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,18 @@ if ($CrawlerDetect->isCrawler()) { Prefer this over `isCrawler($request->getHeaderLine('User-Agent'))`. Some crawlers — Googlebot in particular — send a genuine browser `User-Agent` and identify themselves in another header such as `From` or `Sec-CH-UA`. Passing the full set lets CrawlerDetect check all of them; passing a single string can only ever check one. +## What counts as a crawler + +CrawlerDetect flags any request that isn't a person viewing the page in a browser. That covers search engines, SEO tools and scrapers, and also some apps that fetch pages automatically on someone's behalf: + +- **Link previews.** WhatsApp, Slack and similar apps fetch a page to build the preview card when a link is shared. +- **Office and email clients.** Word, Excel and PowerPoint visit a link themselves before opening it in the browser, and Outlook and mail apps load images when an email is opened. +- **Feed readers, podcast apps and download tools.** These fetch on a schedule or download files rather than show a page to a reader. + +These are flagged on purpose. They aren't someone reading the page, and counting them would inflate page views and click counts. If you want to treat any of them as real visitors, check `getMatches()` and allow the ones you care about. + +A browser that a real person is using should never be flagged, even if an add-on or toolbar puts its name in the `User-Agent`. If you find one that is, please open an issue. + ## Contributing If you find a bot, spider or crawler that CrawlerDetect fails to detect, please open a pull request that: diff --git a/src/Fixtures/Crawlers.php b/src/Fixtures/Crawlers.php index 97cb4ea..405950a 100644 --- a/src/Fixtures/Crawlers.php +++ b/src/Fixtures/Crawlers.php @@ -1046,7 +1046,6 @@ class Crawlers extends AbstractProvider 'Python-urllib', 'Python-xmlrpc', 'Qirina Hurdler', - 'QQDownload', 'QrafterPro', 'Qseero', 'Qualidator', diff --git a/tests/data/user_agent/crawlers.txt b/tests/data/user_agent/crawlers.txt index 211fe7f..306515c 100644 --- a/tests/data/user_agent/crawlers.txt +++ b/tests/data/user_agent/crawlers.txt @@ -3071,9 +3071,6 @@ Mozilla/5.0 (Grabify.link - https://grabify.link/expander) biNu image cacher (info@binu-inc.com) Mozilla/5.0 (compatible; Google-Youtube-Links) ping.blo.gs/2.0 -Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727) -Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322) -Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322) Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector Mozilla/5.0 (X11; Linux x86_64; rv:50.1.0) Gecko/20100101 Firefox/50.1.0 Ghost Inspector diff --git a/tests/data/user_agent/devices.txt b/tests/data/user_agent/devices.txt index a0d9607..cc31e2a 100644 --- a/tests/data/user_agent/devices.txt +++ b/tests/data/user_agent/devices.txt @@ -168537,3 +168537,6 @@ Mozilla/5.0 (Linux; Android 13; octopus Build/R152-16765.49.0) AppleWebKit/537.3 Mozilla/5.0 (Linux; x86_64) LibWeb+LibJS/1.0 Ladybird/1.0 Mozilla/5.0 (Linux; x86_64) Ladybird/1.0 Mozilla/5.0 (macOS; AArch64) Ladybird/1.0 Chrome/146.0.0.0 AppleWebKit/537.36 Safari/537.36 +Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727) +Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322) +Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)