Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,18 @@ if ($CrawlerDetect->isCrawler()) {

Prefer this over `isCrawler($request->getHeaderLine('User-Agent'))`. Some crawlers — Googlebot in particular — send a genuine browser `User-Agent` and identify themselves in another header such as `From` or `Sec-CH-UA`. Passing the full set lets CrawlerDetect check all of them; passing a single string can only ever check one.

## What counts as a crawler

CrawlerDetect flags any request that isn't a person viewing the page in a browser. That covers search engines, SEO tools and scrapers, and also some apps that fetch pages automatically on someone's behalf:

- **Link previews.** WhatsApp, Slack and similar apps fetch a page to build the preview card when a link is shared.
- **Office and email clients.** Word, Excel and PowerPoint visit a link themselves before opening it in the browser, and Outlook and mail apps load images when an email is opened.
- **Feed readers, podcast apps and download tools.** These fetch on a schedule or download files rather than show a page to a reader.

These are flagged on purpose. They aren't someone reading the page, and counting them would inflate page views and click counts. If you want to treat any of them as real visitors, check `getMatches()` and allow the ones you care about.

A browser that a real person is using should never be flagged, even if an add-on or toolbar puts its name in the `User-Agent`. If you find one that is, please open an issue.

## Contributing

If you find a bot, spider or crawler that CrawlerDetect fails to detect, please open a pull request that:
Expand Down
1 change: 0 additions & 1 deletion src/Fixtures/Crawlers.php
Original file line number Diff line number Diff line change
Expand Up @@ -1046,7 +1046,6 @@ class Crawlers extends AbstractProvider
'Python-urllib',
'Python-xmlrpc',
'Qirina Hurdler',
'QQDownload',
'QrafterPro',
'Qseero',
'Qualidator',
Expand Down
3 changes: 0 additions & 3 deletions tests/data/user_agent/crawlers.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3071,9 +3071,6 @@ Mozilla/5.0 (Grabify.link - https://grabify.link/expander)
biNu image cacher (info@binu-inc.com)
Mozilla/5.0 (compatible; Google-Youtube-Links)
ping.blo.gs/2.0
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)
Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64; rv:50.1.0) Gecko/20100101 Firefox/50.1.0 Ghost Inspector
Expand Down
3 changes: 3 additions & 0 deletions tests/data/user_agent/devices.txt
Original file line number Diff line number Diff line change
Expand Up @@ -168537,3 +168537,6 @@ Mozilla/5.0 (Linux; Android 13; octopus Build/R152-16765.49.0) AppleWebKit/537.3
Mozilla/5.0 (Linux; x86_64) LibWeb+LibJS/1.0 Ladybird/1.0
Mozilla/5.0 (Linux; x86_64) Ladybird/1.0
Mozilla/5.0 (macOS; AArch64) Ladybird/1.0 Chrome/146.0.0.0 AppleWebKit/537.36 Safari/537.36
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)