CrawlerDetect is a PHP library for detecting bots, crawlers and spiders via the User-Agent and HTTP_FROM headers. It currently recognises thousands of user agents and is updated regularly.
composer require jaybizzle/crawler-detectuse Jaybizzle\CrawlerDetect\CrawlerDetect;
$CrawlerDetect = new CrawlerDetect;
// Check the user agent of the current visitor
if ($CrawlerDetect->isCrawler()) {
// true if a crawler user agent was detected
}
// Pass a user agent as a string
if ($CrawlerDetect->isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')) {
// true if a crawler user agent was detected
}
// Output the part of the user agent that matched (if any)
echo $CrawlerDetect->getMatches();With no arguments, CrawlerDetect reads $_SERVER. If your headers come from somewhere else — a PSR-7 request, Symfony's HeaderBag, Swoole, or a Lambda event — pass them in directly. Both real header names (User-Agent) and PHP's SAPI names (HTTP_USER_AGENT) are understood, and values may be strings or arrays of strings.
// PSR-7 (Slim, Mezzio, Laminas, League)
$CrawlerDetect = new CrawlerDetect($request->getHeaders());
// Symfony HttpFoundation
$CrawlerDetect = new CrawlerDetect($request->headers->all());
// Swoole
$CrawlerDetect = new CrawlerDetect($request->header);
if ($CrawlerDetect->isCrawler()) {
// ...
}Prefer this over isCrawler($request->getHeaderLine('User-Agent')). Some crawlers — Googlebot in particular — send a genuine browser User-Agent and identify themselves in another header such as From or Sec-CH-UA. Passing the full set lets CrawlerDetect check all of them; passing a single string can only ever check one.
If you find a bot, spider or crawler that CrawlerDetect fails to detect, please open a pull request that:
- adds the regex pattern to the
$dataarray insrc/Fixtures/Crawlers.php - adds the failing user agent string to
tests/data/user_agent/crawlers.txt
The raw/Crawlers.json and raw/Crawlers.txt files are regenerated automatically by export.php after merge — no need to touch them.
If you're not able to submit a PR, open an issue with the user agent string and we'll take it from there.
If you have User-Agents from real visitor traffic, they help stop new signatures flagging real browsers. Share them using the "Contribute real browser user agents" issue form; see CONTRIBUTING.md for the format.
CrawlerDetect has been ported to a number of other languages and frameworks. If you maintain a port not listed here, please open a PR.
| Platform | Project |
|---|---|
| Laravel | Laravel-Crawler-Detect |
| Node.js / ES6 | es6-crawler-detect |
| Python | crawlerdetect |
| .NET / .NET Core | NetCrawlerDetect |
| Ruby | crawler_detect |
| Go | crawlerdetect |
These ports are no longer maintained and do not receive new crawler signatures:
| Platform | Project | Last updated |
|---|---|---|
| Symfony 2 / 3 / 4 | CrawlerDetectBundle | 2021 |
| Yii2 | yii2-crawler-detect | 2016 |
| JVM (Java, Scala, Kotlin) | CrawlerDetect | 2022 |
Please don't report security issues in the public issue tracker. See SECURITY.md for how to report one privately.
Parts of this library are based on the excellent MobileDetect.
Released under the MIT License.