Skip to content

Detect more crawlers, download managers, HTTP clients and runtimes - #677

Open
JayBizzle wants to merge 1 commit into
masterfrom
crawler-tool-signatures
Open

JayBizzle wants to merge 1 commit into
masterfrom
crawler-tool-signatures

Conversation

@JayBizzle

Copy link
Copy Markdown
Owner

Closes #647.

Adds 74 signatures, with 77 test UAs.

Kind Added
Crawlers and lead or domain tools BW\/[0-9.]+; bit\.ly\/ (BuiltWith), sitebulb\.com, TTD-Content, ScamadviserExternalHit, MarketGoo, Morningscore, seolyt, TagInspector, Functionize, Inspici, pageburst, ArtelLeadRadar, PostitLeadDiscovery, ProspectorBR, ImportDomains, KenyaDomainScorer, domainreanimator, domain research project, FinderGo\/, FormFinder\/, Kaupr\/, Dormouse\/, LetrixLabs, LinkBloom, LumeWebScan, PhishBucketRust, Secweb-Sectxt, ORTc\.me, sqalix, xtate\/, Choosito, DepSpid, MuscatFerret, ^ds9 [0-9], Simbiat Software, Rainmeter WebParser, scan@aegis\.network, project_patchwatch
Research and archiving LAC_IAHarvester, Library Of Congress Web Archiving, Authory\/, MaCoCu, CopyvioDetector, Do Not Track Verifier (EFF), UCSBNetworkMeasurement, researchscan\.comsys, Research JLU, Sparkler\/
Download managers and scrapers Download Master, Download Ninja, HiDownload, iGetter\/, Kapere, MFHttpScan, UtilMind HTTPGet, Web Image Collector, ^DA [0-9], colly\/[0-9], puf\/[0-9]
HTTP libraries and API clients Java-http-client\/, PHP-SOAP\/, REST::Client\/, cpprestsdk\/, babashka\.http-client, httprs\/, perlclient\/, HTTP Retriever, uclient-fetch, ^insomnia\/, Apidog\/, jsdom\/
Runtimes (#647) ^Deno\/, ^Playwright\/, ^node$

Notes:

  • Runtimes. Bun was covered in Add signatures for 15 undetected bots #659. Deno and Playwright are anchored to the start of the UA, and node matches only when it's the whole UA, so none of these names can match inside something else. The test UAs for these three are the ones from the issue. Google-Extended, also listed in Some runtimes and automation tools aren't detected #647, is a robots.txt token and never appears in a UA, so there's nothing to add for it.
  • Sitebulb already has Sitebulb\/. This adds sitebulb\.com for its mobile UA, which only names the tool in its URL.
  • No device false positives. None of the 168k UAs in devices.txt match any of the new patterns.

Add 74 signatures:

- Crawlers and research projects, including BuiltWith, Sitebulb's
  mobile UA, The Trade Desk, Scamadviser, Library and Archives Canada,
  the Library of Congress, MaCoCu, and university network measurement
  scanners.
- Download managers and scrapers: Download Master, Download Ninja,
  HiDownload, iGetter, colly, puf and others.
- HTTP libraries and API clients: Java's HttpClient, PHP-SOAP,
  REST::Client, cpprestsdk, babashka, Insomnia, Apidog, OpenWrt's
  uclient-fetch, jsdom.
- Runtimes sending their default UA: Deno, Playwright's request
  context, and a bare "node". These are anchored to the start (and the
  end for node) so the names can't match inside other UAs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Some runtimes and automation tools aren't detected

1 participant