Repository navigation
Conversation
Add 74 signatures: - Crawlers and research projects, including BuiltWith, Sitebulb's mobile UA, The Trade Desk, Scamadviser, Library and Archives Canada, the Library of Congress, MaCoCu, and university network measurement scanners. - Download managers and scrapers: Download Master, Download Ninja, HiDownload, iGetter, colly, puf and others. - HTTP libraries and API clients: Java's HttpClient, PHP-SOAP, REST::Client, cpprestsdk, babashka, Insomnia, Apidog, OpenWrt's uclient-fetch, jsdom. - Runtimes sending their default UA: Deno, Playwright's request context, and a bare "node". These are anchored to the start (and the end for node) so the names can't match inside other UAs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #647.
Adds 74 signatures, with 77 test UAs.
BW\/[0-9.]+; bit\.ly\/(BuiltWith),sitebulb\.com,TTD-Content,ScamadviserExternalHit,MarketGoo,Morningscore,seolyt,TagInspector,Functionize,Inspici,pageburst,ArtelLeadRadar,PostitLeadDiscovery,ProspectorBR,ImportDomains,KenyaDomainScorer,domainreanimator,domain research project,FinderGo\/,FormFinder\/,Kaupr\/,Dormouse\/,LetrixLabs,LinkBloom,LumeWebScan,PhishBucketRust,Secweb-Sectxt,ORTc\.me,sqalix,xtate\/,Choosito,DepSpid,MuscatFerret,^ds9 [0-9],Simbiat Software,Rainmeter WebParser,scan@aegis\.network,project_patchwatchLAC_IAHarvester,Library Of Congress Web Archiving,Authory\/,MaCoCu,CopyvioDetector,Do Not Track Verifier(EFF),UCSBNetworkMeasurement,researchscan\.comsys,Research JLU,Sparkler\/Download Master,Download Ninja,HiDownload,iGetter\/,Kapere,MFHttpScan,UtilMind HTTPGet,Web Image Collector,^DA [0-9],colly\/[0-9],puf\/[0-9]Java-http-client\/,PHP-SOAP\/,REST::Client\/,cpprestsdk\/,babashka\.http-client,httprs\/,perlclient\/,HTTP Retriever,uclient-fetch,^insomnia\/,Apidog\/,jsdom\/^Deno\/,^Playwright\/,^node$Notes:
DenoandPlaywrightare anchored to the start of the UA, andnodematches only when it's the whole UA, so none of these names can match inside something else. The test UAs for these three are the ones from the issue.Google-Extended, also listed in Some runtimes and automation tools aren't detected #647, is a robots.txt token and never appears in a UA, so there's nothing to add for it.Sitebulb\/. This addssitebulb\.comfor its mobile UA, which only names the tool in its URL.devices.txtmatch any of the new patterns.