Skip to content

Detect AI agents, assistants and training crawlers - #672

Open
JayBizzle wants to merge 1 commit into
masterfrom
ai-agent-signatures
Open

JayBizzle wants to merge 1 commit into
masterfrom
ai-agent-signatures

Conversation

@JayBizzle

Copy link
Copy Markdown
Owner

Adds 27 signatures for AI tools that isCrawler() currently misses, with 32 test UAs.

Kind Added
Agents and assistants fetching for a user Amzn-User, AmazonBuyForMe, amazon-QBusiness, Gemini-Deep-Research, GoogleAgent-URLContext, Google-NotebookLM, Google-Pinpoint, Devin\/, Manus-User, Character-AI, GitHubCopilotChat, Trae-Agent, TwinAgent, UseAI\/, bigsur\.ai
Training and research crawlers anthropic-ai, Anthropic-Claude, cohere-ai, Cohere-Command, LAIONDownloader, laion-huggingface-processor, SalamandraVLM, UdV-AI-Research
Search and retrieval Cloudflare-AutoRAG, amazon-kendra, Anomura
Headless browser for agents Lightpanda

A few choices worth checking:

  • Devin\/ and UseAI\/ need the slash, because Devin is a common first name.
  • Gemini is only matched as Gemini-Deep-Research. devices.txt has Gemini Pro phones and the Gemini iOS app (GeminiiOS/…), which must stay undetected.
  • Manus. devices.txt had a …Chrome/147.0.0.0 Safari/537.36; Manus-User/1.0 line, taken from real visitor traffic. This moves it to crawlers.txt. Manus is an AI agent that can drive the user's own browser, so the visit may come from a real account, but the agent is the one browsing. That fits "not a person viewing the page" in the README section from Explain what counts as a crawler, and stop flagging QQDownload #671. If you'd rather treat agent-driven browsing as human, drop Manus-User and the line goes back.
  • Left out: Google-Extended, which is a robots.txt token and never sent as a UA (see Some runtimes and automation tools aren't detected #647). Also Awario, IAS and YaK, which aren't AI tools; they'll go in a later signatures PR.

Add 27 signatures for AI tools that weren't detected:

- Agents and assistants fetching pages for a user: Amazon (Amzn-User,
  AmazonBuyForMe, amazon-QBusiness), Gemini Deep Research,
  GoogleAgent-URLContext, Google NotebookLM and Pinpoint, Devin,
  Manus, Character.AI, GitHub Copilot Chat, Trae, TwinAgent, UseAI,
  bigsur.ai.
- Training and research crawlers: Anthropic's older tokens
  (anthropic-ai, Anthropic-Claude), Cohere, LAION, SalamandraVLM,
  UdV-AI-Research.
- Search and retrieval: Cloudflare AutoRAG, Amazon Kendra, Anomura.
- Lightpanda, a headless browser built for AI agents and scraping.

Devin and UseAI are matched with a trailing slash because Devin is a
common name. Gemini is matched only as Gemini-Deep-Research, since
phones and the Gemini iOS app also carry the name.

The Manus-User UA in devices.txt moves to crawlers.txt: Manus is an
agent doing the browsing, not a person viewing the page.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant