Skip to content

Detect link-preview fetchers, social bots and more Google fetchers - #676

Open
JayBizzle wants to merge 2 commits into
masterfrom
social-preview-signatures
Open

JayBizzle wants to merge 2 commits into
masterfrom
social-preview-signatures

Conversation

@JayBizzle

@JayBizzle JayBizzle commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Adds 32 signatures, with 33 test UAs, for fetchers that isCrawler() currently misses.

Kind Added
Link previews and social Bluesky Cardyb, bluesky-domain-status-classifier, Meta-ExternalHit\/, meta-externalads, Snap ?URL ?Preview, ViberUrlDownloader, kakaotalk-scrap, SteamChatURLLookup, SteamChatImageProxy, Notion\/[0-9.]+ \(https:\/\/notion\.so, ClickUpLinkUnfurler, Yahoo:LinkExpander, OpenGraph\.io, Shareaholic, FediList Agent, Friendica, Zendesk Webhook, workona-favicon-service
Mail image proxies ImageProxy\/ (Rambler), secureurl\.fwdcdn\.com (Ukr.net)
Google Google-Lens, Google-Safety, Google-CWS, Google-ilp, Google-Assess, Google-Shopping-Quality, Google-Sites-Thumbnails, Google-BusinessLinkVerification, GoogleAdSenseInfeed, Google-Cloud-Scheduler
Search NaverJapan, vuhuvRBT

Notes:

  • Link previews fall under "not a person viewing the page" in the README section from Explain what counts as a crawler, and stop flagging QQDownload #671, the same as the WhatsApp and Slack fetchers that are already detected.
  • Notion. The desktop app is Electron and sends Notion/x.y in its UA too, so the pattern requires the (https://notion.so part that only the link-preview fetcher sends. A desktop app UA is added to devices.txt to keep it that way.
  • The Bluesky test UA has Chrome/W.X.Y.Z in place of a real Chrome version, as it appears in the published string.
  • None of the 168k UAs in devices.txt match any of the new patterns.

Add 32 signatures:

- Link previews and social fetchers: Bluesky, Meta (Meta-ExternalHit,
  meta-externalads), Snapchat, Viber, KakaoTalk, Steam chat, Notion,
  ClickUp, Yahoo's link expander, OpenGraph.io, Shareaholic, FediList,
  Friendica, Zendesk webhooks, Workona's favicon service, and the
  Rambler and Ukr.net mail image proxies.
- Google fetchers that weren't covered: Google-Lens, Google-Safety,
  Google-CWS, Google-ilp, Google-Assess, Google-Shopping-Quality,
  Google-Sites-Thumbnails, Google-BusinessLinkVerification,
  GoogleAdSenseInfeed and Google-Cloud-Scheduler.
- Naver Japan and vuhuv search crawlers.
The desktop app is Electron and sends Notion/x.y in its UA, so
Notion\/[0-9] would flag people using it. Require the
(https://notion.so part that only the fetcher sends, and add a desktop
app UA to devices.txt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant