Skip to content

Detect uptime monitors, observability tools and service agents - #674

Open
JayBizzle wants to merge 1 commit into
masterfrom
monitoring-signatures
Open

JayBizzle wants to merge 1 commit into
masterfrom
monitoring-signatures

Conversation

@JayBizzle

Copy link
Copy Markdown
Owner

Adds 46 signatures, with 51 test UAs, for automated checks that isCrawler() currently misses.

Kind Added
Uptime and synthetic monitoring Catchpoint, ThousandEyes, AlertSite, uptrends, Oh Dear, AppInsights, BinaryCanary, EdgeWatch, Hydrozen\.io, NETVIGIE, Netumo, pingping\.io, rocketmonitor, sixy\.ch, wheresitup, WebMon\/, 360JK, BBC-Forge-URL-Monitor, Server ?Density
Observability agents Prometheus\/, Grafana\/, Datadog Agent, Datadog\/Synthetics, collectd\/, munin\/, Blackbox Exporter, solarwinds
WordPress management MainWP, ManageWP, WPUmbrella
Cloudflare Cloudflare-?Diagnostics, Cloudflare-Smart-Transit, Cloudflare Custom Hostname Verification, developers\.cloudflare\.com\/security-center
Other service agents azure-logic-apps (Power Automate), Amazon Route 53 Health Check, Sectigo DCV, Valimail\/, SerpReputationManagementAgent, kiwitcms-gitops, TuneIn-DirMon, Episode Duration Lookup (Castro), Ezgif\/, phpMyAdmin, Chrome Privacy Preserving Prefetch Proxy, internal dummy connection (Apache)

Notes:

  • Catchpoint. devices.txt had a (compatible; Windows NT 6.1; Catchpoint) UA. Catchpoint is a synthetic monitoring service, so it moves to crawlers.txt.
  • Chrome's prefetch proxy fetches pages a user might open next, before they've clicked. Like Office's own visit before opening a link, it isn't someone viewing the page.
  • Placeholders. The Route 53 and Power Automate test UAs have their per-account IDs replaced with xs, since those parts change every time.
  • Left out: WebMoney Advisor, which is a browser add-on token on real Firefox UAs, the same situation as QQDownload in Explain what counts as a crawler, and stop flagging QQDownload #671. Also a few tokens too generic to match safely (HLB/, Trackable/).

Add 46 signatures for automated checks that weren't detected: uptime
and synthetic monitors (Catchpoint, ThousandEyes, AlertSite, Uptrends,
Oh Dear, Azure AppInsights and others), observability agents
(Prometheus, Grafana, Datadog, collectd, Munin, Blackbox Exporter,
SolarWinds), WordPress management services (MainWP, ManageWP,
WP Umbrella), Cloudflare's diagnostic and verification fetchers, and
other service agents such as Power Automate, Sectigo DCV, Valimail,
Chrome's privacy-preserving prefetch proxy and Apache's internal dummy
connection.

The Catchpoint UA in devices.txt moves to crawlers.txt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant