Skip to content

Stop flagging apps and phones whose names contain a crawler word - #670

Open
JayBizzle wants to merge 1 commit into
masterfrom
substring-false-positives
Open

JayBizzle wants to merge 1 commit into
masterfrom
substring-false-positives

Conversation

@JayBizzle

Copy link
Copy Markdown
Owner

Part of #639.

35 real device and app UAs were flagged as crawlers because part of a model or app name hit a pattern:

UAs Name Matched
22 Cronet (Chrome's network stack, named in the UA by Android and iOS apps such as Google Go and TikTok) cron in the catch-all
5 VR BOT 552, XBot_Senior, M_bot_tab_… phones bot
3 Speedy phones and app speedy
1 each Banca Caboto IE build, TIANYU-KTOUCH, DESAY TS1008/1.0, AmigaVoyager, Yandex.Music bot, NYU, 008\/, voyager\/, Yandex

Fix

  • Each name goes into the device and app section of Exclusions.php, the same way cubot is handled. They're plain literals, so nothing changes for the ports.
  • The speedy pattern is removed rather than excluded. Every Speedy Spider UA in crawlers.txt also contains "spider", so the catch-all still detects all eight. A narrower Speedy ?Spider pattern would trip the collision test for the same reason.
  • All 35 UAs go into devices.txt. They're all flagged on master and none are with this change. Every line in crawlers.txt is still detected.

This doesn't settle the broader #639 question about the catch-all matching inside words. It only fixes the cases seen in real UAs.

35 real device and app UAs were flagged because a substring of a model
or app name hit a pattern:

- Cronet, Chrome's network stack, which Android and iOS apps such as
  Google Go and TikTok name in their UA, hit "cron" in the catch-all.
- VR BOT 552, XBot_Senior and M_bot_tab phones hit "bot".
- Banca Caboto (a bank's IE build) hit "bot", TIANYU-KTOUCH hit NYU,
  DESAY TS1008/ hit 008/ (80legs), AmigaVoyager hit voyager/, and
  Yandex.Music hit Yandex.

Add each name to the device and app section of Exclusions, the same
way cubot is handled, and add the 35 UAs to devices.txt.

Also remove the speedy pattern, which flagged a Speedy phone and an app
of that name. Every Speedy Spider UA also contains "spider", so the
catch-all still detects all of them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant