bugfix: XF2.3 compatibility for sending new bots via an email attachment

new bots discovered in June 2021

• btbot: BT Bot
• catchbot: Catchbot
• comodospider: Comodo SSL Spider
• deepnoc: deepnoc bot (network optimized crawling)
• dispenserbot: Dispenser Dab Solver Checklinks Bot
• epicbot: Epictions EpicBot
• esperanzabot: EsperanzaBot
• fast enterprise crawler: Fast enterprise crawler 6 used by Schibsted
• fleabot: Mercadopar Fleabot
• fuseonbot: Fuseon Link Affinity Bot
• google/bot: google/bot
• greenflare seo crawler: Greenflare SEO Crawler
• gsitecrawler: GSiteCrawler
• holmes: Morfeo Holmes Bot
• infotigerbot: InfoTiger Search Engine Bot
• internet security survey bot: Internet Security Survey Bot
• jetpack-bot: JetPack Bot
• mojoo robot: Mojoo Bot
• nicecrawler: NiceCrawler
• prem.moe crawler: Prem.moe Crawler
• sayindexbot: SayIndex Bot
• sbl-bot: SoftByte Labs Bot
• siteliner: Siteliner Bot
• swjschecketbot: Swjschecketbot
• trade desk ads.txt & sellers.json crawler: Trade Desk ads.txt & sellers.json crawler
• vortex: Marty Anstey Vortex Bot
• www.hlabs.co.ke: hLabs Bot
• zspider: Red Kolibri ZSpider

new false positives:
spider v9 phone
new bots:
80legs.com: 80legs Crawler
adssellerscrawler: Ads Sellers Crawler
apesearch crawler: ApeSearch Crawler
aportcatalogrobot: AportCatalogRobot
bot dns-cache.fr: bot dns-cache.fr
crusty broad web crawler: Crusty Broad Web Crawler
docomo: NTT Docomo Goo Bot
dubbotbot: DubBot Bot
electricmonk: DueDil Electric Monk Crawler
finbot: Finbot
flok's crawler: Flok's Crawler
goodbot: GoodBot
grover: Grover web crawler
joc web spider: JOC Web Spider
linkarchiver twitter bot: LinkArchiver Twitter Bot
linkscrawler: LinksCrawler
neevabot: Neevabot Search Engine Bot
newsharecounts.com: Newshare Counts
newslitbot: Newslit Bot
opensearch@mpdl: OpenSearch@MPDL
orbbot: orbbot
pajbot1: Pajbot
richaudience brandsafety bot: Rich Audience Brandsafety Bot
semjibot: SemjiBot
simplecrawler/0.1: SimpleCrawler
snap url preview service: Snap URL Preview Service
spider_bot/3.0: Spider_bot
spotibobot: Spotibo bot
synologychatbot: Synology Chat Bot
terrawizbot: TerrawizBot
top100.rambler.ru crawler: Top-100 Rambler Crawler
webshopchecker bot: Webshop Checker Bot
wi job roboter spider: Web Integration Job Robot
woobot: WooBot
woriobot: Zite.com WorioBot
wp fastest cache preload bot: WP Fastest Cache Preload Bot
xenforo: XenForo
your bot: Your Bot
yunsecuritybot: YunSecurityBot
yurichevbot: Yurichev Bot

new false positives:

• cubot; j5
• baiduboxapp

new bots:

• 200pleasebot
• a8bot
• abilogicbot
• acoonbot
• adform robot
• arhpostbot
• atomseobot
• awariorendererbot
• badoobot
• bl.uk_ldfc_bot
• brobot
• charityengine bot
• charlotte
• cosmos
• coveobot
• crawlbot/1.0.0
• cxensebot
• facebot
• fandomopengraphbot
• freshpingbot
• fuelbot
• geedobot
• getlocalbot
• google-safety
• gpcsupbot
• grub-client
• gynxbot
• healrworld crawler
• hgfalphaxcrawl
• hoodle crawler
• idmarch automatic
• imrbot
• jambot
• justlocal.nl
• kantarsifomediaauditbot
• keobsbot
• keybasebot
• koepabot
• lanaibot
• landsbokasafn
• lapozzbot
• linkpulse metacrawler
• linksmanager.com_bot
• lxrbot
• mbot v
• moreoverbot
• netpeakspiderbot
• www.niraiya.com
• node/simplecrawler
• nu.marginalia.wmsa.edge-crawler
• nutchcvs
• oer commons bot
• omniexplorer_bot
• onefuncbot
• oozbot
• pickybot
• piepmatz bot
• plukkie
• pu_in crawler
• punkspider
• pwa-crawler
• reasonalbot
• revuebot
• runet-research-crawler
• screenerbot crawler
• searchenginecrawler
• sebot-wa
• seekbot
• shopwiki
• showyoubot
• siteauditbot
• sitescorebot
• spinn3r
• squirrobot
• ssl-crawler
• thinkbot
• tsmbot
• tweetedtimes bot
• ucrawl
• umichbot
• urlappendbot
• verticalleap-sitestatusbot
• webgraph
• weblinkchecker
• websquash.com
• wellknownbot
• wizenozebot

new false positives:

• cubot r11
• spider v7 build/lmy47i
• spider v7 (MyCell Spider v7 from Bangladesh)

weekly bot updates:

• adbot/1.0
• ahrefssiteaudit
• amazonadbot
• amg-bot
• ampxfbot
• anybot
• aranea web-crawled corpora project
• backlinkcrawler
• botelaire
• chirpyhubbot
• cookiebot
• crawl/1.0
• dataforseobot
• diffeobot
• digitalshadowsbot
• discoverbot
• dmasslinksafetybot
• domcopbot
• echocrawl
• emeraldshield.com webbot
• historyspider
• iplogger crawler
• mohawk crawler
• mtrobot
• mxbot
• netresearchserver
• openindexspider
• pinllc search robot
• rustbot
• sellers.guide crawler by primis
• snaplocalbot
• test-deep-cocrawler
• vdo.ai bot
• wesee:search
• whizebot
• xaldon_webspider
• xovibot
• yaanibot
• yelpbot

weekly new bots:

• boitho.com-dc
• msc crawl project radboud university
• niocbot
• open web analytics bot
• quetextbot
• rc-crawler
• tokenspider
• womlpefactory
• yeti/1.0

• new Dinobot Android TV false positives:
  • dinobot 4k plus
• weekly new bots:
  • cis455crawler
  • crystalsemanticsbot
  • discoverspider
  • envolk[its]spider
  • geograph linkcheck bot
  • gg peekbot
  • iccrawler
  • mybot
  • psbot
  • suggybot
  • testbot

weekly new bots:

• antbot/1.0
• becomebot
• castlebot
• custom-crawler
• feedsearch bot
• gulper web bot
• istellabot
• lmspider
• marketingminer bot
• my nutch spider
• networking4all bot
• nfwebcrawler
• tombapublicwebcrawler
• uptimebot
• vuhuvbot
• webcrawl.net
• webspider 1.0

new Cubot phone false positives:

• cubot a5
• cubot j3 pro

weekly new bots:

• addsugarspiderbot
• annuairefrancais.fr
• coibotparser
• criteobot
• discobot
• domainspider-bot
• dropboxpreviewbot
• dumbot
• gnowitnewsbot
• hoaxybot
• ichiro
• lamarkbot
• media-bot
• nusearch spider
• radian6_default_
• xovionpagecrawler
• zyborg

weekly new bots:

• experiancrawluk
• impact radius compliance bot
• kauaibot
• leuchtfeuer crawler
• redirectbot
• simpleanalyticsbot
• skimbot
• sogou pic spider
• testcrawler