About the bots

Get help with installation and running phpBB 3.3.x here. Please do not post bug reports, feature requests, or extension related questions here.
bidouille
Registered User
Posts: 17
Joined: Fri Sep 15, 2006 3:27 pm
Location: France

About the bots

Post by bidouille »

Hello folks,

Over the past few months, we've had a lot of visits from Bytedance / Bytespider (TikTok's bot).
It generates up to 600 guest connections and saturates our forum.

How does phpBB treat these bots compared to a normal user?
For example, does a robot's crawling generate fewer requests to the database than a user?
Last edited by Mick on Fri Feb 09, 2024 10:11 am, edited 1 time in total.
Reason: Solved.
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26849
Joined: Fri Aug 29, 2008 9:49 am

Re: About the bots

Post by Mick »

If the bot/spider is correctly entered in the ‘Spiders/Bots’ list phpBB removes certain unnecessary elements from the page that you wouldn't want to index such as profile links, search page etc. If you want to make a section of your board available to guests but do not want bots indexing those pages you can set the permissions to no access for the bots etc.
  • "The more connected we get the more alone we become” - Kyle Broflovski© 🇬🇧
bidouille
Registered User
Posts: 17
Joined: Fri Sep 15, 2006 3:27 pm
Location: France

Re: About the bots

Post by bidouille »

Merci beaucoup Mick
ukautoforums
Registered User
Posts: 101
Joined: Thu Mar 02, 2017 10:00 am

Re: About the bots

Post by ukautoforums »

I had the same, basically killing the forums. I ended up blocking a load in htaccess file:

BrowserMatchNoCase "libwww-perl" bad_bot
BrowserMatchNoCase "wget" bad_bot
BrowserMatchNoCase "LieBaoFast" bad_bot
BrowserMatchNoCase "Mb2345Browser" bad_bot
BrowserMatchNoCase "zh-CN" bad_bot
BrowserMatchNoCase "MicroMessenger" bad_bot
BrowserMatchNoCase "zh_CN" bad_bot
BrowserMatchNoCase "Kinza" bad_bot
BrowserMatchNoCase "Bytespider" bad_bot
BrowserMatchNoCase "Baiduspider" bad_bot
BrowserMatchNoCase "Sogou" bad_bot
BrowserMatchNoCase "Datanyze" bad_bot
BrowserMatchNoCase "AspiegelBot" bad_bot
BrowserMatchNoCase "adscanner" bad_bot
BrowserMatchNoCase "serpstatbot" bad_bot
BrowserMatchNoCase "spaziodat" bad_bot
BrowserMatchNoCase "undefined" bad_bot
BrowserMatchNoCase "PetalBot" bad_bot
BrowserMatchNoCase "trendictionbot" bad_bot
BrowserMatchNoCase "SEMrushbot" bad_bot
BrowserMatchNoCase "AhrefsBot" bad_bot
Order Deny,Allow
Deny from env=bad_bot

Return to “[3.3.x] Support Forum”