About the bots

Get help with installation and running phpBB 3.3.x here. Please do not post bug reports, feature requests, or extension related questions here.
Post Reply
bidouille
Registered User
Posts: 17
Joined: Fri Sep 15, 2006 3:27 pm
Location: France

About the bots

Post by bidouille »

Hello folks,

Over the past few months, we've had a lot of visits from Bytedance / Bytespider (TikTok's bot).
It generates up to 600 guest connections and saturates our forum.

How does phpBB treat these bots compared to a normal user?
For example, does a robot's crawling generate fewer requests to the database than a user?
Last edited by Mick on Fri Feb 09, 2024 10:11 am, edited 1 time in total.
Reason: Solved.
User avatar
Mick
Support Team Member
Support Team Member
Posts: 26387
Joined: Fri Aug 29, 2008 9:49 am

Re: About the bots

Post by Mick »

If the bot/spider is correctly entered in the ‘Spiders/Bots’ list phpBB removes certain unnecessary elements from the page that you wouldn't want to index such as profile links, search page etc. If you want to make a section of your board available to guests but do not want bots indexing those pages you can set the permissions to no access for the bots etc.
  • "The more connected we get the more alone we become" - Kyle Broflovski©
  • "The good news is hell is just the product of a morbid human imagination.
    The bad news is, whatever humans can imagine, they can usually create.
    " - Harmony Cobel
bidouille
Registered User
Posts: 17
Joined: Fri Sep 15, 2006 3:27 pm
Location: France

Re: About the bots

Post by bidouille »

Merci beaucoup Mick
ukautoforums
Registered User
Posts: 82
Joined: Thu Mar 02, 2017 10:00 am

Re: About the bots

Post by ukautoforums »

I had the same, basically killing the forums. I ended up blocking a load in htaccess file:

BrowserMatchNoCase "libwww-perl" bad_bot
BrowserMatchNoCase "wget" bad_bot
BrowserMatchNoCase "LieBaoFast" bad_bot
BrowserMatchNoCase "Mb2345Browser" bad_bot
BrowserMatchNoCase "zh-CN" bad_bot
BrowserMatchNoCase "MicroMessenger" bad_bot
BrowserMatchNoCase "zh_CN" bad_bot
BrowserMatchNoCase "Kinza" bad_bot
BrowserMatchNoCase "Bytespider" bad_bot
BrowserMatchNoCase "Baiduspider" bad_bot
BrowserMatchNoCase "Sogou" bad_bot
BrowserMatchNoCase "Datanyze" bad_bot
BrowserMatchNoCase "AspiegelBot" bad_bot
BrowserMatchNoCase "adscanner" bad_bot
BrowserMatchNoCase "serpstatbot" bad_bot
BrowserMatchNoCase "spaziodat" bad_bot
BrowserMatchNoCase "undefined" bad_bot
BrowserMatchNoCase "PetalBot" bad_bot
BrowserMatchNoCase "trendictionbot" bad_bot
BrowserMatchNoCase "SEMrushbot" bad_bot
BrowserMatchNoCase "AhrefsBot" bad_bot
Order Deny,Allow
Deny from env=bad_bot
Post Reply

Return to “[3.3.x] Support Forum”