Bots eating up all the bandwidth

[supanet]

I look after a website that is getting a lot of bots, everytime i visit the site it has over 200 bots online at any one time.

The site is allocated 100GB of Bandwidth each month and every month it is used up (sometimes making the site go down)

The site has been going for over 10 years and used to be very busy but now only gets a handful of members on each day (less than 10) so i know it's not that.

The site is run through Cloudflare (free version) but with only basic settings as i am not sure how to set things up on it.

It used to be a lot of chinese bots but lately it is more of these:
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Lots of random IP's so hard to ban.

Any suggestions on how i can reduce the bandwidth usage?

[KevC]

Already a couple of topics on this as others have seen the same with other bots

viewtopic.php?t=2652265
viewtopic.php?t=2654353

[supanet]

Already a couple of topics on this as others have seen the same with other bots

viewtopic.php?t=2652265
viewtopic.php?t=2654353

Thanks will take a look.

[supanet]

Ok just in case anyone has a similar problem this one worked for me.
viewtopic.php?p=16018150#p16018150

Code: Select all

BrowserMatchNoCase "libwww-perl" bad_bot
BrowserMatchNoCase "wget" bad_bot
BrowserMatchNoCase "LieBaoFast" bad_bot
BrowserMatchNoCase "Mb2345Browser" bad_bot
BrowserMatchNoCase "zh-CN" bad_bot
BrowserMatchNoCase "MicroMessenger" bad_bot
BrowserMatchNoCase "zh_CN" bad_bot
BrowserMatchNoCase "Kinza" bad_bot
BrowserMatchNoCase "Bytespider" bad_bot
BrowserMatchNoCase "Baiduspider" bad_bot
BrowserMatchNoCase "Sogou" bad_bot
BrowserMatchNoCase "Datanyze" bad_bot
BrowserMatchNoCase "AspiegelBot" bad_bot
BrowserMatchNoCase "adscanner" bad_bot
BrowserMatchNoCase "serpstatbot" bad_bot
BrowserMatchNoCase "spaziodat" bad_bot
BrowserMatchNoCase "undefined" bad_bot
BrowserMatchNoCase "claudebot" bad_bot
BrowserMatchNoCase "facebook" bad_bot
BrowserMatchNoCase "Petalbot" bad_bot
BrowserMatchNoCase "YandexBot" bad_bot
BrowserMatchNoCase "Applebot" bad_bot
BrowserMatchNoCase "aiohttp" bad_bot
Order Deny,Allow
Deny from env=bad_bot
Deny from 47.76.0.0/16

Add this to your .htaccess file

[thecoalman]

It easier and more effective to block them in Cloudlfare's firewall. Go to security section >> WAF and then click the link for custom rules.

You can add a rule for user agents, you add multiple user agents to single rule using OR. e.g if useragent contains zyz OR abc block the request.

As far as the countries if you expect most of your traffic from a single country or a few countries instead of trying to block countries whitelist ones you expect your legitimate traffic. e.g. if a country does equal US or CA then use a JS Challenge. Of course you would use whatever countries you expect legitimate traffic. The JS challenge does not outright block traffic but it's difficult for bots to get through. Legitimate traffic should be fairly seamless with little inconvenience.

[supanet]

Thanks will take a look +1

[HB]

It easier and more effective to block them in Cloudlfare's firewall. Go to security section >> WAF and then click the link for custom rules.

If you want a really blunt axe, go to WAF > Rate limiting rules. Then create a rule like this:

(not cf.bot_management.verified_bot and http.request.uri.path contains ".php") or (cf.bot_management.verified_bot and http.request.uri.path contains ".php")

The free tier only allows one rate limiting rule. If you set the limit to 10 requests per second/10 second timeout, you effectively limit any bot (or person) to less than one page request per second. You can tweak the maximum number of requests as your server capacity allows with a minimum of 1 page every 10 seconds.

[Forex Station]

I look after a website that is getting a lot of bots, everytime i visit the site it has over 200 bots online at any one time.

The site is allocated 100GB of Bandwidth each month and every month it is used up (sometimes making the site go down)

The site has been going for over 10 years and used to be very busy but now only gets a handful of members on each day (less than 10) so i know it's not that.

The site is run through Cloudflare (free version) but with only basic settings as i am not sure how to set things up on it.

It used to be a lot of chinese bots but lately it is more of these:
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Lots of random IP's so hard to ban.

Any suggestions on how i can reduce the bandwidth usage?

The .htaccess guide above will solve your issues and if you really want to stop the bots just go into your Cloudflare or .htaccess and temporary block Facebook's facebookexternalhit/1.1 bot.

Is it only the Facebook bot that's eating up your bandwidth? The ones that smash our site are Ahrefs and Semrush which take up almost the same as Google bot at times.

[supanet]

Thanks for all the advice guys, I have now reduced them down from over 200+ to the odd 7 or 8 this i can handle.

Once again great community.

[ssl]

A little personal feedback on these invasive bots.
I do not use Cloudflare and my .htacces file has not been modified, just this addition to the robots.txt file got rid of these invasions

Code: Select all

# Disallow Bad Bot
User-Agent: Claudebot
User-agent: anthropic-ai
User-Agent: ByteDance
User-agent: Bytespider
User-agent: ChatGPT-User
User-agent: GPTBot
User-Agent: FriendlyCrawler
User-agent: Google-Extended
User-agent: Omgili
Disallow: /

[thecoalman]

Usually if they have a user agent they will obey robots.txt but not always. Side note, Cloudflare has a new tool that specifically targets AI bots soaking up pages for AI learning. One click.


https://blog.cloudflare.com/declaring-y ... ngle-click

[Terceirense]

I used to run a forum with similar issues. Here’s what worked for me:

Added CAPTCHA: I put CAPTCHA on sign-ups and comments. It helped cut down on bot activity.
Tweaked Cloudflare: I went into Cloudflare’s settings and set up rules to challenge or block suspicious traffic. You can do this under Firewall > Tools > IP Access Rules.
Updated .htaccess: I added some rules to block certain bots and IP ranges. If you’re on WordPress, there are plugins for this too.

[SpIdErPiGgY]

https://github.com/ai-robots-txt/ai.rob ... robots.txt

https://github.com/mitchellkrogza/nginx ... robots.txt

[fuolo]

Ok just in case anyone has a similar problem this one worked for me.
viewtopic.php?p=16018150#p16018150

Code: Select all

BrowserMatchNoCase "libwww-perl" bad_bot
BrowserMatchNoCase "wget" bad_bot
BrowserMatchNoCase "LieBaoFast" bad_bot
BrowserMatchNoCase "Mb2345Browser" bad_bot
BrowserMatchNoCase "zh-CN" bad_bot
BrowserMatchNoCase "MicroMessenger" bad_bot
BrowserMatchNoCase "zh_CN" bad_bot
BrowserMatchNoCase "Kinza" bad_bot
BrowserMatchNoCase "Bytespider" bad_bot
BrowserMatchNoCase "Baiduspider" bad_bot
BrowserMatchNoCase "Sogou" bad_bot
BrowserMatchNoCase "Datanyze" bad_bot
BrowserMatchNoCase "AspiegelBot" bad_bot
BrowserMatchNoCase "adscanner" bad_bot
BrowserMatchNoCase "serpstatbot" bad_bot
BrowserMatchNoCase "spaziodat" bad_bot
BrowserMatchNoCase "undefined" bad_bot
BrowserMatchNoCase "claudebot" bad_bot
BrowserMatchNoCase "facebook" bad_bot
BrowserMatchNoCase "Petalbot" bad_bot
BrowserMatchNoCase "YandexBot" bad_bot
BrowserMatchNoCase "Applebot" bad_bot
BrowserMatchNoCase "aiohttp" bad_bot
Order Deny,Allow
Deny from env=bad_bot
Deny from 47.76.0.0/16

Add this to your .htaccess file

won't this have negative impact on SEO if the crawlers are blocked?

[danieltj]

won't this have negative impact on SEO if the crawlers are blocked?

Yes and no. Obviously, if you block a bot and they honour it then they will not index your forum but the bots that are being blocked here aren't necessarily 'good bots' that you would want to allow access to your forum in the first place.

It's important to make sensible decisions when it comes to bots and their access. Blocking Google will prevent your forum from being index in the future so if Google is visiting your site too much, you might need to look at other methods over basic banning so to speak.