BLEXBOT crawler causing extremely high load on the server

Luchtzak · Post by **Luchtzak** » Fri Aug 30, 2019 9:09 am

Yesterday evening, I started having a problem with our server. Which is, in normal circumstances (even to 10K pageviews per hour) working like a charm.

There was extreme load on the server, about 30 on average. I checked the server's CPU, that was constantly running up to the 100%

Checking my forum stats, I noticed that yesterday we had beaten the most users ever online record: 5023 on 21 Aug 2019 15:56

I continued to look into the stats and found one particular crawler called BLEXBOT was causing the high load, opening more than 800 sessions.

Blocking the crawler from robots.txt didn't suffice as on another site I discovered that the crawler doesn't listen to the order. I contacted the company (webmeup) via mail and Facebook without a reply.

I decided to block a series of IP-addresses from the forums, adding this to the IP block list: 159.138.*.*

Server load and CPU has significantly dropped after the measure.

Post by **KevC** » Fri Aug 30, 2019 9:30 am

That won't actually stop it coming to the server. Ask you hosts about blocking it at the server level.

Luchtzak · Post by **Luchtzak** » Fri Aug 30, 2019 2:31 pm

KevC wrote: ↑
Fri Aug 30, 2019 9:30 am
That won't actually stop it coming to the server. Ask you hosts about blocking it at the server level.

Well, it currently did work.

david63 · Post by **david63** » Fri Aug 30, 2019 2:39 pm

Luchtzak wrote: ↑
Fri Aug 30, 2019 2:31 pm

KevC wrote: ↑
Fri Aug 30, 2019 9:30 am
That won't actually stop it coming to the server. Ask you hosts about blocking it at the server level.
Well, it currently did work.

It might have stopped it crawling your site but it has not stopped it accessing your site - it has to access the site to read the robots.txt file. Also bear in mind that some bots do not adhere to robots.txt, or change their behavior

Forex Station · Post by **Forex Station** » Fri Aug 30, 2019 2:52 pm

Block it in your htaccess file. Rogue bots like this don't obey Robots.txt.

david63 · Post by **david63** » Fri Aug 30, 2019 3:11 pm

Forex Station wrote: ↑
Fri Aug 30, 2019 2:52 pm
Block it in your htaccess file. Rogue bots like this don't obey Robots.txt.

Even in .htaccess the bots are still accessing your site. They need to be blocked before then - either by using a firewall or the likes of Cloudflare

Luchtzak · Post by **Luchtzak** » Tue Sep 03, 2019 4:39 pm

david63 wrote: ↑
Fri Aug 30, 2019 2:39 pm

Luchtzak wrote: ↑
Fri Aug 30, 2019 2:31 pm

KevC wrote: ↑
Fri Aug 30, 2019 9:30 am
That won't actually stop it coming to the server. Ask you hosts about blocking it at the server level.
Well, it currently did work.
It might have stopped it crawling your site but it has not stopped it accessing your site - it has to access the site to read the robots.txt file. Also bear in mind that some bots do not adhere to robots.txt, or change their behavior

Behavior indeed changed, server's CPU is constantly full, server load higher than usual. Must be from other IP's ...

Any suggestion ?

david63 · Post by **david63** » Tue Sep 03, 2019 5:50 pm

Luchtzak wrote: ↑
Tue Sep 03, 2019 4:39 pm
Any suggestion ?

Check your server access logs and block any suspicious behaviour in your firewall - works for me

advertisements