Evil bot causes host to shutdown my site

[azhrei_fje]

Basically, there's a bot coming from 85.25.x.y that generates 45,000+ hits in 90 seconds against the various forum pages (like viewtopic and viewforum). The IP appears to be a guest (ie., not a known bot or registered user).

I need a MOD that will track how many hits are coming from a given IP (logged in user or not) and start throttling future requests.

I could see a MOD that adds to the .htaccess and blocks the request entirely, perhaps removing the block after some period of time. Or maybe a MOD that sees a particular incoming IP is happening a lot and just sleeps for 3 seconds before responding; again, for some period of time.

Right now my web hosting service has shutdown my site so no one can use it. That makes this an A-1 priority for me! Unfortunately, I don't have access to the raw Apache web logs in real time, so I can't build such a list myself (using Perl or Python or something) outside of phpBB. I might be able to patch phpBB so that each request writes to a log file and then process that file instead. But it'll likely be too late for a script to add something to the .htaccess before the cpu usage spikes again.

Help!

[spaceace]

i had an issue in December of last year from "The Knowledge AI" bot putting a massive load on my hosting. it was fixed simply by blocking the bot in the .htaccess file with this

Code: Select all

###block "The Knowledge AI" bot
SetEnvIfNoCase User-Agent "The Knowledge AI" bad_bot
Deny from env=bad_bot 

[david63]

The main problem with using .htaccess is that they are still accessing the server (admittedly not the board). A better way, if your hosting account will allow it, is to block access in the firewall.

I would add that doing this within phpBB using an extension (phpBB does not use MODs any more) is that you will still have the problems that you have now with the bot hitting your board.

[azhrei_fje]

Wow, I really appreciate the super-quick response! Thanks to both of you.

i had an issue in December of last year from "The Knowledge AI" bot putting a massive load on my hosting.

Yes, I can see that the tech support for the hosting service has added similar code to the .htaccess file, but it must not have worked. I know this because (a) the site is back down again after they said they reactivated it, and (b) part of the email thread included an admin from their side saying a different IP address was now the issue. (More on that, below. Yes, I know blocking IPs isn't terribly helpful.)

The main problem with using .htaccess is that they are still accessing the server (admittedly not the board). A better way, if your hosting account will allow it, is to block access in the firewall.

Yes, the hosting service doesn't apportion Apache cpu time to individual sites, so if it takes Apache more cpu time, I don't care.

I would add that doing this within phpBB using an extension (phpBB does not use MODs any more) is that you will still have the problems that you have now with the bot hitting your board.

(Oops, my bad. I didn't realize there had been a change in the nomenclature.)

Yes, that's why I was thinking of just slowing down, a.k.a. throttling, the requests instead of trying to block them. I don't mind that they happen — I just mind that they happen so frequently. If each one had a delay of 3 seconds and I was still getting hit with 45,000+ in 90s, then whoever's hitting the site is running a lot of threads! (Well, I know it's not necessarily threaded, but it sounds good that way.)

I realize that blocking an IP address doesn't work long-term, which is why I was looking for something that would block an IP for a time and then re-enable it later. I actually think the "slow them down" approach is better because there's less chance for errors (in terms of editing the .htaccess file and potentially screwing it up).

[ivailo95]

Just talk with your host and tell them to block bad bots they know what to do.

[John connor]

Look into CIDRAM and Ninjafirewall. Ninjafirewall offers paid and free versions. The pro version is free, the pro+ version is not. CIDRAM can help in certain circumstances, but that looks like a classic DDoS and CloudFlare or your host could help. I've also written about CloudFlare on my own board. Ninjafirewall won't stop a DDoS, but CloudFlare will and it's free unless you want other services or an upgraded account.

[azhrei_fje]

Just talk with your host and tell them to block bad bots they know what to do.

Yeah, that's what I thought, too. Clearly, that hasn't been enough.

Look into CIDRAM and Ninjafirewall. [...] CloudFlare or your host could help.

Thanks. I hadn't heard of the last two, but CloudFlare was one of the suggestions that came from the host. I've got my top men working on it. (Yes, "top men".)

So far, it sounds like the consensus is that this should be taken care of outside of phpBB, and I agree. But because the host provides no transparency into the performance statistics and no access to real-time logs from the web server, I was hoping to find a solution that would involve phpBB identifying which IPs were hitting the site the hardest so they could then be blocked by the web server.

Alas, sounds like it's not to be. And I don't have the time right now to dig into the phpBB code for the purpose of creating such an extension. I wouldn't even know where to start with storing the information. (Each page request gets its own context, so the count of hits would have to be stored in a database. But then updating the database with either an INSERT or UPDATE becomes something that needs a transaction around it, so it really slows things down. Or maybe I ignore transactions — if the count is off by a little, so what? Or maybe I only insert new records and use SELECT COUNT() to find out how many there are. And then what should phpBB do? Just return the "disabled" page? Or since I'm looking to reduce cpu usage, just return a static page. Or maybe a 403 error? I can handle PHP coding, but the phpBB environment is a black box for me.)

Thanks for all of the ideas, everyone. I appreciate your time!

[John connor]

If you don't even have access logs in your FTP directory then that doesn't sound like a good host. You might want to venture on over to webhostingtalk.com and find something better.

[</Solidjeuh>]

See this


https://github.com/mitchellkrogza/apach ... ot-blocker

[azhrei_fje]

If you don't even have access logs in your FTP directory then that doesn't sound like a good host. You might want to venture on over to webhostingtalk.com and find something better.

I do have access logs, but not real-time. Since HostGator shuts off the site after 90s of cpu usage >50%, log files that are batched up into 24-hour blocks isn't very useful.

However, I will definitely be checking out the page you suggested; I'm getting pretty pissed at HostGator for the lack of response to my emails referring to this trouble ticket.

See this

https://github.com/mitchellkrogza/apach ... ot-blocker

Ooo, this looks interesting. I would still have to use the [n].htaccess[/b] version, but I will check it out. Thanks, Solid.

[John connor]

Hostgator...

I could have sworn I mentioned CIDRAM.

[</Solidjeuh>]

Ooo, this looks interesting. I would still have to use the [n].htaccess[/b] version, but I will check it out. Thanks, Solid.

I use the "robots.txt", just place that file in the forum root.

[John connor]

Ooo, this looks interesting. I would still have to use the [n].htaccess[/b] version, but I will check it out. Thanks, Solid.

I use the "robots.txt", just place that file in the forum root.

Only good bots follow that. It won't solve this issue.

[ivailo95]

talk with your host if this is not attack of hackers i mean smth floor and tell them to check it and protect ur site with ddos protect

[</Solidjeuh>]

Ooo, this looks interesting. I would still have to use the [n].htaccess[/b] version, but I will check it out. Thanks, Solid.

I use the "robots.txt", just place that file in the forum root.

Only good bots follow that. It won't solve this issue.

Then what would be the use for the bot blocker Disallow:/ function if they don't follow it ... ? Then the .htaccess would be better?