Evil bot causes host to shutdown my site

Looking for an Extension? Have an Extension request? Post your request here for help. (Note: This forum is community supported; while there is an Extensions Development Team, said team does not dedicate itself to handling requests in this forum)
Anti-Spam Guide
azhrei_fje
Registered User
Posts: 15
Joined: Fri Feb 13, 2009 6:28 pm

Evil bot causes host to shutdown my site

Post by azhrei_fje » Tue Jun 11, 2019 8:15 pm

Basically, there's a bot coming from 85.25.x.y that generates 45,000+ hits in 90 seconds against the various forum pages (like viewtopic and viewforum). The IP appears to be a guest (ie., not a known bot or registered user).

I need a MOD that will track how many hits are coming from a given IP (logged in user or not) and start throttling future requests.

I could see a MOD that adds to the .htaccess and blocks the request entirely, perhaps removing the block after some period of time. Or maybe a MOD that sees a particular incoming IP is happening a lot and just sleeps for 3 seconds before responding; again, for some period of time.

Right now my web hosting service has shutdown my site so no one can use it. That makes this an A-1 priority for me! Unfortunately, I don't have access to the raw Apache web logs in real time, so I can't build such a list myself (using Perl or Python or something) outside of phpBB. I might be able to patch phpBB so that each request writes to a log file and then process that file instead. But it'll likely be too late for a script to add something to the .htaccess before the cpu usage spikes again.

Help!

User avatar
spaceace
Registered User
Posts: 1857
Joined: Wed Jan 30, 2008 8:50 pm
Contact:

Re: Evil bot causes host to shutdown my site

Post by spaceace » Tue Jun 11, 2019 9:10 pm

i had an issue in December of last year from "The Knowledge AI" bot putting a massive load on my hosting. it was fixed simply by blocking the bot in the .htaccess file with this

Code: Select all

###block "The Knowledge AI" bot
SetEnvIfNoCase User-Agent "The Knowledge AI" bad_bot
Deny from env=bad_bot 

User avatar
david63
Registered User
Posts: 16165
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: Evil bot causes host to shutdown my site

Post by david63 » Tue Jun 11, 2019 9:13 pm

The main problem with using .htaccess is that they are still accessing the server (admittedly not the board). A better way, if your hosting account will allow it, is to block access in the firewall.

I would add that doing this within phpBB using an extension (phpBB does not use MODs any more) is that you will still have the problems that you have now with the bot hitting your board.
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

azhrei_fje
Registered User
Posts: 15
Joined: Fri Feb 13, 2009 6:28 pm

Re: Evil bot causes host to shutdown my site

Post by azhrei_fje » Tue Jun 11, 2019 9:41 pm

Wow, I really appreciate the super-quick response! Thanks to both of you.
spaceace wrote:
Tue Jun 11, 2019 9:10 pm
i had an issue in December of last year from "The Knowledge AI" bot putting a massive load on my hosting.
Yes, I can see that the tech support for the hosting service has added similar code to the .htaccess file, but it must not have worked. I know this because (a) the site is back down again after they said they reactivated it, and (b) part of the email thread included an admin from their side saying a different IP address was now the issue. (More on that, below. Yes, I know blocking IPs isn't terribly helpful.)
david63 wrote:
Tue Jun 11, 2019 9:13 pm
The main problem with using .htaccess is that they are still accessing the server (admittedly not the board). A better way, if your hosting account will allow it, is to block access in the firewall.
Yes, the hosting service doesn't apportion Apache cpu time to individual sites, so if it takes Apache more cpu time, I don't care. :mrgreen:
I would add that doing this within phpBB using an extension (phpBB does not use MODs any more) is that you will still have the problems that you have now with the bot hitting your board.
(Oops, my bad. I didn't realize there had been a change in the nomenclature.)

Yes, that's why I was thinking of just slowing down, a.k.a. throttling, the requests instead of trying to block them. I don't mind that they happen — I just mind that they happen so frequently. If each one had a delay of 3 seconds and I was still getting hit with 45,000+ in 90s, then whoever's hitting the site is running a lot of threads! (Well, I know it's not necessarily threaded, but it sounds good that way.)

I realize that blocking an IP address doesn't work long-term, which is why I was looking for something that would block an IP for a time and then re-enable it later. I actually think the "slow them down" approach is better because there's less chance for errors (in terms of editing the .htaccess file and potentially screwing it up).

ivailo95
Registered User
Posts: 704
Joined: Tue Sep 05, 2017 8:00 am
Location: Bulgaria
Name: Ivailo
Contact:

Re: Evil bot causes host to shutdown my site

Post by ivailo95 » Wed Jun 12, 2019 8:04 am

Just talk with your host and tell them to block bad bots they know what to do.

User avatar
John connor
Registered User
Posts: 2110
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: Evil bot causes host to shutdown my site

Post by John connor » Wed Jun 12, 2019 11:04 am

Look into CIDRAM and Ninjafirewall. Ninjafirewall offers paid and free versions. The pro version is free, the pro+ version is not. CIDRAM can help in certain circumstances, but that looks like a classic DDoS and CloudFlare or your host could help. I've also written about CloudFlare on my own board. Ninjafirewall won't stop a DDoS, but CloudFlare will and it's free unless you want other services or an upgraded account.

azhrei_fje
Registered User
Posts: 15
Joined: Fri Feb 13, 2009 6:28 pm

Re: Evil bot causes host to shutdown my site

Post by azhrei_fje » Thu Jun 13, 2019 3:08 pm

ivailo95 wrote:
Wed Jun 12, 2019 8:04 am
Just talk with your host and tell them to block bad bots they know what to do.
Yeah, that's what I thought, too. Clearly, that hasn't been enough.
John connor wrote:
Wed Jun 12, 2019 11:04 am
Look into CIDRAM and Ninjafirewall. [...] CloudFlare or your host could help.
Thanks. I hadn't heard of the last two, but CloudFlare was one of the suggestions that came from the host. I've got my top men working on it. (Yes, "top men".)

So far, it sounds like the consensus is that this should be taken care of outside of phpBB, and I agree. But because the host provides no transparency into the performance statistics and no access to real-time logs from the web server, I was hoping to find a solution that would involve phpBB identifying which IPs were hitting the site the hardest so they could then be blocked by the web server.

Alas, sounds like it's not to be. And I don't have the time right now to dig into the phpBB code for the purpose of creating such an extension. I wouldn't even know where to start with storing the information. (Each page request gets its own context, so the count of hits would have to be stored in a database. But then updating the database with either an INSERT or UPDATE becomes something that needs a transaction around it, so it really slows things down. Or maybe I ignore transactions — if the count is off by a little, so what? Or maybe I only insert new records and use SELECT COUNT() to find out how many there are. And then what should phpBB do? Just return the "disabled" page? Or since I'm looking to reduce cpu usage, just return a static page. Or maybe a 403 error? I can handle PHP coding, but the phpBB environment is a black box for me.)

Thanks for all of the ideas, everyone. I appreciate your time!

User avatar
John connor
Registered User
Posts: 2110
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: Evil bot causes host to shutdown my site

Post by John connor » Fri Jun 14, 2019 10:13 am

If you don't even have access logs in your FTP directory then that doesn't sound like a good host. You might want to venture on over to webhostingtalk.com and find something better.

User avatar
</Solidjeuh>
Registered User
Posts: 1605
Joined: Tue Mar 29, 2016 3:45 am
Location: Aalst (Belgium)
Name: Andy Dm
Contact:

Re: Evil bot causes host to shutdown my site

Post by </Solidjeuh> » Fri Jun 14, 2019 11:21 am

Register a free account & Play!!
~~~ https://www.solidjeuh.be ~~~
Have a secret? --> https://www.tellyoursecrets.eu

azhrei_fje
Registered User
Posts: 15
Joined: Fri Feb 13, 2009 6:28 pm

Re: Evil bot causes host to shutdown my site

Post by azhrei_fje » Fri Jun 14, 2019 9:19 pm

John connor wrote:
Fri Jun 14, 2019 10:13 am
If you don't even have access logs in your FTP directory then that doesn't sound like a good host. You might want to venture on over to webhostingtalk.com and find something better.
I do have access logs, but not real-time. Since HostGator shuts off the site after 90s of cpu usage >50%, log files that are batched up into 24-hour blocks isn't very useful. :evil:

However, I will definitely be checking out the page you suggested; I'm getting pretty pissed at HostGator for the lack of response to my emails referring to this trouble ticket.
Ooo, this looks interesting. I would still have to use the [n].htaccess[/b] version, but I will check it out. Thanks, Solid. :)

User avatar
John connor
Registered User
Posts: 2110
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: Evil bot causes host to shutdown my site

Post by John connor » Sat Jun 15, 2019 1:06 am

Hostgator...

I could have sworn I mentioned CIDRAM.

User avatar
</Solidjeuh>
Registered User
Posts: 1605
Joined: Tue Mar 29, 2016 3:45 am
Location: Aalst (Belgium)
Name: Andy Dm
Contact:

Re: Evil bot causes host to shutdown my site

Post by </Solidjeuh> » Mon Jun 17, 2019 1:52 am

azhrei_fje wrote:
Fri Jun 14, 2019 9:19 pm
Ooo, this looks interesting. I would still have to use the [n].htaccess[/b] version, but I will check it out. Thanks, Solid. :)

I use the "robots.txt", just place that file in the forum root.
Register a free account & Play!!
~~~ https://www.solidjeuh.be ~~~
Have a secret? --> https://www.tellyoursecrets.eu

User avatar
John connor
Registered User
Posts: 2110
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: Evil bot causes host to shutdown my site

Post by John connor » Mon Jun 17, 2019 4:17 am

</Solidjeuh> wrote:
Mon Jun 17, 2019 1:52 am
azhrei_fje wrote:
Fri Jun 14, 2019 9:19 pm
Ooo, this looks interesting. I would still have to use the [n].htaccess[/b] version, but I will check it out. Thanks, Solid. :)

I use the "robots.txt", just place that file in the forum root.
Only good bots follow that. It won't solve this issue.

ivailo95
Registered User
Posts: 704
Joined: Tue Sep 05, 2017 8:00 am
Location: Bulgaria
Name: Ivailo
Contact:

Re: Evil bot causes host to shutdown my site

Post by ivailo95 » Mon Jun 17, 2019 10:55 am

talk with your host if this is not attack of hackers i mean smth floor and tell them to check it and protect ur site with ddos protect

User avatar
</Solidjeuh>
Registered User
Posts: 1605
Joined: Tue Mar 29, 2016 3:45 am
Location: Aalst (Belgium)
Name: Andy Dm
Contact:

Re: Evil bot causes host to shutdown my site

Post by </Solidjeuh> » Mon Jun 17, 2019 11:35 am

John connor wrote:
Mon Jun 17, 2019 4:17 am
</Solidjeuh> wrote:
Mon Jun 17, 2019 1:52 am
azhrei_fje wrote:
Fri Jun 14, 2019 9:19 pm
Ooo, this looks interesting. I would still have to use the [n].htaccess[/b] version, but I will check it out. Thanks, Solid. :)

I use the "robots.txt", just place that file in the forum root.
Only good bots follow that. It won't solve this issue.
Then what would be the use for the bot blocker Disallow:/ function if they don't follow it ... ? Then the .htaccess would be better?
Register a free account & Play!!
~~~ https://www.solidjeuh.be ~~~
Have a secret? --> https://www.tellyoursecrets.eu

Post Reply

Return to “Extension Requests”