Evil bot causes host to shutdown my site

Looking for an Extension? Have an Extension request? Post your request here for help. (Note: This forum is community supported; while there is an Extensions Development Team, said team does not dedicate itself to handling requests in this forum)
Scam Warning
User avatar
canonknipser
Registered User
Posts: 1995
Joined: Thu Sep 08, 2011 4:16 am
Location: Germany
Name: Frank Jakobs
Contact:

Re: Evil bot causes host to shutdown my site

Post by canonknipser » Mon Jun 17, 2019 3:20 pm

Disallow is for the good bots to tell them not to index certain files or directories. It a decision made by the bot (better: the bot's programmer) to follow the instruction. Bad bots don't respect the robots.txt, so you need to block them on web server level at the latest, which is done eg. by .htaccess
It's the same as putting a sign "Private road, no entry" - some people respect, some not. Those who don't respect, need harder actions to stop them
Greetings, Frank
phpbb.de support team member
English is not my native language - no support via PM or mail
New arrival - Extensions and scripts for phpBB

User avatar
John connor
Registered User
Posts: 2118
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: Evil bot causes host to shutdown my site

Post by John connor » Tue Jun 18, 2019 2:13 am

</Solidjeuh> wrote:
Mon Jun 17, 2019 11:35 am
John connor wrote:
Mon Jun 17, 2019 4:17 am
</Solidjeuh> wrote:
Mon Jun 17, 2019 1:52 am
azhrei_fje wrote:
Fri Jun 14, 2019 9:19 pm
Ooo, this looks interesting. I would still have to use the [n].htaccess[/b] version, but I will check it out. Thanks, Solid. :)

I use the "robots.txt", just place that file in the forum root.
Only good bots follow that. It won't solve this issue.
Then what would be the use for the bot blocker Disallow:/ function if they don't follow it ... ? Then the .htaccess would be better?


No, the htaccess file can only handle so much. If you use it to start blocking massive amounts of IPs it will slow your site down and you'll start to get errors. You're better off with something that is coded in PHP like CIDRAM that can handle tons of IP blocks and still allow your site to load fast.

https://github.com/CIDRAM/CIDRAM

https://gitter.im/CIDRAM/Lobby

CIDRAM also has a WordPress plugin. The author of this script is a good friend of mine and I have offered several suggestions for it and submit ASNs that need to be blocked.

If this is a DDoS or layer 7 DDoS, you'll need to use something like CloudFlare.

The bot file is followed by the good bots liie Google, Bing, Yahoo, etc. Just because you say don't follow this directory doesn't mean they have to honor it. Also, CIDRAM has bot verification so one can't change their useragent to Google bot and probe your site, etc.

azhrei_fje
Registered User
Posts: 15
Joined: Fri Feb 13, 2009 6:28 pm

Re: Evil bot causes host to shutdown my site

Post by azhrei_fje » Tue Jun 18, 2019 3:06 pm

John connor wrote:
Tue Jun 18, 2019 2:13 am
No, the htaccess file can only handle so much. If you use it to start blocking massive amounts of IPs it will slow your site down and you'll start to get errors. You're better off with something that is coded in PHP like CIDRAM that can handle tons of IP blocks and still allow your site to load fast.

https://github.com/CIDRAM/CIDRAM
https://gitter.im/CIDRAM/Lobby

CIDRAM also has a WordPress plugin. The author of this script is a good friend of mine and I have offered several suggestions for it and submit ASNs that need to be blocked.

If this is a DDoS or layer 7 DDoS, you'll need to use something like CloudFlare.
I've been reading the docs for CIDRAM. Very cool! Thanks for the info!

We will put it in our WP site using the plugin, then I'll see about using the same code with a different config file for phpBB. We've installed a couple of other WP plugins as well (SuperCache and Wordfence).

We're looking at CloudFlare as well.

In the short term, we seem to have solved the cpu issue, but we're still looking for a new hosting service that will be a little more transparent about this type of issue. From a post on WebHostingTalk.com, it seems that our $20/mo budget should be able to get us a shared server and maybe some better monitoring capabilities.
The bot file is followed by the good bots liie Google, Bing, Yahoo, etc. Just because you say don't follow this directory doesn't mean they have to honor it. Also, CIDRAM has bot verification so one can't change their useragent to Google bot and probe your site, etc.
Actually, I learned that Google does NOT use the robots.txt — but they DO use the contents of some of the meta links in the <head> block.

User avatar
John connor
Registered User
Posts: 2118
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: Evil bot causes host to shutdown my site

Post by John connor » Tue Jun 18, 2019 3:25 pm

You're welcome. If you go CloudFlare, be sure to read my post on that. If you go to the link in my Sig, there's a follow up post on CloudFlare there. Let me know if you have any questions. If you want layer 7 DDoS protection you'll have to pay CloudFlare. I do and it's about 30 cents a month, though, I don't have a lot of traffic either unfortunately. Since my forum is mainly on computers, etc, I have a lot of competition. :lol:
Actually, I learned that Google does NOT use the robots.txt — but they DO use the contents of some of the meta links in the <head> block.
Where did you read that? With me, all indications lead to Google actually honoring what's in robots.txt. I know becasue they've hit certain areas of the forum and I then disallowed it in robots and they never came back. Also, I block access to the renamed CIDRAM folder from being indexed and my Google search doesn't show Google indexing that folder. Now Bing is another story. Despite me saying don't go here or there, they do anyway and get a 404. Sometimes I feel like blocking all of Bing's ASNs but if it weren't for the fact that a lot of my visitors using Bing I would have blocked them long ago. Don't get too many Yahoo bot visits. Can't really say if they honor bots like they should or not.


If you have any questions about CIDRAM, go to the Gitter link I gave a link to. Caleb is the Dev. If there is an issue, open and issue at the Github repository.

Use this site with your WordPress domain. https://hackertarget.com/wordpress-security-scan/

User avatar
</Solidjeuh>
Registered User
Posts: 1618
Joined: Tue Mar 29, 2016 3:45 am
Location: Aalst (Belgium)
Name: Andy Dm
Contact:

Re: Evil bot causes host to shutdown my site

Post by </Solidjeuh> » Tue Jun 18, 2019 3:33 pm

I'm trying to add those Bad Bots directly in Fail2Ban, but there is an error. So I'm waiting on a reply there. That would be better then robots.txt or .htaccess. https://github.com/mitchellkrogza/apach ... -503180991
Register a free account & Play!!
~~~ https://www.solidjeuh.be ~~~
Have a secret? --> https://www.tellyoursecrets.eu

User avatar
</Solidjeuh>
Registered User
Posts: 1618
Joined: Tue Mar 29, 2016 3:45 am
Location: Aalst (Belgium)
Name: Andy Dm
Contact:

Re: Evil bot causes host to shutdown my site

Post by </Solidjeuh> » Tue Jun 18, 2019 8:22 pm

Okey, the Fail2Ban seems to work perfectly. So if you have Fail2Ban on your server, use that function instead of .htaccess or robots.txt

https://github.com/mitchellkrogza/apach ... ot-blocker

Just note that the readme file tells you to add code in jail.local, if you cannot restart Fail2Ban with that code, use this code instead:

Code: Select all

[INCLUDES]
before = paths-debian.conf
enabled = true

[apacherepeatoffender]
enabled = true
logpath = %(apache_access_log)s
filter = apacherepeatoffender
banaction = apacherepeatoffender
bantime = 86400 ; 1 day
findtime = 604800 ; 1 week
maxretry = 20
https://github.com/mitchellkrogza/apach ... -503294129

Fail2ban restart: sudo service fail2ban restart
Register a free account & Play!!
~~~ https://www.solidjeuh.be ~~~
Have a secret? --> https://www.tellyoursecrets.eu

azhrei_fje
Registered User
Posts: 15
Joined: Fri Feb 13, 2009 6:28 pm

Re: Evil bot causes host to shutdown my site

Post by azhrei_fje » Tue Jun 18, 2019 11:57 pm

John connor wrote:
Tue Jun 18, 2019 3:25 pm
You're welcome. If you go CloudFlare, be sure to read my post on that. If you go to the link in my Sig, there's a follow up post on CloudFlare there. Let me know if you have any questions. If you want layer 7 DDoS protection you'll have to pay CloudFlare. I do and it's about 30 cents a month, though, I don't have a lot of traffic either unfortunately. Since my forum is mainly on computers, etc, I have a lot of competition. :lol:
Heh, yep.

Yes, I read the article you linked in your first reply, but I haven't checked out the links in your signature. I do appreciate that you put all that info up about protecting the IP address and such. When we move to another hosting service, we'll get a new IP and I'll make sure CloudFlare is done first. :)
Actually, I learned that Google does NOT use the robots.txt — but they DO use the contents of some of the meta links in the <head> block.
Where did you read that? With me, all indications lead to Google actually honoring what's in robots.txt.
Hm, I can't find it now. I was a page on search.google.com that talked about setting the crawl rate, and I went from there down a link chain that took me to a FAQ page. But everything I find in my browser history isn't relevant, and I can't find it now by searching... I'm going to pull a mea culpa and say I must've had too many rum&cokes (or not enough!).
Use this site with your WordPress domain. https://hackertarget.com/wordpress-security-scan/
Cool, I'll check that one out, too. Thanks!

Post Reply

Return to “Extension Requests”