Robot.txt

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
User avatar
AmigoJack
Registered User
Posts: 5588
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Robot.txt

Post by AmigoJack » Fri Mar 17, 2017 10:57 am

As for "non ssl htaccess.zip": no. Still 97 duplicates and still entries like

Code: Select all

SetEnvIfNoCase User-Agent "^Wget/1.5.2" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.3" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.6" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.7" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.8" bad_bot
instead of just

Code: Select all

SetEnvIfNoCase User-Agent "^Wget/1.5.[23]" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.[6-8]" bad_bot
as well as redundancies like

Code: Select all

SetEnvIfNoCase User-Agent "^Webster.*" bad_bot
SetEnvIfNoCase User-Agent "^Webster.Pro.*" bad_bot

Don't try it, learn it. Use an editor that is capable of both sorting lines and killing duplicates. Learn about regular expressions.
The worst thing about censorship is ███████████
Affin wrote:
Tue Nov 20, 2018 9:51 am
The problem is probably not my English but you do not want to understand correctly.
...
We will not come anybody anyway, nevertheless, it's best to shit this.

kaspir
Registered User
Posts: 203
Joined: Sun Jul 27, 2008 5:05 am
Name: Greg
Contact:

Re: Robot.txt

Post by kaspir » Sat Mar 18, 2017 11:05 am

HiFiKabin wrote:
Fri Mar 17, 2017 9:34 am
New and improved (I hope) list of bad/aggressive bots and scrapers now available in .htaccess form for ssl and non ssl sites as well as the raw list for your own use https://phpbb.hifikabin.me.uk/viewtopic.php?f=6&p=324
Knew you would! BOUSS
World of Phaos RPG online is making it's come back! Play free now!
Check out phpBB contributions & extension downloads. :P

Post Reply

Return to “phpBB Discussion”