Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Anti-Spam Guide
Post Reply
User avatar
3Di
Former Team Member
Posts: 16155
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano 🇮🇹 Frankfurt 🇩🇪
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

Marcus Wendel wrote:scott_thewspot,

I've just reposted the script provided by Raimon earlier in the thread and unfortunately I don't know anything about php-programming, I hope is able to answer your questions.

My input here is limited to posting information on new bots I come across on my sites.

/Marcus
Okay, I'll try to share something as a MOD then, no ETA though.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

New today:

Bot name: Jyxo [Bot]
Agent match: Jyxobot
Information on the bot: http://jyxo.cz/ (Czech search engine)

/Marcus
User avatar
reptileguy
Registered User
Posts: 146
Joined: Thu Jan 31, 2008 3:54 pm
Location: The Netherlands
Contact:

Re: Spiders & bots to add to phpbb3

Post by reptileguy »

Is this one on your list?

Bot name: MLbot [bot]
Agent match: mlbot
Information on the bot: http://www.metadatalabs.com/mlbot
IP-address: 71.41.200.74
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

reptileguy wrote:Is this one on your list?

Bot name: MLbot [bot]
Agent match: mlbot
Information on the bot: http://www.metadatalabs.com/mlbot
IP-address: 71.41.200.74
No, I've not yet seen it in my forums when I've checked.

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Has anyone seen the below in their forums?

Guest IP: 74.52.139.226 » Whois
libwww-perl/5.810

/Marcus
User avatar
karlsemple
Former Team Member
Posts: 39802
Joined: Mon Nov 01, 2004 8:54 am
Location: Hereford, UK
Contact:

Re: Spiders & bots to add to phpbb3

Post by karlsemple »

Marcus Wendel wrote:Has anyone seen the below in their forums?

Guest IP: 74.52.139.226 » Whois
libwww-perl/5.810

/Marcus

http://en.wikipedia.org/wiki/Libwww-perl is
It provides an application programming interface to an HTTP
often this is someone running an automated script to test the vulnerability of your board to old hacks. Using phpBB3 and being up to date makes these nothing to worry about, for the record I bounce anyone with this useragent to a 403 permissions page using htaccess
Image
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Thanks karlsemple. What do I need to add to my .htaccess file to handle it that way?

/Marcus
User avatar
karlsemple
Former Team Member
Posts: 39802
Joined: Mon Nov 01, 2004 8:54 am
Location: Hereford, UK
Contact:

Re: Spiders & bots to add to phpbb3

Post by karlsemple »

Marcus Wendel wrote:Thanks karlsemple. What do I need to add to my .htaccess file to handle it that way?

/Marcus

Pretty sure it is the second example in this article http://www.besthostratings.com/articles ... -bots.html
Image
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Thanks again.

/Marcus
User avatar
karlsemple
Former Team Member
Posts: 39802
Joined: Mon Nov 01, 2004 8:54 am
Location: Hereford, UK
Contact:

Re: Spiders & bots to add to phpbb3

Post by karlsemple »

Marcus Wendel wrote:Thanks again.

/Marcus

it would be a good idea to install firefox with the user agent switcher extension installed so you can test it is working as it should :)
Image
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

New today:

Bot name: Wikia Search [Bot]
Agent match: GrubNG
Information on the bot: http://search.wikia.com/wiki/GrubNG_Protocol


btw. If you want to deal with the libwww-perl issue mentioned above within phpbb instead of the htaccess method mentioned above (if you for example are not able to add/edit such files) you can create a bot like the below and then ban it.
Bot name: libwww script [BadBot]
Agent match: libwww-perl

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

A spam bot I suggest banning:

Bot name: Indy Library [BadBot]
Agent match: Indy Library

/Marcus
User avatar
Raimon
Former Team Member
Posts: 12088
Joined: Tue May 30, 2006 5:31 pm
Location: Netherlands
Name: Raimon Meuldijk
Contact:

Re: Spiders & bots to add to phpbb3

Post by Raimon »

3Di wrote: Tell him to have a look how phpbb3 (3.0.2 checked here) does that job. ;)

Reference file: install_install.php that lies into the install folder (there are also a lot of bots as candidates, there) ;)

Function: function add_bots($mode, $sub)

Regards.
phpBB3 install_install.php add the bots, and truncate the existing one IIRC.
The script have i edited from the development trunk , instead that it delete the existing ones , it adds the new bots, and the one that are added , will not be effected.
And the script i posted use also the add_bots function , so what is wrong on that file then ?
When i tested the script, it works just fine on my wamp board.
Need phpBB installation, extenstions, Styles or integrate phpBB with you website?
Contact me for fair prices and good service!
User avatar
3Di
Former Team Member
Posts: 16155
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano 🇮🇹 Frankfurt 🇩🇪
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

Raimon wrote:phpBB3 install_install.php add the bots, and truncate the existing one IIRC.
The function released into the 3.0.2 package does not.
Raimon wrote:The script have i edited from the development trunk , instead that it delete the existing ones , it adds the new bots, and the one that are added , will not be effected.
Will only work if no BOTs were added.

Regards.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
Raimon
Former Team Member
Posts: 12088
Joined: Tue May 30, 2006 5:31 pm
Location: Netherlands
Name: Raimon Meuldijk
Contact:

Re: Spiders & bots to add to phpbb3

Post by Raimon »

3Di wrote:
Raimon wrote:phpBB3 install_install.php add the bots, and truncate the existing one IIRC.
The function released into the 3.0.2 package does not.
Aha i didn't know that.
Will only work if no BOTs were added.

Regards.
What i have tested , if you also have bots added, it truncate the bots that are added, since this code is added ;

Code: Select all

$db->sql_query('TRUNCATE TABLE ' . BOTS_TABLE); 
But i have edited the script , so it don't effect the one that are already on the database.
Need phpBB installation, extenstions, Styles or integrate phpBB with you website?
Contact me for fair prices and good service!
Post Reply

Return to “phpBB Discussion”