Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
Post Reply
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

3Di wrote:
Marcus Wendel wrote:PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......
It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards. :)
I agree, but I'm not sure how much information I can include in one single post.

/Marcus
User avatar
3Di
Former Team Member
Posts: 16147
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano 🇮🇹 Frankfurt 🇩🇪
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

Marcus Wendel wrote:
3Di wrote:
Marcus Wendel wrote:PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......
It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards. :)
I agree, but I'm not sure how much information I can include in one single post.

/Marcus
Maybe (in case) a link to an external web page does the trick. ;)
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
Luchtzak
Registered User
Posts: 267
Joined: Sat Jul 20, 2002 7:13 pm
Contact:

Re: Spiders & bots to add to phpbb3

Post by Luchtzak »

Suggestion: a sql-query to run in the database so all the bots are added in an instant ;-)
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Marcus Wendel wrote:Bot name: Twiceler [Bot]
Agent match: Twiceler
Information on the bot: http://cuill.com/twiceler/robot.html
Corrected the name to Cuil [Bot]

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Luchtzak wrote:Suggestion: a sql-query to run in the database so all the bots are added in an instant ;-)
Raimon posted one earlier in the thread and you can find it updated with the current bots at http://flagsforum.skalman.nu/viewtopic.php?f=2&t=282

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

3Di wrote:
Marcus Wendel wrote:
3Di wrote:
Marcus Wendel wrote:PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......
It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards. :)
I agree, but I'm not sure how much information I can include in one single post.

/Marcus
Maybe (in case) a link to an external web page does the trick. ;)
I'll keep this thread updated with the bots: http://flagsforum.skalman.nu/viewtopic.php?f=2&t=282

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Two new ones today:

Bot name: Yebol [Bot]
Agent match: YebolBot
Information on the bot: http://tech.groups.yahoo.com/group/yeboltechnology/

Bot name: Microsoft Research [Bot]
Agent match: MSRBOT
Information on the bot: http://research.microsoft.com/research/sv/msrbot/

/Marcus
User avatar
3Di
Former Team Member
Posts: 16147
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano 🇮🇹 Frankfurt 🇩🇪
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

Marcus Wendel wrote:
Luchtzak wrote:Suggestion: a sql-query to run in the database so all the bots are added in an instant ;-)
Raimon posted one earlier in the thread and you can find it updated with the current bots at http://flagsforum.skalman.nu/viewtopic.php?f=2&t=282

/Marcus
Tell him to have a look how phpbb3 (3.0.2 checked here) does that job. ;)

Reference file: install_install.php that lies into the install folder (there are also a lot of bots as candidates, there) ;)

Function: function add_bots($mode, $sub)

Regards.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
Eelke
Registered User
Posts: 2903
Joined: Thu Dec 20, 2001 8:00 am
Location: NL, Bussum
Name: Eelke Blok
Contact:

Re: Spiders & bots to add to phpbb3

Post by Eelke »

Luchtzak wrote:Suggestion: a sql-query to run in the database so all the bots are added in an instant ;-)
If you want to make it really fancy: a web page that allows you to pick which bots you want to add, which then generates the SQL for your selection :) [Edit: or a PHP script that uses the phpBB funcion, per 3Di]
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

One more today:

Bot name: BoardReader [Bot]
Agent match: BoardReader
Information on the bot: http://boardreader.com/info/robots.htm

/Marcus
Darthmat
Registered User
Posts: 114
Joined: Mon Jun 23, 2008 4:08 pm

Re: Spiders & bots to add to phpbb3

Post by Darthmat »

I got this

Code: Select all

Parse error: syntax error, unexpected T_CONSTANT_ENCAPSED_STRING, expecting ')' in /home/soulrise/public_html/twb/forum/add_bots.php on line 60
You forgot to add comas to almost all of the ones following that line. ;)
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Darthmat wrote:I got this

Code: Select all

Parse error: syntax error, unexpected T_CONSTANT_ENCAPSED_STRING, expecting ')' in /home/soulrise/public_html/twb/forum/add_bots.php on line 60
You forgot to add comas to almost all of the ones following that line. ;)
Sorry about that, I've corrected it.

/Marcus
scott_thewspot
Registered User
Posts: 202
Joined: Mon Dec 05, 2005 4:13 am
Location: Australia
Contact:

Re: Spiders & bots to add to phpbb3

Post by scott_thewspot »

Greetings,

I just tried applying this and it seems to work, but I do have some questions.

1. For the SpamBots, you mention you have them disabled. From what I can tell, after the bots are added, they are *all* enabled, including the spambots. There is nothing obvious in your script I can find that would treat them differently when adding them. Am I missing something here?

2. What happens if the script tries to add a bot that already exists in the DB?

It does seem odd that there isn't an "official" bot thread or update system already from the phpbb team themselves, even as part of the patch updates.
Forcing phpBB to submit on Microsoft platforms since 2005
Windows Server 2012R2, IIS8.5, MSSQL2012
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

scott_thewspot,

I've just reposted the script provided by Raimon earlier in the thread and unfortunately I don't know anything about php-programming, I hope is able to answer your questions.

My input here is limited to posting information on new bots I come across on my sites.

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Three new ones today:

Bot name: Best of the Web [Bot]
Agent match: BOTW Spider
Information on the bot: http://botw.org/

Bot name: Brandwatch [Bot]
Agent match: Jakarta Commons
IP match: 80.82.139
Information on the bot: http://www.brandwatch.net/crawler

Bot name: Kalooga [Bot]
Agent match: Kalooga
Information on the bot: http://www.kalooga.com

/Marcus
Post Reply

Return to “phpBB Discussion”