Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Ideas Centre
Post Reply
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Mon Jul 28, 2008 6:56 am

3Di wrote:
Marcus Wendel wrote:PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......
It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards. :)
I agree, but I'm not sure how much information I can include in one single post.

/Marcus

User avatar
3Di
Former Team Member
Posts: 14223
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milan (IT) Frankfurt (DE)
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di » Mon Jul 28, 2008 11:16 am

Marcus Wendel wrote:
3Di wrote:
Marcus Wendel wrote:PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......
It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards. :)
I agree, but I'm not sure how much information I can include in one single post.

/Marcus
Maybe (in case) a link to an external web page does the trick. ;)
Please PM me only to request paid works. Thx.
Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user
Extensions, Scripts, MOD porting, Update/Upgrades
👨‍🏫 | Take a tour to | The Studio | 👨‍🏫

Luchtzak
Registered User
Posts: 261
Joined: Sat Jul 20, 2002 7:13 pm
Contact:

Re: Spiders & bots to add to phpbb3

Post by Luchtzak » Tue Jul 29, 2008 2:23 pm

Suggestion: a sql-query to run in the database so all the bots are added in an instant ;-)

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Tue Jul 29, 2008 3:40 pm

Marcus Wendel wrote:Bot name: Twiceler [Bot]
Agent match: Twiceler
Information on the bot: http://cuill.com/twiceler/robot.html
Corrected the name to Cuil [Bot]

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Tue Jul 29, 2008 3:42 pm

Luchtzak wrote:Suggestion: a sql-query to run in the database so all the bots are added in an instant ;-)
Raimon posted one earlier in the thread and you can find it updated with the current bots at http://flagsforum.skalman.nu/viewtopic.php?f=2&t=282

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Tue Jul 29, 2008 4:10 pm

3Di wrote:
Marcus Wendel wrote:
3Di wrote:
Marcus Wendel wrote:PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......
It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards. :)
I agree, but I'm not sure how much information I can include in one single post.

/Marcus
Maybe (in case) a link to an external web page does the trick. ;)
I'll keep this thread updated with the bots: http://flagsforum.skalman.nu/viewtopic.php?f=2&t=282

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Tue Jul 29, 2008 5:58 pm

Two new ones today:

Bot name: Yebol [Bot]
Agent match: YebolBot
Information on the bot: http://tech.groups.yahoo.com/group/yeboltechnology/

Bot name: Microsoft Research [Bot]
Agent match: MSRBOT
Information on the bot: http://research.microsoft.com/research/sv/msrbot/

/Marcus

User avatar
3Di
Former Team Member
Posts: 14223
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milan (IT) Frankfurt (DE)
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di » Tue Jul 29, 2008 6:09 pm

Marcus Wendel wrote:
Luchtzak wrote:Suggestion: a sql-query to run in the database so all the bots are added in an instant ;-)
Raimon posted one earlier in the thread and you can find it updated with the current bots at http://flagsforum.skalman.nu/viewtopic.php?f=2&t=282

/Marcus
Tell him to have a look how phpbb3 (3.0.2 checked here) does that job. ;)

Reference file: install_install.php that lies into the install folder (there are also a lot of bots as candidates, there) ;)

Function: function add_bots($mode, $sub)

Regards.
Please PM me only to request paid works. Thx.
Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user
Extensions, Scripts, MOD porting, Update/Upgrades
👨‍🏫 | Take a tour to | The Studio | 👨‍🏫

User avatar
Eelke
QA Team
Posts: 2903
Joined: Thu Dec 20, 2001 8:00 am
Location: NL, Bussum
Name: Eelke Blok
Contact:

Re: Spiders & bots to add to phpbb3

Post by Eelke » Wed Jul 30, 2008 8:06 am

Luchtzak wrote:Suggestion: a sql-query to run in the database so all the bots are added in an instant ;-)
If you want to make it really fancy: a web page that allows you to pick which bots you want to add, which then generates the SQL for your selection :) [Edit: or a PHP script that uses the phpBB funcion, per 3Di]

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Wed Jul 30, 2008 4:49 pm

One more today:

Bot name: BoardReader [Bot]
Agent match: BoardReader
Information on the bot: http://boardreader.com/info/robots.htm

/Marcus

Darthmat
Registered User
Posts: 114
Joined: Mon Jun 23, 2008 4:08 pm

Re: Spiders & bots to add to phpbb3

Post by Darthmat » Wed Jul 30, 2008 10:19 pm

I got this

Code: Select all

Parse error: syntax error, unexpected T_CONSTANT_ENCAPSED_STRING, expecting ')' in /home/soulrise/public_html/twb/forum/add_bots.php on line 60
You forgot to add comas to almost all of the ones following that line. ;)

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Thu Jul 31, 2008 5:22 am

Darthmat wrote:I got this

Code: Select all

Parse error: syntax error, unexpected T_CONSTANT_ENCAPSED_STRING, expecting ')' in /home/soulrise/public_html/twb/forum/add_bots.php on line 60
You forgot to add comas to almost all of the ones following that line. ;)
Sorry about that, I've corrected it.

/Marcus

scott_thewspot
Registered User
Posts: 202
Joined: Mon Dec 05, 2005 4:13 am
Location: Australia
Contact:

Re: Spiders & bots to add to phpbb3

Post by scott_thewspot » Thu Jul 31, 2008 1:08 pm

Greetings,

I just tried applying this and it seems to work, but I do have some questions.

1. For the SpamBots, you mention you have them disabled. From what I can tell, after the bots are added, they are *all* enabled, including the spambots. There is nothing obvious in your script I can find that would treat them differently when adding them. Am I missing something here?

2. What happens if the script tries to add a bot that already exists in the DB?

It does seem odd that there isn't an "official" bot thread or update system already from the phpbb team themselves, even as part of the patch updates.
Forcing phpBB to submit on Microsoft platforms since 2005
Windows Server 2012R2, IIS8.5, MSSQL2012

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Thu Jul 31, 2008 1:39 pm

scott_thewspot,

I've just reposted the script provided by Raimon earlier in the thread and unfortunately I don't know anything about php-programming, I hope is able to answer your questions.

My input here is limited to posting information on new bots I come across on my sites.

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Fri Aug 01, 2008 5:30 pm

Three new ones today:

Bot name: Best of the Web [Bot]
Agent match: BOTW Spider
Information on the bot: http://botw.org/

Bot name: Brandwatch [Bot]
Agent match: Jakarta Commons
IP match: 80.82.139
Information on the bot: http://www.brandwatch.net/crawler

Bot name: Kalooga [Bot]
Agent match: Kalooga
Information on the bot: http://www.kalooga.com

/Marcus

Post Reply

Return to “phpBB Discussion”