Spiders & bots to add to phpBB

[Marcus Wendel]

PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......

It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards.

I agree, but I'm not sure how much information I can include in one single post.

/Marcus

[3Di]

PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......

It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards.

I agree, but I'm not sure how much information I can include in one single post.

/Marcus

Maybe (in case) a link to an external web page does the trick.

[Luchtzak]

Suggestion: a sql-query to run in the database so all the bots are added in an instant

[Marcus Wendel]

Bot name: Twiceler [Bot]
Agent match: Twiceler
Information on the bot: http://cuill.com/twiceler/robot.html

Corrected the name to Cuil [Bot]

/Marcus

[Marcus Wendel]

Suggestion: a sql-query to run in the database so all the bots are added in an instant

Raimon posted one earlier in the thread and you can find it updated with the current bots at http://flagsforum.skalman.nu/viewtopic.php?f=2&t=282

/Marcus

[Marcus Wendel]

PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one........ snip ......

It's time to resume all of this into the Topic starter, for a better reading... do you agree?
Regards.

I agree, but I'm not sure how much information I can include in one single post.

/Marcus

Maybe (in case) a link to an external web page does the trick.

I'll keep this thread updated with the bots: http://flagsforum.skalman.nu/viewtopic.php?f=2&t=282

/Marcus

[Marcus Wendel]

Two new ones today:

Bot name: Yebol [Bot]
Agent match: YebolBot
Information on the bot: http://tech.groups.yahoo.com/group/yeboltechnology/

Bot name: Microsoft Research [Bot]
Agent match: MSRBOT
Information on the bot: http://research.microsoft.com/research/sv/msrbot/

/Marcus

[3Di]

Suggestion: a sql-query to run in the database so all the bots are added in an instant

Raimon posted one earlier in the thread and you can find it updated with the current bots at http://flagsforum.skalman.nu/viewtopic.php?f=2&t=282

/Marcus

Tell him to have a look how phpbb3 (3.0.2 checked here) does that job.

Reference file: install_install.php that lies into the install folder (there are also a lot of bots as candidates, there)

Function: function add_bots($mode, $sub)

Regards.

[Eelke]

Suggestion: a sql-query to run in the database so all the bots are added in an instant

If you want to make it really fancy: a web page that allows you to pick which bots you want to add, which then generates the SQL for your selection [Edit: or a PHP script that uses the phpBB funcion, per 3Di]

[Marcus Wendel]

One more today:

Bot name: BoardReader [Bot]
Agent match: BoardReader
Information on the bot: http://boardreader.com/info/robots.htm

/Marcus

[Darthmat]

I got this

Code: Select all

Parse error: syntax error, unexpected T_CONSTANT_ENCAPSED_STRING, expecting ')' in /home/soulrise/public_html/twb/forum/add_bots.php on line 60

You forgot to add comas to almost all of the ones following that line.

[Marcus Wendel]

I got this

Code: Select all

Parse error: syntax error, unexpected T_CONSTANT_ENCAPSED_STRING, expecting ')' in /home/soulrise/public_html/twb/forum/add_bots.php on line 60

You forgot to add comas to almost all of the ones following that line.

Sorry about that, I've corrected it.

/Marcus

[scott_thewspot]

Greetings,

I just tried applying this and it seems to work, but I do have some questions.

1. For the SpamBots, you mention you have them disabled. From what I can tell, after the bots are added, they are *all* enabled, including the spambots. There is nothing obvious in your script I can find that would treat them differently when adding them. Am I missing something here?

2. What happens if the script tries to add a bot that already exists in the DB?

It does seem odd that there isn't an "official" bot thread or update system already from the phpbb team themselves, even as part of the patch updates.

[Marcus Wendel]

scott_thewspot,

I've just reposted the script provided by Raimon earlier in the thread and unfortunately I don't know anything about php-programming, I hope is able to answer your questions.

My input here is limited to posting information on new bots I come across on my sites.

/Marcus

[Marcus Wendel]

Three new ones today:

Bot name: Best of the Web [Bot]
Agent match: BOTW Spider
Information on the bot: http://botw.org/

Bot name: Brandwatch [Bot]
Agent match: Jakarta Commons
IP match: 80.82.139
Information on the bot: http://www.brandwatch.net/crawler

Bot name: Kalooga [Bot]
Agent match: Kalooga
Information on the bot: http://www.kalooga.com

/Marcus