Page 1 of 60

Spiders & bots to add to phpBB

Posted: Sat May 03, 2008 8:37 pm
by Marcus Wendel
PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one:
Bot name: Twiceler [Bot]
Agent match: Twiceler
Information on the bot: http://cuill.com/twiceler/robot.html

/Marcus

(I did a search but found no existing topic for this, so I hope this is not a redundant thread)

Re: Spiders & bots to add to phpbb3

Posted: Sun May 04, 2008 11:00 am
by Marcus Wendel
I've seen a few more frequenting my forums:

Bot name: Voila [Bot]
Agent match: VoilaBot
Information on the bot: http://www.useragentstring.com/pages/VoilaBot/

Bot name: Omgili [Bot]
Agent match: omgilibot
Information on the bot: http://www.omgili.com/Crawler.html

Bot name: Noxtrum [Bot]
Agent match: noxtrumbot
Information on the bot: http://www.useragentstring.com/noxtrumb ... d_1642.php

Bot name: Spinn3r [Bot]
Agent match: Spinn3r
Information on the bot: http://spinn3r.com/robot

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Fri May 09, 2008 9:01 pm
by Marcus Wendel
Two more added today:

Bot name: Furl [Bot]
Agent match: FurlBot
Information on the bot: http://www.useragentstring.com/FurlBotF ... d_4614.php

Bot name: CommonCrawl [Bot]
Agent match: CCBot
Information on the bot: http://www.commoncrawl.org/bot.html

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 9:23 am
by Marcus Wendel
Two more:

Bot name: Naver [Bot]
Agent match: Yeti
Information on the bot: http://help.naver.com/robots/ (in Korean)

Bot name: BDProtect [Bot]
Agent match: BPImageWalker
Information on the bot: http://www.bdbrandprotect.com/solutions_5.html

/Marcus

Re: Bots to add to phpbb3

Posted: Sat May 10, 2008 9:36 am
by 3Di
Marcus Wendel wrote:Bot name: Omgili [Bot]
Agent match: omgilibot
Information on the bot: http://www.omgili.com/Crawler.html
This one I banned its IP, I recall (lurking at my database and my online guests flags) that creates a lot of sessions, like a Spider, it is not a BOT, it's a crawler or spider IMO.

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 9:52 am
by Marcus Wendel
So far I've not seen any problems with Omgili but thanks for the heads up, I'll keep an eye on it.

I'm not really good at the terminology of these things so I've changed the title of this thread to follow the terminology used in phpBB.

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 10:02 am
by 3Di
Marcus Wendel wrote:So far I've not seen any problems with Omgili but thanks for the heads up, I'll keep an eye on it.

I'm not really good at the terminology of these things so I've changed the title of this thread to follow the terminology used in phpBB.

/Marcus
Well, I have nothing against to list all the crawlers/spiders/bots on the online list but I do consider the fact it creates about 9/10 different sessions in the same time in my test live board where I have 4 forums with 100 posts, I don't know if in a bigger forum it creates many more sessions, anyway it's too much for my point of view (also annoying to see 9/10 past guests flags all togheter when I know it is just ONE automa).

So I banned its IP, banning the IP avoid to have so many sessions there, in the database. :)

Regards.

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 10:04 am
by 3Di
And yes.. thanks for this Topic you made, I found it very interesting and useful. :)

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 10:05 am
by Marcus Wendel
One more:

Bot name: Snap Shots [Bot]
Agent match: Snapbot
Information on the bot: http://www.snap.com

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 10:05 am
by Marcus Wendel
3Di wrote:And yes.. thanks for this Topic you made, I found it very interesting and useful. :)
Thanks, I'm glad to hear it.

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 10:11 am
by Marcus Wendel
Another one:

Bot name: Whitevector [Bot]
Agent match: Whitevector Crawler
Information on the bot: www.whitevector.com


I'm not sure what to make of this one, should it be included or not?:

Bot name: Hatena Antenna [Bot]
Agent match: Hatena Antenna
Information on the bot: http://www.akaiho.com/hatenaA_En.html

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 5:43 pm
by Marcus Wendel
Marcus Wendel wrote:Bot name: Snap Shots [Bot]
Agent match: Snapbot
Information on the bot: http://www.snap.com
Related to the above one:

Bot name: Snap Shots Preview [Bot]
Agent match: SnapPreviewBot
Information on the bot: http://www.snap.com/snapshots_faq.php
http://www.snap.com/snapshots_faq.php wrote:My site got a lot of automated traffic that seems to have been from Snap. What is going on?

There are two possibilities:

1. When someone rolls over a Snap Shot-enabled link for the first time, Snap takes a picture of the linked-to Web page so the image can be displayed in the preview bubble. If someone links to many pages at your site, you may notice a temporary increase in automated traffic. However this is generally quite brief because Snap stores the image so it does not generally need to fetch it again. The user-agent for our snapshot-taking bot is "SnapPreviewBot".
2. Snap.com, our search engine, uses an indexing bot that may occasionally visit your Web pages. The user-agent for our indexing bot is "SnapBot/1.0".
/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 6:53 pm
by Marcus Wendel
Two more:

Bot name: Ilse [Bot]
Agent match: IlseBot
Information on the bot: http://www.ilse.nl/

Bot name: ImageShack [Bot]
Agent match: ImageShack Image Fetcher
Information on the bot: http://www.imageshack.us

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 7:45 pm
by Marcus Wendel
One more:

Bot name: Entireweb [Bot]
Agent match: Speedy Spider
Information on the bot: http://www.entireweb.com/about/search_t ... dy_spider/

/Marcus

Re: Spiders & bots to add to phpbb3

Posted: Sat May 10, 2008 8:05 pm
by Marcus Wendel
One more:

Bot name: Yandex [Bot]
Agent match: Yandex
Information on the bot: http://en.wikipedia.org/wiki/Yandex

/Marcus