Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
Post Reply
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Spiders & bots to add to phpBB

Post by Marcus Wendel » Sat May 03, 2008 8:37 pm

PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one:
Bot name: Twiceler [Bot]
Agent match: Twiceler
Information on the bot: http://cuill.com/twiceler/robot.html

/Marcus

(I did a search but found no existing topic for this, so I hope this is not a redundant thread)
Last edited by Marcus Wendel on Fri Jan 30, 2015 8:40 pm, edited 5 times in total.

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sun May 04, 2008 11:00 am

I've seen a few more frequenting my forums:

Bot name: Voila [Bot]
Agent match: VoilaBot
Information on the bot: http://www.useragentstring.com/pages/VoilaBot/

Bot name: Omgili [Bot]
Agent match: omgilibot
Information on the bot: http://www.omgili.com/Crawler.html

Bot name: Noxtrum [Bot]
Agent match: noxtrumbot
Information on the bot: http://www.useragentstring.com/noxtrumb ... d_1642.php

Bot name: Spinn3r [Bot]
Agent match: Spinn3r
Information on the bot: http://spinn3r.com/robot

/Marcus
Last edited by Marcus Wendel on Sat May 10, 2008 9:50 am, edited 1 time in total.

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Fri May 09, 2008 9:01 pm

Two more added today:

Bot name: Furl [Bot]
Agent match: FurlBot
Information on the bot: http://www.useragentstring.com/FurlBotF ... d_4614.php

Bot name: CommonCrawl [Bot]
Agent match: CCBot
Information on the bot: http://www.commoncrawl.org/bot.html

/Marcus
Last edited by Marcus Wendel on Sat May 10, 2008 9:51 am, edited 1 time in total.

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 10, 2008 9:23 am

Two more:

Bot name: Naver [Bot]
Agent match: Yeti
Information on the bot: http://help.naver.com/robots/ (in Korean)

Bot name: BDProtect [Bot]
Agent match: BPImageWalker
Information on the bot: http://www.bdbrandprotect.com/solutions_5.html

/Marcus
Last edited by Marcus Wendel on Sat May 10, 2008 9:51 am, edited 1 time in total.

User avatar
3Di
Registered User
Posts: 11810
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano - Frankfurt
Name: Marco
Contact:

Re: Bots to add to phpbb3

Post by 3Di » Sat May 10, 2008 9:36 am

Marcus Wendel wrote:Bot name: Omgili [Bot]
Agent match: omgilibot
Information on the bot: http://www.omgili.com/Crawler.html
This one I banned its IP, I recall (lurking at my database and my online guests flags) that creates a lot of sessions, like a Spider, it is not a BOT, it's a crawler or spider IMO.
Want to compensate me for my interest? Donate
Please PM me only to request paid works. Thx.
Extensions, Scripts, MOD porting, Update/Upgrades

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 10, 2008 9:52 am

So far I've not seen any problems with Omgili but thanks for the heads up, I'll keep an eye on it.

I'm not really good at the terminology of these things so I've changed the title of this thread to follow the terminology used in phpBB.

/Marcus

User avatar
3Di
Registered User
Posts: 11810
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano - Frankfurt
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di » Sat May 10, 2008 10:02 am

Marcus Wendel wrote:So far I've not seen any problems with Omgili but thanks for the heads up, I'll keep an eye on it.

I'm not really good at the terminology of these things so I've changed the title of this thread to follow the terminology used in phpBB.

/Marcus
Well, I have nothing against to list all the crawlers/spiders/bots on the online list but I do consider the fact it creates about 9/10 different sessions in the same time in my test live board where I have 4 forums with 100 posts, I don't know if in a bigger forum it creates many more sessions, anyway it's too much for my point of view (also annoying to see 9/10 past guests flags all togheter when I know it is just ONE automa).

So I banned its IP, banning the IP avoid to have so many sessions there, in the database. :)

Regards.
Last edited by 3Di on Sat May 10, 2008 10:05 am, edited 1 time in total.
Want to compensate me for my interest? Donate
Please PM me only to request paid works. Thx.
Extensions, Scripts, MOD porting, Update/Upgrades

User avatar
3Di
Registered User
Posts: 11810
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano - Frankfurt
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di » Sat May 10, 2008 10:04 am

And yes.. thanks for this Topic you made, I found it very interesting and useful. :)
Want to compensate me for my interest? Donate
Please PM me only to request paid works. Thx.
Extensions, Scripts, MOD porting, Update/Upgrades

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 10, 2008 10:05 am

One more:

Bot name: Snap Shots [Bot]
Agent match: Snapbot
Information on the bot: http://www.snap.com

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 10, 2008 10:05 am

3Di wrote:And yes.. thanks for this Topic you made, I found it very interesting and useful. :)
Thanks, I'm glad to hear it.

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 10, 2008 10:11 am

Another one:

Bot name: Whitevector [Bot]
Agent match: Whitevector Crawler
Information on the bot: www.whitevector.com


I'm not sure what to make of this one, should it be included or not?:

Bot name: Hatena Antenna [Bot]
Agent match: Hatena Antenna
Information on the bot: http://www.akaiho.com/hatenaA_En.html

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 10, 2008 5:43 pm

Marcus Wendel wrote:Bot name: Snap Shots [Bot]
Agent match: Snapbot
Information on the bot: http://www.snap.com
Related to the above one:

Bot name: Snap Shots Preview [Bot]
Agent match: SnapPreviewBot
Information on the bot: http://www.snap.com/snapshots_faq.php
http://www.snap.com/snapshots_faq.php wrote:My site got a lot of automated traffic that seems to have been from Snap. What is going on?

There are two possibilities:

1. When someone rolls over a Snap Shot-enabled link for the first time, Snap takes a picture of the linked-to Web page so the image can be displayed in the preview bubble. If someone links to many pages at your site, you may notice a temporary increase in automated traffic. However this is generally quite brief because Snap stores the image so it does not generally need to fetch it again. The user-agent for our snapshot-taking bot is "SnapPreviewBot".
2. Snap.com, our search engine, uses an indexing bot that may occasionally visit your Web pages. The user-agent for our indexing bot is "SnapBot/1.0".
/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 10, 2008 6:53 pm

Two more:

Bot name: Ilse [Bot]
Agent match: IlseBot
Information on the bot: http://www.ilse.nl/

Bot name: ImageShack [Bot]
Agent match: ImageShack Image Fetcher
Information on the bot: http://www.imageshack.us

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 10, 2008 7:45 pm

One more:

Bot name: Entireweb [Bot]
Agent match: Speedy Spider
Information on the bot: http://www.entireweb.com/about/search_t ... dy_spider/

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 10, 2008 8:05 pm

One more:

Bot name: Yandex [Bot]
Agent match: Yandex
Information on the bot: http://en.wikipedia.org/wiki/Yandex

/Marcus

Post Reply

Return to “phpBB Discussion”

Who is online

Users browsing this forum: njs and 27 guests