Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Ideas Centre
Post Reply
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Spiders & bots to add to phpBB

Post by Marcus Wendel »

PhpBB3 has a nice built in list of bots, but it is not complete so I'm wondering which bots you all have added to your forums, please share in this thread.

So far I've only added one:
Bot name: Twiceler [Bot]
Agent match: Twiceler
Information on the bot: http://cuill.com/twiceler/robot.html

/Marcus

(I did a search but found no existing topic for this, so I hope this is not a redundant thread)
Last edited by Marcus Wendel on Fri Jan 30, 2015 8:40 pm, edited 5 times in total.
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

I've seen a few more frequenting my forums:

Bot name: Voila [Bot]
Agent match: VoilaBot
Information on the bot: http://www.useragentstring.com/pages/VoilaBot/

Bot name: Omgili [Bot]
Agent match: omgilibot
Information on the bot: http://www.omgili.com/Crawler.html

Bot name: Noxtrum [Bot]
Agent match: noxtrumbot
Information on the bot: http://www.useragentstring.com/noxtrumb ... d_1642.php

Bot name: Spinn3r [Bot]
Agent match: Spinn3r
Information on the bot: http://spinn3r.com/robot

/Marcus
Last edited by Marcus Wendel on Sat May 10, 2008 9:50 am, edited 1 time in total.
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Two more added today:

Bot name: Furl [Bot]
Agent match: FurlBot
Information on the bot: http://www.useragentstring.com/FurlBotF ... d_4614.php

Bot name: CommonCrawl [Bot]
Agent match: CCBot
Information on the bot: http://www.commoncrawl.org/bot.html

/Marcus
Last edited by Marcus Wendel on Sat May 10, 2008 9:51 am, edited 1 time in total.
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Two more:

Bot name: Naver [Bot]
Agent match: Yeti
Information on the bot: http://help.naver.com/robots/ (in Korean)

Bot name: BDProtect [Bot]
Agent match: BPImageWalker
Information on the bot: http://www.bdbrandprotect.com/solutions_5.html

/Marcus
Last edited by Marcus Wendel on Sat May 10, 2008 9:51 am, edited 1 time in total.
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco
Contact:

Re: Bots to add to phpbb3

Post by 3Di »

Marcus Wendel wrote:Bot name: Omgili [Bot]
Agent match: omgilibot
Information on the bot: http://www.omgili.com/Crawler.html
This one I banned its IP, I recall (lurking at my database and my online guests flags) that creates a lot of sessions, like a Spider, it is not a BOT, it's a crawler or spider IMO.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

So far I've not seen any problems with Omgili but thanks for the heads up, I'll keep an eye on it.

I'm not really good at the terminology of these things so I've changed the title of this thread to follow the terminology used in phpBB.

/Marcus
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

Marcus Wendel wrote:So far I've not seen any problems with Omgili but thanks for the heads up, I'll keep an eye on it.

I'm not really good at the terminology of these things so I've changed the title of this thread to follow the terminology used in phpBB.

/Marcus
Well, I have nothing against to list all the crawlers/spiders/bots on the online list but I do consider the fact it creates about 9/10 different sessions in the same time in my test live board where I have 4 forums with 100 posts, I don't know if in a bigger forum it creates many more sessions, anyway it's too much for my point of view (also annoying to see 9/10 past guests flags all togheter when I know it is just ONE automa).

So I banned its IP, banning the IP avoid to have so many sessions there, in the database. :)

Regards.
Last edited by 3Di on Sat May 10, 2008 10:05 am, edited 1 time in total.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

And yes.. thanks for this Topic you made, I found it very interesting and useful. :)
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

One more:

Bot name: Snap Shots [Bot]
Agent match: Snapbot
Information on the bot: http://www.snap.com

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

3Di wrote:And yes.. thanks for this Topic you made, I found it very interesting and useful. :)
Thanks, I'm glad to hear it.

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Another one:

Bot name: Whitevector [Bot]
Agent match: Whitevector Crawler
Information on the bot: www.whitevector.com


I'm not sure what to make of this one, should it be included or not?:

Bot name: Hatena Antenna [Bot]
Agent match: Hatena Antenna
Information on the bot: http://www.akaiho.com/hatenaA_En.html

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Marcus Wendel wrote:Bot name: Snap Shots [Bot]
Agent match: Snapbot
Information on the bot: http://www.snap.com
Related to the above one:

Bot name: Snap Shots Preview [Bot]
Agent match: SnapPreviewBot
Information on the bot: http://www.snap.com/snapshots_faq.php
http://www.snap.com/snapshots_faq.php wrote:My site got a lot of automated traffic that seems to have been from Snap. What is going on?

There are two possibilities:

1. When someone rolls over a Snap Shot-enabled link for the first time, Snap takes a picture of the linked-to Web page so the image can be displayed in the preview bubble. If someone links to many pages at your site, you may notice a temporary increase in automated traffic. However this is generally quite brief because Snap stores the image so it does not generally need to fetch it again. The user-agent for our snapshot-taking bot is "SnapPreviewBot".
2. Snap.com, our search engine, uses an indexing bot that may occasionally visit your Web pages. The user-agent for our indexing bot is "SnapBot/1.0".
/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Two more:

Bot name: Ilse [Bot]
Agent match: IlseBot
Information on the bot: http://www.ilse.nl/

Bot name: ImageShack [Bot]
Agent match: ImageShack Image Fetcher
Information on the bot: http://www.imageshack.us

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

One more:

Bot name: Entireweb [Bot]
Agent match: Speedy Spider
Information on the bot: http://www.entireweb.com/about/search_t ... dy_spider/

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

One more:

Bot name: Yandex [Bot]
Agent match: Yandex
Information on the bot: http://en.wikipedia.org/wiki/Yandex

/Marcus
Post Reply

Return to “phpBB Discussion”