Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Suggested Hosts
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Tue May 13, 2008 6:36 pm

One more:

Bot name: WebCorp [Bot]
Agent match: WebCorp
Information on the bot: http://www.webcorp.org.uk

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Wed May 14, 2008 5:42 pm

One more:

Bot name: WebAlta [Bot]
Agent match: WebAlta
Information on the bot: http://www.webalta.net/ru/about_webmaster.html (Russian search engine)

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Fri May 16, 2008 10:14 pm

One more:

Bot name: Powerset [Bot]
Agent match: zermelo
Information on the bot: http://www.powerset.com/

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sat May 17, 2008 8:35 pm

A bad bot added and banned:

Bot name: Boston Project [SpamBot]
Agent match: Boston Project
Information on the bot: http://www.projecthoneypot.org/bsh_X19t ... dCt2KzEuMA..

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sun May 18, 2008 8:31 am

Two more:

Bot name: Startpagina [Bot]
Agent match: Startpagina
Information on the bot: http://www.startpagina.nl/

Bot name: Heeii [Bot]
Agent match: Heeii
Information on the bot: http://www.heeii.com

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sun May 18, 2008 10:23 am

A bad bot added and banned:

Bot name: Wget [SpamBot]
Agent match: Wget

/Marcus

[Note. bot is not really a correct description as pointed out later in this thread]
Last edited by Marcus Wendel on Wed May 21, 2008 7:46 pm, edited 1 time in total.

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Sun May 18, 2008 10:34 am

Four more:

Bot name: Yodao [Bot]
Agent match: YodaoBot
Information on the bot: http://www.yodao.com/ [Chinese search engine]

Bot name: vBSEO [Bot]
Agent match: vBSEO
Information on the bot: http://www.vbseo.com/

Bot name: WiseGuys [Bot]
Agent match: Vagabondo
Information on the bot: http://webagent.wise-guys.nl/

Bot name: Searchme [Bot]
Agent match: Charlotte
Information on the bot: http://www.searchme.com/support/pages/spider.php

/Marcus

User avatar
Eelke
QA Team
Posts: 2903
Joined: Thu Dec 20, 2001 8:00 am
Location: NL, Bussum
Name: Eelke Blok
Contact:

Re: Spiders & bots to add to phpbb3

Post by Eelke » Mon May 19, 2008 11:33 am

Marcus Wendel wrote:A bad bot added and banned:

Bot name: Wget [SpamBot]
Agent match: Wget

/Marcus
Information about the "bot": http://en.wikipedia.org/wiki/Wget

You'll find that wget is not a bot in itself, but just a utility to retrieve HTML content. It is probably used to write shell script bots, but I think anyone considering to add this should be aware what it actually is. For example, you yourself, as a board admin, could use it to schedule, on a local server, the "retrieval" of some URL that actually starts some task, that way simulating a server-side cron setup.

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Mon May 19, 2008 11:39 am

I know what wget is but you are right, I should have been more clear about that, thanks.

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Wed May 21, 2008 7:47 pm

Two more:

Bot name: Exalead [Bot]
Agent match: Exabot
Information on the bot: http://www.exalead.com/about/document/53#3

Bot name: Yahoo Search Marketing [Bot]
Agent match: YahooYSMcm
Information on the bot: no offical page found

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Thu May 22, 2008 6:27 pm

One more:

Bot name: Daum [Bot]
Agent match: Daumoa
Information on the bot: http://ws.daum.net/abouten.html

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Thu May 29, 2008 5:05 pm

One more:

Bot name: webcollage [Bot]
Agent match: webcollage
Information on the bot: http://www.jwz.org/webcollage/

/Marcus

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Thu May 29, 2008 7:47 pm

One more:

Bot name: Babaloo [Bot]
Agent match: BabalooSpider
Information on the bot: http://www.babaloo.si

/Marcus

HB
Registered User
Posts: 139
Joined: Mon May 16, 2005 9:30 pm
Contact:

Re: Bots to add to phpbb3

Post by HB » Thu May 29, 2008 8:47 pm

3Di wrote:
Marcus Wendel wrote:Bot name: Omgili [Bot]
Agent match: omgilibot
Information on the bot: http://www.omgili.com/Crawler.html
This one I banned its IP, I recall (lurking at my database and my online guests flags) that creates a lot of sessions, like a Spider, it is not a BOT, it's a crawler or spider IMO.
I disallowed this crawler in robots.txt because it scrapes and then presents the crawled forum content nearly in its entirety. The omgili FAQ offers this explanation:
omgili FAQ wrote: What is the preview feature?
The previews are a neat feature that gives the users the ability to view most of the discussion rather then the two snippets the search engine returns on the results page. The content may also be available in case the forum is down or the post was deleted due to database restrains. The original post link is marked in bold on the top of the page.

How do I tell Omgili to disable previews for my forum?
If you decide you don't want to enable the preview feature for your board, add the noarchive meta tag to your board's topics:

<meta name="omgilibot" content="noarchive">
or
<meta name="googlebot" content="noarchive">
Dan Kehn

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel » Fri May 30, 2008 4:47 pm

Thanks HB, I think I'll add that meta tag to my forums.

/Marcus

Post Reply

Return to “phpBB Discussion”