Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Anti-Spam Guide
Post Reply
User avatar
3Di
Former Team Member
Posts: 16174
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano 🇮🇹 Frankfurt 🇩🇪
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

Code: Select all

foreach($bot_ids as $bot)
{
    user_delete('remove', $bot);
}
 
this from the trunk, what about it?
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
Raimon
Former Team Member
Posts: 12088
Joined: Tue May 30, 2006 5:31 pm
Location: Netherlands
Name: Raimon Meuldijk
Contact:

Re: Spiders & bots to add to phpbb3

Post by Raimon »

And your point is ?
But instead of "you need to do this" , or "do that", you could write a better one, this discussion about what's wrong or good, is not good for this topic, since Marcus have posted a lot useful bots ;)
And helping nobody with such discussions, so if you can write a better one be my guess , only users would be happy about that, and i believe on STG somebody write also a script to add multiply bots.
I didn't tested that script, so i don't know if / how it works.
Need phpBB installation, extenstions, Styles or integrate phpBB with you website?
Contact me for fair prices and good service!
User avatar
3Di
Former Team Member
Posts: 16174
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano 🇮🇹 Frankfurt 🇩🇪
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

Raimon wrote:And your point is ?
But instead of "you need to do this" , or "do that", you could write a better one, this discussion about what's wrong or good, is not good for this topic, since Marcus have posted a lot useful bots ;)
And helping nobody with such discussions, so if you can write a better one be my guess , only users would be happy about that, and i believe on STG somebody write also a script to add multiply bots.
Yes, I saw that topic there right after I posted here the above statement:
3Di wrote:Okay, I'll try to share something as a MOD then, no ETA though.
also I do not want to re-invent the wheel, I have not checked that code yet. ;)

I do like what Markus is doing here for the community's sake indeed, I just posted something about that by my coder point of view.

Feel free to PM me if you are willing to cooperate with me making a good MOD for BOTs, indeed. :)

Best regards.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

No a bot as such but one you might want to add any way:

"Bot" name: HTTrack [Bot]
Agent match: HTTrack
Information on the bot: http://www.httrack.com/

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

New today:

Bot name: NextLinks [Bot]
Agent match: findlinks
Information on the bot: http://wortschatz.uni-leipzig.de/nextli ... ex_en.html

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Two new ones today:

Bot name: Brandwatch/Magpie [Bot]
Agent match: magpie-crawler
Information on the bot: http://www.magpie.net

Bot name: Yanga [Bot]
Agent match: Yanga
Information on the bot: http://www.yanga.co.uk/

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

New today:

Bot name: Goo [Bot]
Agent match: ichiro
Information on the bot: http://help.goo.ne.jp/help/article/853/

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

Modification to one included in the default installation:

Bot name: Nutch [Bot]
Agent match: Nutch
Information on the bot: http://lucene.apache.org/nutch/

/Marcus
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpbb3

Post by Marcus Wendel »

New today:

Bot name: ScoutJet [Bot]
Agent match: ScoutJet
Information on the bot: http://www.scoutjet.com/

/Marcus
User avatar
3Di
Former Team Member
Posts: 16174
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milano 🇮🇹 Frankfurt 🇩🇪
Name: Marco
Contact:

Re: Spiders & bots to add to phpbb3

Post by 3Di »

that's a fake BOT, at least it seems to be:
NetRange: 91.0.0.0 - 91.255.255.255
Yanga WorldSearch Bot v1.1/beta h t t p://www.yanga.co.uk/

City: Amsterdam
StateProv:
PostalCode: 1001EB
Country: NL



http://www.forumpostersunion.com/showthread.php?t=4088

This is the first time we have seen this "so called bot", their search engine is using results from others at this time.

The Yanga search engine is part of GigaBase.com which is part of Russian search engine spoofing spam botnet WebAlta.com under a new name.

Yanga is also a fake search engine, they are calling all their results from Yahoo using their API.

Yanga is part of a large Russian spam harvesting network that runs giant botnets worldwide (a front for professional spammers).
Actually I banned its IP range.
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Want to compensate me for my interest? Donate
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
ameeck
Former Team Member
Posts: 6559
Joined: Mon Mar 21, 2005 6:57 pm

Re: Spiders & bots to add to phpbb3

Post by ameeck »

I've seen quite a few bots here, e.g. Twiceller/Cuill etc., that aren't too friendly, they often ignore robots.txt, index rapidly while taking quite a lot of bandwith and such. I'd be careful before adding them, it's a good idea to Google a bit about the bot and in some cases, rather than add is at a bot, deny access to the forum for it (a 403 with .htaccess is fine) or ban it.
Captain Oats
Registered User
Posts: 48
Joined: Thu Jan 17, 2008 5:07 pm

Re: Spiders & bots to add to phpbb3

Post by Captain Oats »

Bot name: Synoo [Bot]
Agent match: Synoobot
Information on the bot: http://www.synoo.com/search/bot.html

Bot name: Nextthing [Bot]
Agent match: nextthing.org
Information on the bot: http://www.nextthing.org/bot

Bot name: Megaglobe [Crawler]
Agent match: Megaglobe Crawler
Information on the bot: http://www.megaglobe.com

Bot name: BuzzRanking [Bot]
Agent match: BuzzRankingBot
Information on the bot: http://www.buzzrankingbot.com/

Bot name: MyFamily [Bot]
Agent match: MyFamilyBot
Information on the bot: http://www.ancestry.com/learn/bot.aspx

Bot name: Girafa [Bot]
Agent match: Girafabot
Information on the bot: http://www.girafa.com/

Bot name: DepSpid [Bot]
Agent match: DepSpid
Information on the bot: http://about.depspid.net/

Bot name: Mirago [Bot]
Agent match: Mirago-Test-Robot
Information on the bot: http://www.miragorobot.com

Bot name: Trovatore [Bot]
Agent match: IlTrovatore
Information on the bot: http://www.iltrovatore.it/bot.html

Bot name: Favorstar [Bot]
Agent match: favorstarbot
Information on the bot: http://favorstar.com/bot.html

Bot name: Exooba [Crawler]
Agent match: exooba
Information on the bot: http://www.exooba.com/

Bot name: Enaball [Crawler]
Agent match: EnaBot
Information on the bot: http://www.enaball.com/crawler.html

Bot name: Envolk [Bot]
Agent match: envolk
Information on the bot: http://www.envolk.com/envolkspiderinfo.html

Bot name: DotNet [Bot]
Agent match: DotBot
Information on the bot: http://www.dotnetdotcom.org/

Bot name: Searchles [Bot]
Agent match: Dumbot
Information on the bot: http://www.searchles.com/

Bot name: Cydral [Bot]
Agent match: CydralSpider
Information on the bot: http://www.cydral.com

Bot name: Cazoodle [Bot]
Agent match: CazoodleBot
Information on the bot: http://www.cazoodle.com/cazoodlebot

Bot name: Amfibi [Bot]
Agent match: Amfibibot
Information on the bot: http://www.amfibi.com
http://www.support-phpbb2.com : Support for your phpBB2 Board
User avatar
FladeX
Registered User
Posts: 244
Joined: Tue Jul 08, 2008 10:38 am
Location: Russia, Saransk
Name: Max Istlyaev
Contact:

Re: Spiders & bots to add to phpbb3

Post by FladeX »

ichiro [Crawler] is not detected now, because it use new user-agent "ichiro/3" instead "ichiro/2".
I change to simply "ichiro" on my forum.
and sorry for my bad english [ phpBB Adept ]
My mods: Sitemap FX, Referrals FX, Topiclist FX, Alphabet FX, Polls Results FX.
User avatar
djchrisnet
Registered User
Posts: 95
Joined: Thu Sep 20, 2007 3:39 pm
Contact:

Re: Spiders & bots to add to phpbb3

Post by djchrisnet »

have anybody a list of sql-querys to simply add all new bots??
User avatar
A_Jelly_Doughnut
Former Team Member
Posts: 34457
Joined: Sat Jan 18, 2003 1:26 am
Location: Where the Rivers Run
Contact:

Re: Spiders & bots to add to phpbb3

Post by A_Jelly_Doughnut »

There is a script in the MOD Database called Add Bots by T50.
A Donut's Blog
"Bach's Prelude (Cello Suite No. 1) is driving Indiana country roads in Autumn" - Ann Kish
Post Reply

Return to “phpBB Discussion”