Spiders & bots to add to phpBB

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel » Mon Sep 16, 2013 6:15 pm

Bot name: SEOkicks [Bot]
Agent match: SEOkicks
User agent string: Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
Website: http://www.seokicks.de/robot.html

Bot name: UberMetrics [Bot]
Agent match: ubermetrics
User agent string: Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)
Website: http://www.ubermetrics-technologies.com

(mentioned before, just adding them in the standard format)

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel » Tue Sep 17, 2013 8:40 am

Bot name: Tiscali [Bot]
Agent match: tiscali
User agent string: Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)
Website: http://www.tiscali.it/

maria777
I've Been Banned!
Posts: 7
Joined: Fri Sep 13, 2013 7:19 am

Re: Spiders & bots to add to phpBB

Post by maria777 » Thu Sep 19, 2013 7:08 am

Bot name: Entireweb [Bot]
Agent match: Speedy Spider

Tomba
Registered User
Posts: 131
Joined: Tue Jun 18, 2002 7:52 am
Location: Belgium
Name: Steven De Groote
Contact:

Re: Spiders & bots to add to phpBB

Post by Tomba » Fri Sep 27, 2013 10:57 am

Having seen many many different feed fetchers recently, has anyone ever considered just using one global feedfetcher bot, rather than identifying each one of them seperately?

Perhaps something like
Name: Feed Fetcher [Crawler]
Agent match: feed (or perhaps rss, but I presure we can't use | for OR can we?)

User avatar
AmigoJack
Registered User
Posts: 5349
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Spiders & bots to add to phpBB

Post by AmigoJack » Fri Sep 27, 2013 10:43 pm

Tomba wrote:one global feedfetcher bot
If two or more parallely access the board they'd always force a race condition for the session. Another downside is: you wouldn't get accurate "last visited" timestamps which should be inspected periodically to sort out deceased crawlers.
The worst thing about censorship is ███████████

User avatar
Marcus Wendel
Registered User
Posts: 534
Joined: Sun Mar 10, 2002 5:58 pm
Location: Sweden
Contact:

Re: Spiders & bots to add to phpBB

Post by Marcus Wendel » Sat Oct 19, 2013 11:57 am

Bot name: WeSEE [Bot]
Agent match: WeSEE
User agent string: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
Website: http://www.wesee.com

Bot name: Digg [Bot]
Agent match: Digg
User agent string: Mozilla/5.0 (Digg/1.0; support@digg.com)
Website: http://www.digg.com

Bot name: InAGist [Bot]
Agent match: InAGist
User agent string: InAGist URL Resolver (http://inagist.com)
Website: http://www.inagist.com

kenax
Registered User
Posts: 14
Joined: Thu Nov 21, 2013 3:36 am

Re: Spiders & bots to add to phpBB

Post by kenax » Tue Dec 03, 2013 4:29 am

Jeepers, I'm really having problems getting your forum to run properly. It is giving a 503 server down error to googlbot and only seems to be allowing in magpie. I followed all the instructions but said that anonymous does not have permissions, even though I logged in correctly. Once logged in I simply add manage_bots.php to the address in the browser, right? I blocked access to the folder so that I can give full admin rights to user Anonymous, but now I'm receiving this message:
The user you are logged in as, Anonymous, is not an Administrator with Manage Bots permission
Also, this script seems to be from 2010. Is this bot script still applicable? I wonder why bots can't access the forum.
If I were to reinstall the forum from scratch, which tables could I backup and restore in order to save all existing posts etc., usergroups and permissions?
Thank you for your help.
Work at Home Translation Jobs:
http://homeworktranslationjobs.com/
Earn great money on your own schedule, whatever time zone, and in the comfort of your own environment.

User avatar
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpBB

Post by Pony99CA » Wed Dec 04, 2013 3:39 am

kenax wrote:Jeepers, I'm really having problems getting your forum to run properly. It is giving a 503 server down error to googlbot and only seems to be allowing in magpie.
That makes no sense. manage_bots is a script, not a forum, and shouldn't affect what Google sees at all beyond the typical Bot vs. Guest permissions. If somebody is getting a 503, I doubt it's because of my script.
kenax wrote:I followed all the instructions but said that anonymous does not have permissions, even though I logged in correctly. Once logged in I simply add manage_bots.php to the address in the browser, right? I blocked access to the folder so that I can give full admin rights to user Anonymous, but now I'm receiving this message:
The user you are logged in as, Anonymous, is not an Administrator with Manage Bots permission
That also makes no sense. You shouldn't be doing anything with Anonymous. That's the Guest account that all guest users share, and you should never give that any Admin permissions. You're supposed to log in as an Admin and then add manage_bots,php to the end of the forum root. For example, if they ran the script here, it would be http://www.phpbb.com/manage_bots.php.
kenax wrote:Also, this script seems to be from 2010. Is this bot script still applicable?
Why wouldn't it be. phpBB hasn't changed the layout of the Bot or User tables in any significant way that I'm aware of.

And while the post containing the code was first posted in 2010, if you looked at the last edited line, you'd see that the post was updated Sun May 27, 2012 1:07 am, which is probably when the last release was.
kenax wrote:I wonder why bots can't access the forum.
If I were to reinstall the forum from scratch, which tables could I backup and restore in order to save all existing posts etc., usergroups and permissions?
As mentioned above, manage_bots is just a script that runs along side your board. The changes that it makes to the database are basically the same as you would make through the ACP and it doesn't affect the phpBB code at all, so there's no reason to reinstall.

Did you apply any MODs to your phpBB installation? If so, that's where I'd look first.

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
Volksdevil
Registered User
Posts: 2414
Joined: Sun Oct 03, 2010 2:03 pm
Location: Lancashire, UK
Name: Neil
Contact:

Re: New Manage_Bots Script 5.5

Post by Volksdevil » Sun Dec 08, 2013 1:43 am

Pony99CA wrote:manage_bots has been updated to version 5.5. manage_bots is basically version 4.0 of add_bots with the functions of delete_bots and list_bots merged in. This makes things simpler for me as I don't have to keep the bots tables in sync and simpler for you because there's only one script to worry about.
Steve
Hi Steve, wonder if you can help regards the 5.5 script?

Obviously over time I've been manually adding/deleting bots. I recently deleted a few that I probably shouldn't have (All google bots for example) but I struggle to understand how toy use your script.

Can you explain in simple terms (For a simpleton) what I need to adjust in the php file to simply add/delete all the bots that are listed within it? I can upload the php file and call it with my browser no problem, just unsure on the setting to simply run it 'as is'.

EDIT: Just to add the below bot to your list which is absolutely hammering my forum.

Bot name:
BLEXBot [Crawler]
Agent match:
BLEXBot
User agent string:
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
Website:
http://webmeup-crawler.com
My phpBB Extensions
Finally found great Website Hosting from Image KUALO!
Do NOT use 123-reg.co.uk - Incapable of running phpBB!
:ugeek: TekNeil - Streamer on Mixer | My Volkswagen Corrado G60

User avatar
Ozo
Registered User
Posts: 307
Joined: Mon Dec 13, 2010 7:57 pm

Re: New Manage_Bots Script 5.5

Post by Ozo » Sun Dec 08, 2013 9:46 am

Volksdevil wrote:...

EDIT: Just to add the below bot to your list which is absolutely hammering my forum.

Bot name:
BLEXBot [Crawler]
Agent match:
BLEXBot
User agent string:
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
Website:
http://webmeup-crawler.com
Most of the bots in the script are not meant to roam your site with bot forum privileges :) generally you want to ban or restrict these 'bad' bots from crawling your pages since they really contribute nothing other than take up bandwidth.

User avatar
Volksdevil
Registered User
Posts: 2414
Joined: Sun Oct 03, 2010 2:03 pm
Location: Lancashire, UK
Name: Neil
Contact:

Re: Spiders & bots to add to phpBB

Post by Volksdevil » Sun Dec 08, 2013 9:56 pm

Fair point, I just struggle to distinguish between good/bad bots. I did have many blocked via htaccess anyway, but the one above seems legit, but I may be wrong.
My phpBB Extensions
Finally found great Website Hosting from Image KUALO!
Do NOT use 123-reg.co.uk - Incapable of running phpBB!
:ugeek: TekNeil - Streamer on Mixer | My Volkswagen Corrado G60

Schwpz
Registered User
Posts: 335
Joined: Wed May 07, 2003 1:33 pm
Location: Planet Zot
Contact:

Re: Spiders & bots to add to phpBB

Post by Schwpz » Tue Dec 10, 2013 8:20 pm

Got a new one:

Code: Select all

Bot name: Trendictionbot [Bot]
Agent match: trendictionbot
User agent string: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search;
Website: http://www.trendiction.de/bot; 
..:: PlanetZot.com - Your ultimate source for animation! ^^

User avatar
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: New Manage_Bots Script 5.5

Post by Pony99CA » Tue Dec 10, 2013 11:42 pm

Volksdevil wrote:
Pony99CA wrote:manage_bots has been updated to version 5.5. manage_bots is basically version 4.0 of add_bots with the functions of delete_bots and list_bots merged in. This makes things simpler for me as I don't have to keep the bots tables in sync and simpler for you because there's only one script to worry about.
Hi Steve, wonder if you can help regards the 5.5 script?

Obviously over time I've been manually adding/deleting bots. I recently deleted a few that I probably shouldn't have (All google bots for example) but I struggle to understand how toy use your script.

Can you explain in simple terms (For a simpleton) what I need to adjust in the php file to simply add/delete all the bots that are listed within it? I can upload the php file and call it with my browser no problem, just unsure on the setting to simply run it 'as is'.
Have you tried using the Help command? http://www.example.com/phpbb/manage_bots.php?? That explains the syntax and meanings of the parameters.

I can't test here, but you should be able to use something like http://www.example.com/phpbb/manage_bots.php?operation=add&level=0 to add the phpBB default bots back or http://www.example.com/phpbb/manage_bots.php?operation=add&level=9999 to add all of the bots in the script back.
Volksdevil wrote:Fair point, I just struggle to distinguish between good/bad bots. I did have many blocked via htaccess anyway, but the one above seems legit, but I may be wrong.
manage_bots 6.0 (Beta) changes how the Level option works. Levels now represent the class of bots, not who reported the bots, so bad bots are classified with higher levels than good bots.

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

User avatar
Ozo
Registered User
Posts: 307
Joined: Mon Dec 13, 2010 7:57 pm

Re: Spiders & bots to add to phpBB

Post by Ozo » Mon Dec 16, 2013 1:31 am

By the way, tweetmeme bot is obsolete and if you find it crawling your forum, it's probably a bad bot disguising as tweetmeme user agent.
tweetmeme.com wrote:We're really proud of TweetMeme but all good things must come to an end. Thank you for your fantastic support over the past 3 years.

User avatar
Pony99CA
Registered User
Posts: 4783
Joined: Thu Sep 30, 2004 3:13 pm
Location: Hollister, CA
Name: Steve
Contact:

Re: Spiders & bots to add to phpBB

Post by Pony99CA » Thu Dec 19, 2013 3:11 am

Ozo wrote:By the way, tweetmeme bot is obsolete and if you find it crawling your forum, it's probably a bad bot disguising as tweetmeme user agent.
tweetmeme.com wrote:We're really proud of TweetMeme but all good things must come to an end. Thank you for your fantastic support over the past 3 years.
Thanks for the update, but it won't matter for my script because I keep the bots in the script for historical purposes. The Bots table will still show the last visit of the bot, obsolete or not, so they stay in the script.

Maybe I could create an Obsolete class in manage_bots 6.0, but keeping up with that would be a big job.

Steve
Silicon Valley Pocket PC (http://www.svpocketpc.com)
Creator of manage_bots and spoof_user (ask me)
Need hosting for a small forum with full cPanel & MySQL access? Contact me or PM me.

Post Reply

Return to “phpBB Discussion”

Who is online

Users browsing this forum: KYPREO, topicecold and 27 guests